Following on from my last post, there are a couple of more considerations that we have to take into account if we want to measure a spam filter's effectiveness. I have already hit on three points (continuous, automated, need a lot of mail). Let's look at the next one.
Measuring effectiveness cannot interfere with legitimate mail flow but should mimic legitimate mail flow
This requirement is a "passive" requirement. It effectively says "Look, but don't touch." What do I mean by that?
Essentially, I mean that if you're going to take collect a lot of mail, you need to collect it from somewhere. Ergo, you will need to take a copy of the mail from somewhere. Let's suppose you are forking the mail from production and running it through a parallel pipe. This means that the original mail must pass through the filters unharmed and treated as if it never was touched to begin with. Meanwhile, the copy of the mail should be passed through a parallel network dedicated only to doing measurement. Afterwards, that mail is discarded.
In other words, the measured mail is a copy of the original mail and the original mail is not interfered with. Because you are taking a copy of so much mail, there must be an intermediary in between the original mail and the copy. Another agent, either a mail server or some other piece of software, must fork and copy this mail. We have routers that can route mail different based upon DNS lookups, but this is redirected mail away from its original destination -- and that fails my requirement for not interfering with legitimate mail flow. Thus, I am going to assume that in order to implement this, you need some additional piece of software and/or hardware.
This software will copy the mail to the parallel network and the parallel network needs to know that it is coming from a forked feed. Therefore, the measurement network must know that the source IP address, ie, the hand-off IP, is not the actual originating IP. All subsequent IP checks, like reverse DNS checks of the sending IP , RBL checks and SPF checks must account for this. So, now you have two different sets of roles for your filters - one that filters mail normally in production and one that mimics filtering mail but has to do some extra work to extract the correct sending IP... and then subsequently delivers the mail to a different location afterwards (ie, it discards it rather than deliver it to an end-user mailbox). It has to work around the limits that are inherently built in to the requirement that regular mail flow must be transparent to the end user.
This means that all of your software builds will now have additional config file set ups. It also means that the Operations department cannot treat these as regular mail filters, they are special mail filters and need to be treated as such. Deployment and monitoring is not the same as everything else. If these mail filters go down, bringing them back up is not the same urgency as the rest.
It is this cost that is something that I did not originally envision when we embarked on this project. Maintaining duel sets of roles for a small set of filters that do not quite the same thing is one of the costs that must be absorbed. But, if you are going to continuously measure your spam effectiveness, it is a price that must be paid.