The other day, I was taking a look at some of our traffic statistics. One of the challenges that I have is determining what our catch rate is. We know how much traffic we see (more or less), we know how much we catch with blocklists and we know how much mail we filter as spam. We also know how much mail we deliver to the end user. What we don’t know is how much of that mail we deliver to the end user is spam.
In order to do so, we’d either need to have every customer submit their spam to us (which will never happen) or we’d have to randomly sample and manually grade the mail that makes it to the end-user and extrapolate that to the rest of the network (which is also equally unlikely to happen).
I decided to do a worst case scenario. I can see that the traffic on weekends always dips, and the amount of mail we deliver always drops by about 2/3. For example, if we delivered 30 million messages during the week, on the weekends we deliver 10 million. These numbers are fairly consistent regardless of our inbound traffic.
I made an approximation that the amount of spam we deliver to users is about the same on weekends as it is during the week, and that that all mail we deliver on weekends is spam. In other words, 2/3 of the mail we deliver is non-spam, 1/3 is spam. This is difficult to believe, but it is also a worst-case scenario. Using these numbers, I calculated that our spam filtering is over 99% effective.
If I didn’t know better, I’d be tempted to say "Wow, that’s pretty good." Unfortunately, being a spam analyst, all I ever hear is how much our service "sucks" (why did this mail come through, we’re getting spoofed, why do I submit this spam over and over again and not see any improvements even though in reality I only submitted it twice and the message headers were missing and the body contents were garbled, etc). In other words, even though we block 99% of spam, we still get plenty of complaints from end users and are reminded all the time that we have to improve our service.
I guess I should rephrase that, I hear plenty of complaints that get filtered up to me, but that’s natural because nobody ever calls to compliment the service, they only call to complain. That comes with the territory. The point is that even hitting 99% spam effectiveness isn’t enough because of the sheer volume of spam being sent; it’s entirely dependent on end-user perception. If a user receives 10,000 spam messages and we block 99%, they still see 100 spam messages. That means there’s some work to do.
On the other hand, I’m pretty proud of our false positive rate.