Last month, Seattle had a snowstorm that shut the city down for about a week. Don't laugh, all of you mid-westerners, here in Seattle we have hills that go nearly straight up and down so in icy weather you can't drive anywhere. Not only that, but snow plows in Seattle are as alien as vegetarians in Texas.
For this reason, I had to work from home for a while. Sure, it sounds like all fun and games, you know, like that movie... Space Balls. But after a while, with everybody gone I was unable to follow up on anything and my Internet VPN connection wasn't that great. Towards the end it was starting to get a little boring, you know, like that other movie... Police Academy.
To help pass the time at home, I have this habit of having some background noise on. Invariably, this is with the TV on. I don't necessarily watch it, I just like the noise and occasionally I look up and catch a few scenes (that's why you see me make so many references to TV shows on this blog). While I was working from home, I finally caught an episode of the animated series The Clone Wars, which is a series of short stories in the Star Wars universe which takes place between Attack of the Clones and Revenge of the Sith.
In this particular episode, the Republic was holding one of the leaders of the separatists captive, Nute Gunray. While guarding Gunray in his holding cell while the Jedi are off searching for a Sith witch (those of you who watch the show will know what I'm talking about... those of you who don't, never mind), two clone soldiers hold a conversation with the young Jedi apprentice. There is a battle engaging down below but the debate is whether or not they should leave their post. Finally, the young Jedi apprentice says "I know my master told me to stay here, but I have to disobey her orders and go help her!"
"Yes," agree the clone troopers. "That's why humans are better than droids, sometimes they have to disobey orders." The young Jedi goes on to save her master and the evildoers of the episode are thwarted.
I've gone through this big spiel to illustrate an analogy between Star Wars and spam fighting. We have a lot of automated techniques to deal with spam. We create machine-generated rules and blocklists that parse through logs and create them dynamically. Yet for all the automatic spam filtering we do, the most reliable and accurate antispam technologies are the ones that require human analysis.
For example, when detecting outbound spam, I haven't stumbled across a reliable algorithm that separates the outbound spam from the false positive noise. We've got some alerting going on but for the most part, when a human looks over the reports, that is far more reliable for detecting when a customer has been compromised and is sending out spam through our outbound pool. Similarly, we can add individual IPs to blocklists, but when a human goes through and starts investigating behavior from similar IPs, we often end up adding large entire netblocks (pre-emptively) that we might normally not have added automatically. The automated portions are FP prone and we tend to be risk averse.
Even Bayesian filtering has its drawbacks. At this point in time, I think that Bayesian filters are great. But I don't think that they are much better (catch more spam) than spam analysts writing regular expressions. The difference is that they scale a heck of a lot better, and therein lies the difference.
That's the point I am trying to make; automated antispam techniques (droids) work well because they scale nicely since there's no way a handful of people can deal with 50,000 spam submissions. But on the other, I still have a bias towards humans (clones -- but not literally clones) to make the really good spam rules, the stuff that separates a good filter from a mediocre one.