Hotmail and Gmail are tied in antispam effectiveness

A couple of days ago, the New York Times reported on a study by Cascade Insights that measured the effectiveness of the spam filters in Gmail, Hotmail, and Yahoo Mail.  The results?  Hotmail and Gmail are about equal in terms of how good they are, with Yahoo lagging by a good margin.

Previously, Hotmail suffered from a perception that its spam blocking wasn’t very good, and by their own admission, it was true.  However, since then, Hotmail has invested a lot of resources towards improving their content filters, as well as their bulk mail filtering, and now that perception is inaccurate.

I went and read through Cascade’s study and the methodology they used was to set up a bunch of test accounts and over a period of five weeks, seed them with spam and measure the results. 

The problem with the report is that they did not report on how many total messages were sent to the spam folders.  Why is that important?  Because over five weeks, 250 spam messages is only 50 messages per week, or about seven spam messages per day.  Even if we allow that 10x that many went to the junk mail folder, that’s only 77 messages per day.

77 messages per day is a tiny sample of the billions of spams that are sent each day.  At the 95th percentile, it produces a margin of error of +/- 11%.  I’ve written about how to measure spam effectiveness previously, and to summarize, you need a lot of messages.  How many?  To prove that Hotmail is the definitive winner requires a confidence interval of +/- 0.15% which requires nearly 3/4 million spam samples per day (you can calculate this yourself here).

Thus, while this study is a step in the right direction and shows that Hotmail and Gmail are close to each other, we cannot conclude that one is better than the other.