Hotmail’s spam percentage declines significantly
I was reading an article on PC Mag today and he reviews Hotmail’s claims that it has made significant progress against the spam problem during the past several years. However, the language in the post is confusing so I will clear things up.
To begin with, Microsoft has made significant improvements in the amount of spam messages that get to the user’s inbox – they get fewer spam messages per mailbox. Hotmail measures this with a metric that they call Spam in the inbox. It is the proportion of bad mail to good mail in a person’s email inbox. It is calculated this way:
SITI = Spam messages / (spam messages + good mail)
Where spam messages are the amount of missed spam. So, if you have 100 total messages in your inbox, and 3 are spam, then SITI = 3 / (3 + 97) = 3 / 100 = 3%.
Hotmail has a good mechanism in place for determining this. Volunteer users in their feedback loops grade their mail. Hotmail will send them a message saying “Is this message spam or non-spam?” Based upon what the user says, they can compare what the filter said. Thus, if Hotmail marks a message as spam and then sends a copy to the user saying “Is this spam?” and the user says yes, then they know that the filter made the correct decision. If not, then it made a wrong decision. Taking all of this user feedback, they can measure their effectiveness. When they say they’ve reduced their user’s spam by 90%, they mean that a few years ago that SITI was 30% (as measured by clicks from their users) whereas today it is 3%.
Some users misclassify mail but with large enough sample spaces from the entire user base, it evens out. Thus, if Microsoft’s metrics show that they are getting better at fighting spam, then they really are getting better. This is a metric that has been around for years and it is accurate.
Next up, the article says that Microsoft’s improvements in its Smartscreen spam filter has cut the amount of spam on the Internet by 15%. This isn’t accurate, the blog post said that they’ve cut outbound spam by 75%, and Microsoft played a key role in reducing total amount of spam on the Internet by 15%.
To understand this, we all know that spammers sign up for Hotmail accounts and use them to spam. But Hotmail understands that spammers are gaming them and they have put measures in place to counteract this (so have we) and this has cut down on the amount of outbound spam that they send, which helps to reduce the total amount of spam on the Internet.
Microsoft, not just Hotmail, also played a role with the Rustock shutdown this year that did have an effect on the total amount of spam this year. While spam levels have remained stable for a few months, taking down Rustock was a very big victory against spammers.
On the other hand, there have been a number of factors that have contributed to the decline in spam. Microsoft did play a big role, but other factors include the disappearance of some spam operations (like SpamIt late last year), criminal prosecution against other spammers, and a general shift in the cyber abuse industry from spam to malware and black search engine optimization. To many criminals, there is more money to be made from narrow targeted against then mass spamming.
The battle continues.