More Thoughts on Measuring Anti-Phishing Accuracy


Some of you may have seen stories comparing IE7’s anti-phishing accuracy with our competitors, citing different studies than the one I blogged about earlier that showed IE7’s Phishing Filter had the best overall accuracy. Paul Robichaux, from 3Sharp (the company that ran the study I cited), provides his initial take on this other study here.

Tony Chor
Group Program Manager

Comments (17)

  1. lewis says:

    you guys said you are the better, but the firefox team said that they are the better one. who is more believable. see below:

    http://www.mozilla.org/security/phishing-test.html

  2. Big Al says:

    It’s great to see a competition between IE and FF about which product has a better phishing filter. Consumers will benefit from this as theses filters will become better and better. IE7 really is a lot better than IE6. I specially like the new printing engine, which finally prints out what I want. Great work!

  3. hAl says:

    It is not good that the Mozilla team has not checked for false positives in their tests or used any other product that did better in earlier tests then their product.

    It is much much more easy to catch phishing url’s if you risk false positives.

  4. Sandi says:

    I’ve been having a very close look at the raw data (unfortunately spending way too much time putting the data into chronological order) and concentrating on only those URLs where one browser detected the phish, and the other didn’t – I’ve ignored all URLs where both detected, or both didn’t.

    Overall things are pretty steady when it comes to hit/miss percentages between the two browsers, but I note that there are three days with extraordinary spikes in IE7 detection failures – almost as if the phishing filter was down or otherwise having issues – on 29 October, 3 November and 6 November.  The same spikes did not occur with FF.  I also note that from 3 October (when PhishTank’s FF plug in was released) there was an almost complete drop-away in failures from FF (in fact, from 3 to 6 November inclusive, over which time 185 URLs were tested, FF only failed 3 times).  

    Considering PhishTank provided the URLs for testing, the marked drop off in FF failures from the day of their FF plug-in release is suspicious, to say the least.

    If we take away the three days mentioned above then the miss score drop from IE243:FF128 to IE107:FF102.

    I can’t get past the fear that by not making sure that test URLs had not been submitted to Google during or before testing, that the tests are at best skewed, and at worst fatally flawed.  True, the URLS were downloaded every hour, and tested within 15 minutes, but that still allows up to 1 hour and 15 minutes between a phish being submitted to PhishTank and making it to the list – plenty of time for the PhishTank reporters, who in all likelihood use FF as their Web browser, to also submit the same URLs to Google via FF2’s inbuilt Google based service.

    I’ll be putting up a PDF of my checks to http://www.msmvps.com/spywaresucks in the next 24 hours or so. I was going to send it live tonight, but I see that FF says that there were 243 instances where Firefox blocked but IE did not, and that there were 117 instances where IE blocked but Firefox did not,  but my count makes it 243 and 128.  Because of this discrepancy I want to double check my numbers before going live.

  5. Dao says:

    > It is not good that the Mozilla team has not checked for false positives in their tests

    As far as I remember, both IE and the Google Toolbar had 0 false positives in the MS study, so I wouldn’t say that’s critical here.

    > plenty of time for the PhishTank reporters, who in all likelihood use FF as their Web browser

    How do you know that?

  6. Sandi says:

    @Dao

    >>plenty of time for the PhishTank reportes, who in all likelihood use FF as their Web browser

    >How do you know that?

    Balance of probabilities.  PhishTank provides a FF add-in (and a Greasemonkey add-in for Opera), but nothing for IE.  To be honest, I’d never heard of the service before the study although I see there is a spike in publicity in the past week.

  7. Dao says:

    > PhishTank provides a FF add-in (and a Greasemonkey add-in for Opera)

    Well, go ahead and write one for IE, I guess they’ll welcome you. Personally, I wouldn’t know how to do that. For Opera and Firefox, you have to know JavaScript only. That could be a deciding factor for developing add-ons, but it doesn’t mean that the whole community uses Firefox (heh, why would they support Opera? :). Note that you don’t need an add-on in order to report phishing sites, but to automatically check against the list.

  8. goose says:

    Anti-Phishing is ESSENTIAL to my safety. I cannot possibly be expected to pick out the genuine from the fake with my busy schedule and low mental capacity! Just last week I bought Fyagra. It turns out it was fake! I have no idea what I’m doing half the time with all these fakesters.

    I trust IE more!

    Study shmuddy. IE has always cared more for our security. Over the past several years, not one spyware/adware infection, but we keep hearing about them from lesser companies – constantly. Cleaning them up is an industry unto itself.

    IE is right to come with Windows without an uninstall feature. Tightly integrating it with Windows and Windows applications was a stroke of pure genius.

  9. BigAl says:

    http://www.53.com.portal.ehasbee.jp/startproc.id/ gets blocked by FF, but not IE. I’ve reported it using the IE feature, let’s see how long it takes to block this site.

  10. Scorpion3003 says:

    Lex

    Now FF2 Does block that site.

  11. Aedrin says:

    goose has lost his/her magic. Sounds like Fduch now.

  12. Fduch says:

    Don’t beat me to tell you do how you sound yourself.

  13. …I think that compertion between IE and FF will only benefit the consumer in the end.

    And that is a good thing

    Marc Liron – Microsoft MVP

    http://www.updatexp.com/ie7-issues.html

  14. hAl says:

    this studie from Carnegie Mellon University in Pittsburg concludes that ALL anti-phising tools are still pretty crappy:

    http://www.cylab.cmu.edu/files/cmucylab06018.pdf

  15. Keith Cash says:

    I turned mine off. Kept slowing down searching. I  do not feel like I have missed anything.

Skip to main content