Anti-Phishing Accuracy Study

As we’ve worked on the new Phishing Filter in IE7, we knew the key measure would be how effective it is in protecting customers. In addition to our internal tests, we wanted to find some external measure of our progress to date as well as pointing to ways we could improve. We didn’t know of a publicly available study covering the area, only some internal and media product reviews. (We’ve blogged a few times about the new Phishing Filter in IE7; in addition to these technical details we published the results of a 3rd party privacy audit.)

To help us answer this question, we asked 3 Sharp LLC to conduct a study of the Phishing Filter in IE7 along with seven other products designed to protect against phishing threats. In order to establish an accurate methodology on a level field, they utilized four sources of independent data that are not used to populate the IE7 Phishing Filter service today. They worked hard to build large enough sample sizes of actual phishing sites to draw meaningful conclusions.

3Sharp LLC tested eight browser-based products to evaluate their overall accuracy in catching 100 live confirmed phishing websites over a six week period (May – July 2006) and also understand the false-positive error rate on 500 good sites. In addition to IE7, the toolbar and browser solutions tested included the offerings from EarthLink, eBay, GeoTrust, Google Safe Browsing using Firefox, McAfee SiteAdvisor, Netcraft, and Netscape. You can see actual version numbers in the detailed report.

We are pleased to see that Internet Explorer 7’s Phishing Filter finished at the top of 3Sharp’s list as most accurate anti-phishing technology, catching nearly 9 out of 10 phishing sites while generating no warning or block errors on the 500 legitimate websites tested. You can read the report for yourself at 3Sharp’s website. The report contains details on the methodology, the data sources used and even a list of every single URL tested.

It’s great to see so many companies looking for different ways to address the significant problem of phishing. We think that the results reported by 3Sharp validate the unique approach we’ve taken of combining a service-backed block list with client-side heuristics. That said, we understand that the threat posed by phishing is constantly evolving as are the tools designed to protect users, so this set of results represents only the relative performance during that period. We know we need to keep working to keep up with the changes in the attacks and are already using the results of this test to further improve the efficacy of the Phishing Filter.

If you’re using IE7 but not already using the Phishing Filter, I encourage you to turn it on (you can find it under the Tools icon) and browse with more confidence. If you’re not using IE7 yet, you can install our latest version here.

Tony Chor
Group Program Manager

Comments (60)

  1. Anonymous says:

    Nice 😀

  2. Anonymous says:

    Keep up the good work team.

  3. goose says:

    With Microsoft AntiPhish nothing can fool me. Thank you for AntiPhish!! I feel more confident clicking random things now without care. Microsoft, you are GREAT. Innovation! Best CSS support. Best record of blocking spyware. Good GUI. Thank you.

  4. __hAl__ says:

    Really funny is the score of McAfee Site Advisor scoring 3 points out of 200 where the top two products including IE7 score 172 and 168.

    Hilarious even !!!

  5. Jeff says:

    Congratulations, guys. Those are some pretty impressive scores. Poor McAffee. 🙂

  6. 3sharp, a Redmond based technical services company, has been commissioned by Microsoft to undertake a

  7. David Taylor says:


    I will definately turn it on for my parents and non-technical friends.  However I am not convinced it wont slow down my browsing.

    Could you give us an outline in technical detail of what steps are taken to ensure this does not slow down browsing.

  8. Francis says:

    Good work! However, I must note that my installation of RC1 identified a Microsoft web site under as a phishing site. So either it’s a false positive or… ? 😉

    (Unfortunately, I no longer have the link.)

  9. Sheep and Duck says:

    3Sharp was founded in 2002 by three friends: Paul Robichaux, Peter Kelly, and John Peltonen, all experts in their respective fields. Their goal was to establish a company that could demonstrate the robustness, flexibility, and sheer native capabilities of the Microsoft communication and collaboration technologies. By working closely with Microsoft’s Information Worker Group, 3Sharp has always been able to stay on the cutting-edge of the Office System technologies.

    Somehow I don’t trust this "study".

  10. Everyone: thanks for the comments far!

    David: In order to minimize the performance impact, we do the phishing filter checks asynchronously from the navigations, that is we don’t block navigation while we check. If you prefer, you can also run the Phishing Filter manually, by turning PF off and then choosing "Check this website" from the Phishing Filter option under the Tools menu. This allows spot checking of sites you think might be suspicious.

    Francis: We’re continuing to tune the Phishing Filter heuristics to minimize false positives to prevent good sites from being flagged.

    Sheep and Duck: I’m not sure how to convince you to trust the study; it doesn’t help us improve the product if we weight the results. Ultimately, you should be the judge based on your personal experience with the tools.

  11. Sheep and Duck, I understand why you’re skeptical. No matter who commissioned the study, *someone* would distrust the results on that basis alone. However, I think if you read the report, you’ll see that we have been transparent about our test methods and the data we used for the test. If you read the report and still have questions, feel free to contact me via e-mail ( or my blog ( and I’ll do my best to address them.  

  12. Jill says:

    I would be very interested to know, in cold hard facts (not a comissioned study), how many end users have been "scammed" by a phishing site? per month, per year, and ideally by geographic location.

    As an avid user of the Internet for many years, it would take a heck-uv-a-lot of a scam, to actually get me to part way with my money, or be convinced that the site was legitimate.

    7 times out of 10, the url gives it away, or the graphics, or a complete mis-guided attempt at English. ie. "For our security, your credentials you must enter below and click submit"

    Profesional web sites, just don’t have glaring gramatical errors on them (or if they do, they are cleared up within minutes of posting)

    My other comment, is a request.  Can the phishing filter be added as a right click option on the url?  Since this is the address of the site, it is the most logical place on the screen to go to, if in doubt.

  13. Steve says:

    Interesting.  Microsoft’s main competitor in the Web Browser market is of course Mozilla Firefox.  Why in the study, was there no comparison against Firefox 2.0 Beta X or RC1?  It is by far a superior browser, and now too includes anti-phishing technology.  Were the study’s commissioners worried that going head to head with a better browser, might indeed show that their anti-phishing technology was also not up to par?

    Very interesting, because in the Press Release, at 3Sharp, it mentions testing with Google Safe search… which, well, is all very good, but not an apples/apples comparison in the slightest… but interestingly, only here, in your comparison list, is Firefox mentioned…

    It sounds like Firefox was an active player in the tests, when it most certainly was not.

  14. Microsoft sponsors an antiphishing technology bake-off. Guess who wins…

  15. Shane Keats says:

    This is Shane from McAfee SiteAdvisor here. We’re not surprised to find out that we came in last in Microsoft’s anti-phishing study.

    Why? Because we don’t offer anti-phishing.

    We test for a lot of important things that no one else does, like whether a site’s e-mail practices result in spam, or whether an offered download bundles spyware, or whether the site attempts to breach browser security, or whether the site agressively links to known bad sites.

    But we don’t offer anti-phishing protection, at least not yet. We’re pretty explicit about that too:

    <a href=""&gt;

    "SiteAdvisor’s software does not currently provide automated or real-time phishing detection."</a>

    SiteAdvisor’s protections complement McAfee’s other products which do deliver strong anti-phishing protection.

  16. Steve: we didn’t include Firefox version 2 because at the time our test started it wasn’t available in beta. Because 2.0 incorporats the same code base as Google’s Safe Browsing add-on for Firefox 1.5. I think the results are representative, although I’m sure you’re right that the release version of 2.0 will do a better job.

    Shane: I wrote you a lengthy mail explaining why we thought SiteAdvisor was an anti-phishing tool. Just in case your mail filtering system blocked it, I’ve explained our reasoning at <;. If you’d like to discuss this further, my door is always open.

  17. Aedrin says:

    "Why in the study, was there no comparison against Firefox 2.0 Beta X or RC1"

    I don’t see how a minor version update matters in the results.

    I still don’t understand which feature made it require a major version increment.

    Sounds to me like they’re worried about IE’s major version increment.

  18. Joe says:

    "we didn’t include Firefox version 2 because at the time our test started it wasn’t available in beta."

    A valid explanation, but ultimately futile. In no time some other reason to distrust this study will be brought up, simply because to some, studies that have MS coming out on top are automatically ‘suspicious.’

  19. Joe says:

    Wow, looks like the firefox lobby has arrived. I guess this story was posted on slashdot?

  20. Steve says:

    @ Jill

    You said: "As an avid user of the Internet for many years, it would take a heck-uv-a-lot of a scam, to actually get me to part way with my money, or be convinced that the site was legitimate."

    That may be correct for me and you (let’s call ourselves ‘power users’ – not sure if you would brand yourself as that, but I would, and I’ll assume you are fairly competent computer user).

    However lets take my mum… or my Gran.. or heck my 25 year old friend – varying degrees of computer literacy, but would all probably fool for a phishing scam if it was good enough.

    I agree some are bad… no, in fact, some are really bad – but some are also good. Came across one in my mum’s inbox ‘from PayPal’… HTML email was well crafted, good English, website looked very authentic. The link text in the email was different from the actual URL – but not everyone would check.

    Suppose all i’m saying is this feature isn’t for you and me… its for your mum, dad, grand, aunt, or anyone else you know who isn’t a fairly knowledgeable web user.

  21. I hate fanboys says:

    Hey, you, YES YOU, firefox fan boys, please don’t post off-topic posts. If you don’t like IE just don’t go there. JUST DON’T.

  22. EdH says:

    Hrm…. Seems to have done the best job of blocking what it detected, where you get 2X the points than for just detecting and warning users. When it comes to warning, IE7 didn’t win in any category.

    Great work so far, but I’d like to see better performance in detection.

  23. Dao says:

    I don’t know about Microsoft Phishing Filter, but I think Google Safe Browsing gets better the more users participate. Thus speaking of accuracy, I could imagine a boost once Firefox 2 ships.

    > I still don’t understand which feature made it require a major version increment.

    > Sounds to me like they’re worried about IE’s major version increment.

    Let’s be fair. Firefox has been a decent browser since 2004, whereas Internet Explorer wasn’t. It made a straight progress with 1.5, whereas Internet Explorer didn’t. Now it’s no surprise that Mozilla doesn’t have to fix as much as MS. If you want a comparison, then take IE 5.5 and 6.

  24. Mike Jackson says:

    Anti Phishing keeps picking on Yahoo! Answers every time I try to log on

  25. Sandi says:


    From February to Mid Aug 2006 the Phishing Filter helped block over 800,000 instances of people trying to access reported phishing websites using IE7 or MSN/Windows Live Toolbar.  This figure includes almost 500,000 blocks since IE7 Beta 2 was released.

    IE7 users are reporting up to 4,500 potential phishing sites per week.

    Microsoft has been adding up to 17,000 URLS a month to its Phishing Filter service.

  26. Anomynous says:

    Unfortuantly, anything that shows Microsoft came out at the top will immediatly be looked upon with suspicion because of the well known sucurity problems with Internet Explorer 6 and Windows XP. Microsoft will have to work hard to turn that negative image around.

  27. Anomynous says:

    Unfortuantly, anything that shows Microsoft came out at the top will immediatly be looked upon with suspicion because of the well known sucurity problems with Internet Explorer 6 and Windows XP. Microsoft will have to work hard to turn that negative image around.

  28. Fduch says:

    How beta testing of IE7 on Connect goes with regards to the filter:

    q> IE7 crashes on these websites: {…}

    A> Do you have phishing filter on?

    q> yes

    A> Turn it off. Closed

  29. Luc says:


    Firefox 2.0 uses Google antiphishing

  30. Schreuder says:

    Can i test myself the antiphishing filter? Like the EICAR tests? Any URL known to be malicious?

  31. tseving says:

    I’ve been using Sitehound, a product of Firetrust, a NZ company.  Sitehound uses an internally stored URL database, updated daily.  I turned it off and clicked a few banners that are known scam/spyware sites.  The antiphishing filter let me into about half of them.  If addition, the antiphishing filter gave me several false positives, including my own Comcast web mail beta email account.  While I don’t have any measurable data to support this, my browser seemed sluggish with antiphishing filter engaged.  So I disabled it and enabled my Sitehound again.  I’m not intending to bad-mouth IE7.  I love it.  This is just feedback FYI.

  32. Does Opera or Safari have or will have this feature as IE7 and Firefox 2 will? Good to see this being added. Any idea when we’ll get a slightly newer build then 7.0.5700.6?

  33. Eduardo Valencia says:

    it’s a challenge for you guys (IE Team) to make this broswer the best of all,please

    It’s important you keep improving, adding new functionality prior to release

  34. Sandi says:

    Shane Keats,

    As I said in response to your comment on my blog, your service warns of "fraudulent practices" and has tested "sites representing more than 95% of worldwide Web traffic" and performs "tens of thousands" of tests every day, but phishing sites aren’t included?

    No exclusion of phishing sites here either:

    Perhaps you should be more specific about what these "fraudulent practices" are (fraud, but not phishing, despite phishing being a type of fraud?) and add a mention about not covering phishing in the FAQ in addition to the Support Centre (people won’t go to the support centre unless they have problems).

  35. Il est amusant de constater que les simples faits qu’une &#233;tude ait &#233;t&#233; financ&#233;e par Microsoft et que…

  36. TONY CHOR:

    Thank you for a fine product!

    READ THE ABOVE COMMENTS AT LENGTH..and must comment myself. WE have a trust issue to face!

    I have installed IE7 RCI with live toolbar and used now over a month. Note I as well have McAffee Anti-virus and Spam Filter.

    My dedication to Microsoft Products is founded

    on all you are doing to improve each day and each week.

    YOU and your teams are amazing me so I vote

    to use Microsoft now and in the future.

  37. Harold says:

    @George  You seem very pro-Microsoft in your comments.  Thats good.  Out of interest, you say you use MS products now/and intend to in the future, also good.  But one question.  Have you tried other Web browsers? e.g. Opera, Maxthon, Firefox?

    I would hazzard to guess you haven’t, since I think you’ll find the other products even more impressive. Even the "tabs" are better in IE6, with the MSN Toolbar.

    (((Yes MS Developers… that’s right! The tabs in IE6, with the MSN Toolbar, are BETTER than the tabs in IE7)))

  38. EricLaw [MSFT] says:

    @Harold: I’d love to hear more about what you think is better about the MSN Toolbar’s tab implementation.  Thanks!

  39. RRWW says:

    actually IE7 tabs are much better than MSN Toolbar

  40. Reddy says:

    It is nice to have  Anti-Phishing  Services available at individual Enterprise level. As I understood that if  Anti-Phishing  options enabled , then every link will be scanned by "Anti-Phishing  services or by servers at " . This is kind of privacy concern. If Microsoft come up having tool to have Anti-Phishing  service or server available at individual enterprises ( like the way SUS is) , then individual enterprises sync with Microsoft Servers and route all the traffic thru  enterprise owned and controlled  Anti-Phishing  Services/Servers.

  41. IEBlog says:

    Some of you may have seen stories comparing IE7’s anti-phishing accuracy with our competitors, citing

  42. McAfee, which originally disputed SiteAdvisor’s inclusion in the 3Sharp phishing filter tests back in

  43. Paul Thurrott, in his somewhat late review, takes a look at Firefox 2’s new features. Unfortunately, and something I didn’t expect from Paul, it’s either the most intentionally misleading review of Firefox, or it’s a completely unserious piece of writing.

  44. IEBlog says:

    Back in November, we announced our intention to bring Extended Validation SSL Certificates to IE7 . This