The Basics of the IE Testing Matrix

The information published in this post is now out-of-date.

—IEBlog Editor, 20 August 2012

As we recently shipped security update MS04-025, I thought it would be good to talk about the testing coverage that we do for each security update.  Testing is about risk management, and we have to make judgments about where to invest heavily in testing and where to invest less in testing to achieve the highest quality IE when we release.  Code change is one main driver of this (test where the code is changing), but another important part of the process is to understand the larger landscape of what you are testing and how it fits into its overall software ecosystem.  

If you take a look at the security bulletin for MS04-25 (, you will the see two of the main variables for our testing – IE version and Windows version.  I have copied those lists here for reference.  These lists are generated from the Microsoft Lifecycle policies (;en-us;lifecycle).  

Internet Explorer Version (for MS04-025)
Internet Explorer 5.01 Service Pack 2
Internet Explorer 5.01 Service Pack 3
Internet Explorer 5.01 Service Pack 4
Internet Explorer 5.5 Service Pack 2
Internet Explorer 6
Internet Explorer 6 Service Pack 1
Internet Explorer 6 Service Pack 1 (64-Bit Edition)
Internet Explorer 6 for Windows Server 2003
Internet Explorer 6 for Windows Server 2003 (64-Bit Edition)
Windows Version (for MS04-025)
Microsoft Windows NT® Workstation 4.0 Service Pack 6a
Microsoft Windows NT Server 4.0 Service Pack 6a
Microsoft Windows NT Server 4.0 Terminal Server Edition Service Pack 6
Microsoft Windows 2000 Service Pack 2
Microsoft Windows 2000 Service Pack 3
Microsoft Windows 2000 Service Pack 4
Microsoft Windows XP and Microsoft Windows XP Service Pack 1
Microsoft Windows XP 64-Bit Edition Service Pack 1
Microsoft Windows XP 64-Bit Edition Version 2003
Microsoft Windows Server® 2003
Microsoft Windows Server 2003 64-Bit Edition
Microsoft Windows 98
Microsoft Windows 98 Second Edition (SE)
Microsoft Windows Millennium Edition (ME)

Not all versions of IE run on all versions of Windows.  For example, IE 5.01 SP2 is specifically produced for Microsoft Windows 2000 Service Pack 2.

There are more variables than just IE version and Windows version, and some like language create a multiplier on IE and Windows version.  For IE 5.5 Service Pack 2 there are 26 language specific binaries.  With the general assumption of 26 languages for each IE version, we have at minimum 234 installs to test for each security update we release.  My test team has to make sure that each one of these install packages installs correctly and works as expected – that is a lot of setups! It is definitely the type of testing where we utilize automation and tools to reduce the need to manually do all those setups.
We also include other test variables in our coverage, like applications and sites, which can range from a variety of client based applications that host the IE browser to web sites that leverage different scripting technologies.  Application and site compatibility is very important to IE, and in order to get good testing coverage we have to anticipate it will impact the Internet as well as corporate Intranets.  In order to address this, we classify apps, sites and technologies into buckets, which allows us to test specific sites or apps that are representative of the overall technology bucket.   We also look at a variety of measures that tell us what the top sites are, and then test IE against those sites.  

This is just a quick snapshot of some of the test coverage issues we grapple with in testing IE.  More testing tidbits later.

Comments (18)

  1. Anonymous says:


    wow – this is an impressive test matrix and i really do not admire the people running and controlling all those steps. Patience isn’t my strong side 😉

    Because of my impatience i found a tutorial on how to install multiple IE Version on one Windows XP machine ( This is a lot faster than running multiple virtual pcs or stuff like that and it works in most cases like a "real" installation (i only test scripting/CSS/rendering though).

    Maybe a feature like that could be implemented in IE7. Some kind of "history mode" for developers to easily switch back to ancient versions of IE.


    P.S. is a great site anyway. For me the #1 ressource for all browser related stuff. Probably tons of valuable infos what needs to be done in IE7 😉

  2. Anonymous says:

    Wait a second — you guys test? Now, that’s a shocker.

  3. Anonymous says:

    How do you actually do these tests across multiple configurations?

    Are you using a VPC environment or do you have a custom solution that emulates versions of IE on certain Windows releases?

    It would be hear more along this line 🙂

  4. Anonymous says:

    Firefox has 13 languaages, and 3 platforms. I wonder if they bother testing it for every Windows platform. Probably not.

    If they don’t, you can’t complain though can you? It’s free after all, and you should be _grateful_ for that.

  5. Anonymous says:

    Hmm now that we have a fairly functional and non invasive popup blocker (for free)do you think MS could do something about a DHTML show/hide block too. Now, that the popups are blocked the nuisance sites have expanded on this one and it is as potentially as dangerous as a popup?

  6. Anonymous says:

    We use a variety of re-imaging systems to test across the multiple configurations, including VPC. I will take that feedback and work to come up with a future post with more details on how we handle the multiple configurations.


    IE Team

  7. Anonymous says:

    This is completely the wrong place to post this, but I’m going to anyways 🙂

    Want to see an awesome bug in IE I discovered, to my dismay, a few days ago?

    Open IE’s FTP client to a site where you have read/write access. Move some files to your desktop, and once you get to about 1%, press Cancel. The copy cancels, but IE will happily delete all of your files!

    OUCH, needless to say, it ate about 3 gigs worth of web logs on me – I wisely decided to cancel the move after I noticed the file size and wanted to zip them… But YIKES, talk about data loss.. 😛

  8. Anonymous says:

    > Firefox has 13 languaages, and 3 platforms. I wonder if they bother testing it for every Windows platform. Probably not.

    Becasue Firefox is not so tied to the system that IE is therefore it’s likely to work the same on any service pack of a given MS system. Exceptions to this may be big changes like XP SP2 which are like a major upgrade.

    Also due to the fact that nightly builds are available and many people download them to test them we can be sure that there’s people with all sorts of weird configurations from early versions of Windows 9x up to XP with various levels of patches applied, some people are probably also running it on test builds of Longhorn.

    We also have users testing nightly builds in literally hundreds of different Linux configurations, and various releases of MacOS.

    Having volunteers to test nightly builds can really help fix bugs that wouldn’t be detected otherwise, bugs that may only occur with some obscure hardware installed.

    It’d be nice to see nightly builds of IE, it’d certainly keep people interested in the progress being made. In the Firefox world Ben can post on his blog about this cool feature he’s just implemented and people can download a test build the next day.

    The only problem is the IE install is a bit of a pain, unless you follow the hack that allows multiple versions on the same machine.

    Please, make it so we don’t need to reboot after installing IE anymore. Make the install as simple as Firefox (you can run an installer, or you can even just unzip a file into any directory)

    Note: in Firefox Windows 95 is not supported (just like the IE team have found it too much hassle to support), however the Mozilla suite (at version 1.7.2 at this time) supports Windows 95 still, so Win95 users haven’t been totally neglected.

  9. Anonymous says:

    IE is a system component, not just a browser. It provides APIs and COM interfaces that are consumed by dozens of applications. Browsers like AOL and Netcaptor, as well as many applications that use the WebBrowser for content rendering.

    I’d love to see the application compatibility test list – which apps that rely on IE that Microsoft tests for IE compatibility. I’m sure AOL is top of the list, as well as a bunch of companies that have their lawyers on speed dial the minute they think Microsoft broke their application.

  10. Anonymous says:

    This calls for an immediate redesign, 234+ test environments is asking for trouble. Take your programmers away for 1 week to move code, it will leave you with maybe 10 additonal modules which have to be tailored, the rest can stay the same. You are all too much focussed on the tasks at hand that you forgot the foundation, without a good base to work with it is not worth the effort.

  11. Anonymous says:

    ASP sucks :/

  12. Anonymous says:

    Surely, 200++ test environments is just asking for trouble!

    One of the first things any CS student will probably (I say probably as I have yet to go to uni) learn is that coupling classes tightly with other classes is a very bad thing, especially if there is a large number of dependencies. It makes it hard for the class to be updated without breaking code in other classes. This is very basic software engineering knowledge.

    This concept also applies to this scaled up scenario. IE is so closely coupled with the base of the OS that is makes it a nightmare to do updates to the codebase.

    People have been against IE being tightly bound to the OS since IE4. As the days go by it becomes more obvious why. Now Microsoft are having to pay for their corporate arrogance / stubbornness to listen to others.

    I find it very disturbing that IE7 (from what I last read) will only run on Longhorn. This basically says to users that IE7 is highly integrated into the OS.

    Why does a browser have to be so tightly integrated? Sure, some apps rely on it’s rendering engine, but it doesn’t mean a user’s computing experience should revolve round one of the troublesome pieces of software known to man.

    I understand that current versions of IE aren’t suddenly going to decouple from the OS. However doesn’t this illustrate why such tight coupling with the OS is so bad.

    If IE was not so tightly integrated, it would make it much easier to modify it’s codebase, and therefore things like updates and patches could be done it a more response fashion.

    Surely things like languages should not be a major issue when it comes to testing. Generally, language specific string can be handled with defined constants or variables, and provided each language constant is defined in the source (and correctly translated) it makes sense, there is no reason why things should screw up.

    With well designed code, testing should not be a major issue. With things like unit tests, testing to ensure known bugs don’t resurface should be a very straight forward affair. Likewise Continuous integration techniques would allow you to automate testing each time you commit code to the main codebase.

    Fair enough I realise this isn’t going to change for current versions of IE out there, but hopefully this might be a little push factor when it comes to development of future browsers.

    I really do hope that IE does get sorted out fully. There is a long way to go. As a web programmer / designer, I spend a very height percentage of my time trying to fix up designs to make sure they work in IE. I have no problem with getting Safari, Konqueror, or Opera to work. IE’s rendering issues are something that need addressing now that you are starting to get on top of security issues. IE has a lot of potential. If IE wasn’t left to rot for so long, IE would not have been half as threatened by other browsers. IE is holding back web technology, due to it’s poor security and bad standards support. A company like Microsoft should be setting the standards in this area, and certainly not be trying to play catch up with other browsers.

  13. Anonymous says:

    Oh, and on a side note. Well dones for getting this page tableless, it loads a lot quicker. It would be nice to see it validate under XHTML 1.x Strict though!

  14. Anonymous says:

    Shadowchaser –

    We have tried to repro the FTP issue you ran into, but are not reproducing the failure.

    Do you have more details (exact repro steps, server config, etc)? Can you repro with smaller amounts of data?



    IE Team

  15. 3 weeks on, IE 8 Beta 2 is now available in an additional 21 languages, bringing the total to 25. As a comparison, as far as I can tell that’s only one fewer than IE 5.5 and 6 were released in, ever. Another sample Microsoft beta, that for Windows Server