Summarizing Common Browser Tests


To deliver on the industry promise of “same markup,” browsers must implement features in the same way. Open Web standards and their associated public test suites provide the means for making sure this happens. Using the same process for tests as for the standards themselves is so important that Microsoft contributes to this industry effort in many ways, such as submitting conformance tests to the W3C.

In addition, we believe it’s important for these tests to be comprehensive and test the depth and quality of standards support, not just the existence of features. Compliance tests from standards bodies do that.

Standards Compliance

We’ve written about these tests before. They come from the W3C working groups that write the specifications. The purpose of the test suite is to ensure the specification is complete and has two conforming implementations, the requirement for a W3C specification to become a Recommendation.

Examples of W3C conformance test suites:

The HTML5 Test Suite is particularly noteworthy given the excitement around HTML5 and we’ll blog more about that next week.

Many different types of tests exist beyond those created by the W3C and take various approaches. You’ve probably seen many of these already. While not an exhaustive list, here are a few examples with descriptions of what they test and how.

Feature Existence

Some tests check for just the existence of features. Examples include The HTML5 Test, “When can I use…,” and findmebyIP. They perform a basic feature check, but don’t test the depth of an implementation or how well the feature’s behavior matches the standard. Thee tests rarely reflect the current status of a specification. For example, in the list above, only “When can I use…” lets you filter the view to only features with a chosen W3C specification status.

Some tests in this category compute a total score, though there is no consistent approach to calculating one. While a single score is easy to communicate, it doesn’t help people understand what the score means. When reading these tests, it’s important to see which features are and are not included and to what depth, rather than depending only on a final score.

Depth Testing

These tests attempt to verify more than whether a feature exists, and typically focus on a single specification. They’re usually developed by various people outside the W3C. They can be useful when the W3C working group hasn’t finished their test suite or as additional testing, but there is no assurance they represent the entire specification or are otherwise complete.

Examples:

Compatibility Charts

These suites use test cases to create compatibility information so Web developers can understand the behavior of certain features across different browsers. Similar to the Feature Existence tests, they check support for a feature but typically test the implementation’s correctness to some degree. This is helpful when you need information on a specific element, property, or other feature. Like the previous group of tests, these don’t benefit from the working group’s review and evaluation, and it isn’t clear what they actually test.

Examples:

Other Often-cited Tests

Some often-cited tests are a mix of the kinds described above.

Acid3, for example, is unique in that it tests about 100 fragments of a dozen different technologies. Some parts check for the existence of a feature, some check for specific behavior of a feature, and others check for behaviors where browsers were known to differ when the test was created. Acid3 gives each test one point and computes a final score based on the number of tests passed. While people often communicate the score, it’s unclear what that score represents given the variety of tests included.

Browserscope is a collection of tests in many areas, ranging from security and network behavior to RichText and Selectors API. Many of the tests check if a feature exists, as in the tests for postMessage and JSON.parse. Other tests check for specific behavior. For example, the RichText section tests many small behaviors in different categories. Various groups participate in creating these tests, but there’s no assurance they represent an entire specification and they don’t benefit from a working group’s evaluation.

An Industry Goal

Allowing developers to have the same HTML, CSS, and JavaScript work the same way across many browsers is an industry goal that benefits the developers who build the Web and the people who browse it. Testing is a critical step towards achieving this goal, and that’s why we work closely with standards bodies to create comprehensive test suites so people can assess not just whether a browser supports a standard, but the quality and depth of that support.

—John Hrvatin, Lead Program Manager, Internet Explorer

Comments (30)

  1. Tests are good says:

    It's almost like some of the other guys are taking HTML5 pass/fail…

  2. Craig says:

    This all sounds good – but it is also a cop-out for IE not being able to meet these benchmarks.

    What does IE score on Geolocation tests? Last time I checked ALL versions of IE scored a big fat ZERO.

    What about HTML5 tests?  Does Microsoft plan to post their .innerHTML setter score against other browsers? Or are they afraid that would show that IE has FAILED to fix .innerHTML in IE for over 12 years!

    What about HTML5 video tests?  Microsoft has yet to get together with the other browser vendors to work on an Open format (free from licensing restrictions that TOTALLY CONFLICT WITH THE OPEN WEB), to work with the Open Web… so how can we run video tests across ALL browsers?  As discussed here multiple times… H.264 is NOT a valid format we can use for the Open Web, so please stop wasting our time talking about it and get on with picking a format that will work!

    How is IE scoring on HTML Web Forms 2.0? – Last time I checked IE was scoring a big fat ZERO on this too.

    Finally on a slightly related note:  Well after IE8 was released MSFT blogged on here about the massive flowchart that determined rendering modes for IE8 (standards, quirks, real-standards, etc.).  Since IE9 adds a bunch of new levels to this matrix – can we get an update to this flowchart diagram to help us comprehend the dozens of possible rendering paths (well, not just rendering… script execution, API availability, and API correctness too).

    Isn't it time for another beta release of IE9? There's so much still that needs fixing – it would be good to see how much has been resolved that we can depend on before the RC comes out – that basically locks us in for zero-features/fixes before RTM.

  3. IE _sucks says:

    IE9 sucks in The HTML5 Test !

  4. Neil Dunensach says:

    "Allowing developers to have the same HTML, CSS, and JavaScript work the same way across many browsers is an industry goal"

    If I recall, that was the goal when the Web was first conceived.  Write once, run anywhere – remember?  Microsoft – and Netscape but to a lesser degree – are responsible for the HTML fragmentation by using proprietary tags and language when it suited them.  Now you're trying to fix it, fine, but don't make it sound like you're reshaping the world – all you're doing is trying to close the stable door while trying to push the horse back in the barn at the same time.

  5. Andrew says:

    Microsoft doesn't believe in fixing rendering issues with patches, only security. I find this extremely daunting. We have to wait a whole year or more for another IE and the fragmentation of browsers continues yet again. Firefox and Chrome teams both encourage updating IMMEDIATELY and they support developers creating extensions to ensure compatibility.

  6. Anonymous says:

    @Craig

    Video rendering is actually IE9's strong point. Part of the reason Chrome removed h.264 support is that they don't want to craft a h.264 decoder that can compete with IE9, since it is much faster, being able to play video more smoothly and using less system resources than chrome could.

    @Neil Dunensach

    That wasn't the goal at all. If it was, html and technologies would have stagnated. Html was established to be a markup language which could easily to extended and interact with new technologies. The reason why html was able to supplant gopher as the definitive protocol of the internet was because the Mosaic browser introduced a then proprietary element, the <img> tag which made that browser and html the killer app of the time. Without fragmentation, HTML would be dead.

  7. wwhat? says:

    Andrew,

    Near each month Microsoft releases a cumulative patch to IE and it usually doesn't only fix security issues, it always comes with several non-security related fixes: you can check their descriptions in the unique KB article of each cumulative patch.

    But I agree that Microsoft should make this information more accessible to the public that keeps believing that Microsoft only fixes security issues between each IE first number version change.

  8. Tim says:

    @wwhat although the patches include some non security items they NEVER fix rendering bugs in IE nor do they fix bad, incorrect or incomplete implementations of JavaScript objects/properties/methods.  On one hand this sucks horribly as fixes will take at least a year to make it into IE… On the other hand we do appreciate a moment of stability in IE so we can catch up on all the required fixes we need to do to accommodate IEs lack of standards.

  9. YANG Xudong says:

    Oh God. You would fare much better if you simply focused on improving your damn browser instead of posting this irresponsible crap. Does it hurt _so_ much to not talk nonsense and just write code?

  10. Rob^_^ says:

    Google does it again…

    code.google.com/…/Welcome trips Compatibility View in IE9 Beta…(see Doctype link above in your post).

    Appears that they (Big G) don't bother to test in IE (any version)

    <meta content=IE=edge,chrome=1 http-equiv=X-UA-Compatible>

  11. David Bruant says:

    You forgot the ecmascript test suite http://test262.ecmascript.org/

    Cheers,

  12. Leister says:

    @Anonymous – IE9's HTML5 video will be the defining point of the downfall of the Open Web.  Microsoft will be squarely to blame and developers will find a new level of "low" respect for IE.

    The RC is coming out today – I seriously hope that Microsoft is seriously re-considering just how foolish their actions are right now and that the future of the Open Web and HTML5 rides on them correcting this mistake.

  13. Jane says:

    @Leister – are you refering to Microsoft not implementing a Google owned proprietary video codec that is not in any way part of the HTML5 standard, or in any way or form even a standard?

    blogs.msdn.com/…/html5-and-web-video-questions-for-the-industry-from-the-community.aspx

  14. YANG Xudong says:

    @Jane: There is not one "Google owned proprietary video codec" in this world, not one that I know of. When you make a statement and still want to look good, please provide proof. No one likes someone who talks about things that are *downright wrong*. See en.wikipedia.org/…/WebM for a start.

    Also, be as irresponsible as you want when posting a comment, but for the sake of common sense, do not quote an MSDN article when arguing for Microsoft. It's the most unconvincing way of doing citations there is.

  15. Alexis Deveria says:

    Your statement about the "When can I use" site using "Feature Existence" to test support is incorrect. All support is based on tests done by humans, though they are indeed not Standards Compliance tests.

    For example, based on automated tests contenteditable is supported on most mobile browsers, but for practical purposes it just doesn't work, and the "When can I use" site reflects this correctly.

    Thus it would be much more correct to list the site under "Compatibility Charts". Thank you.

  16. Jane says:

    @YANG: How can Google determine the spec and licensing of VP8 if they don't own it? The spec even point to Google source code as the defining standard if in conflict with published bitstream spec.(!) I know they have open sourced it and that it is free (given no patent backlash), that doesn't make it a standard, or out of Google control. It is not adopted or controlled by any standard organization.

    The link you obviously didn't bother to read wasn't to a MSDN article, but to a user comment.

  17. mentas says:

    #1 Sometimes this website (IE blog) break on IE9

    #2 HTML5 sites like http://www.apple.com broken on IE9 pre-RC

    #3 IE9 still missing some HTML5 features like forms and CSS3 features like transitions, text shadow, box layout, …

  18. Ben says:

    Please do something about the lack of support for many css3 and HTML5 features (transitions, text shadows, animations, html5 form elements). You're Microsoft, you have an absurd amount of money and resources, how come your browser is not even able to be compared to modern browsers like chrome (or any webkit browser), opera, FF4 (and yes, I do not consider IE9 "modern")?  Not to mention the many, many other quirks/weird rendering issues and lack of feature support for javascript. It's sort of pathetic and it sucks as a front-end developer to make the decision between making sites that aren't nearly as good as they could be or implementing a second version for IE's terrible lack of support and rendering issues. The web could be a better place and move forward at a tremendously faster pace if we didn't have to sit around and wait for your browsers to catch up.

    It's sad

  19. The Penguin says:

    And IE's share continues to slide, based on user dissatisfaction and the ongoing failure of Microsoft to understand that, in the real world, you cannot just continue inventing new "standards" by implementing them in a browser that only works on an operating system that will always be badly-broken until rewritten from scratch… which of course, due to the commercial need for backwards-compatibility, will never happen.

                                    There was an error saving your comment. There was an error processing the request.

    And in a delicious slice of irony, I had to fire up a Windows box and run Internet Exploder just to post this comment, as this site generated the above error when I tried to post it from Firefox… :))

  20. Gareth says:

    What I want to know is … how do you know what I may or may not want to use in a web application in the future? Web developers would love to use offline storage but don't because one of the major browsers doesn't support it. And it still won't so oh well, a great feature I would love to use but i can't.

    Microsoft isn't so much trying to support what developers WANT to use but forcing developers to use what Microsoft SUPPORTS. Big difference and one I don't like being shackled with.

  21. Anonymous says:

    if ( IE ) return false;

  22. Tim says:

    Microsoft, you're thankfully slowly becoming more and more irrelevant. Why don't you, instead of keeping with your old tactics of attempting wherever possible to force people to things your way, instead embrace open standards? Web developers will thank you. Users will thank you, though they won't know they are – they'll like the new functionality though. And your own web-team will thank you too.

    I can't even imagine the scowls the Microsoft WebDev team give to the IE team as they walk past each other in the corridor.

    Also, even better – why not use Webkit in IE? Maybe us developers might laugh at you a little, but the general public wouldn't know the difference (besides speed and new functionality) and hey – we'd stop laughing after a bit :)

  23. Tim says:

    Microsoft, you're thankfully slowly becoming more and more irrelevant. Why don't you, instead of keeping with your old tactics of attempting wherever possible to force people to things your way, instead embrace open standards? Web developers will thank you. Users will thank you, though they won't know they are – they'll like the new functionality though. And your own web-team will thank you too.

    I can't even imagine the scowls the Microsoft WebDev team give to the IE team as they walk past each other in the corridor.

    Also, even better – why not use Webkit in IE? Maybe us developers might laugh at you a little, but the general public wouldn't know the difference (besides speed and new functionality) and hey – we'd stop laughing after a bit :)

  24. Tim says:

    (Apologies for double post – now triple post – I did refresh a few times to check that my first posting attempt which hung had gone through and it hadn't. Though apparently now it has)

  25. ben says:

    Yeah, Tim, it's because teh site is written in .NET, which aside from IE being the worst browser, .NET is the worst thing to ever happen to web development. Yes, Microsoft has done it twice, congrats!

  26. Mladen says:

    Everything you tell us is known as:

    "FUD":

    en.wikipedia.org/…/Fear,_Uncertainty_and_Doubt

    and "Embrace, extend and extinguish":

    en.wikipedia.org/…/Embrace,_extend_and_extinguish

    Microsofts tactics and your as an MS employee is exactly what is described at wikipedia.

    And what is MS panning for the future? The same. See "How Microsoft plans to market against the iPad":

    http://www.zdnet.com/…/8513

    The same is seen here:

    "Ballmer was speaking at a Power Developers event in Sydney, Australia, when someone asked the following question:

    Why is IE still relevant and why is it worth spending money on rendering engines when there are open source ones available that can respond to changes in Web standards faster?"

    "That's cheeky, but a good question, but cheeky"…"Open source is interesting, Apple has embraced Webkit and we may look at that, but we will continue to build extensions for IE 8." – Steve Ballmer

    Due to this tactics and the ignorance of MS IE has lost a lot of market share:

    http://www.netmarketshare.com/browser-market-share.aspx

    and

    gs.statcounter.com

    And MS lies at:

    http://www.microsoft.com/…/default.aspx

    Because of this I wrote a jQuery plugin to show the people why they should avoid using IE:

    github.com/…/ieSucks

    Go to the demo and click on the "Why?" button to see the reasons. Every reason has a link where you can read the full story.

    So John, prove me that I'm wrong!

    My advice for Microsoft is:

    – integrate Webkit as the render engine

    – put the features you think they should be integrated inside

    – wrap the whole program in a nice UI window

    – compile it as a standalone application (no use of the registry, no use of DLL's)

    Lijep pozdrav,

    Mladen

  27. Sal says:

    what this guy said, word for word ——^^^^^^