Test the Web Forward


The quality and correctness of different browsers’ HTML5 engines continue to vary widely. We continue to contribute to the test suites under development at W3C to further the goal of web platform interoperability and same markup. In total, we have submitted 7573 tests that you can view at the IE Test Center as well. As different browsers improve their same-markup support, we can all realize the promise of HTML5.

The title of this post refers to the week-end event hosted by Adobe on June 15 and 16 at their San Francisco office. Dozens of volunteers joined W3C experts and members from Adobe, Google, Mozilla, Apple, HP, and Microsoft to learn about Web standards testing, how to write CSS and SVG tests, how to file good bugs, as well as the tools available for test suite management.

The meeting then turned into a test ‘hack-a-thon’ fueled by free drinks and food throughout the day. Volunteers spent most of their Saturday writing new test cases for the CSS OM, Transforms, Backgrounds & Borders, Exclusions, SVG, and other modules. Participants were then nominated for several prizes.

Testing Web Standards

Adobe’s Alan Stearns introduced the participants to the general principles of W3C testing and the role of testing in moving specifications forward. In particular, establishing browsers’ individual pass rate for a given specification is not a goal of W3C test suites. In order for a specification to become a W3C Recommendation the Working Group must prove it can be implemented. In practice this means:

  • Creating a test case for each requirement in the specification (these are known as normative statements)
  • Verifying that at least two separate implementations pass each test

Note the difference between ‘at least two browsers must pass the entire test suite’ and ‘at least two browsers must pass each test in the test suite’. Browser testers usually describe this phase as ‘testing the spec’.

But an important side-effect of this testing process is to establish a common interoperable baseline that all browsers can develop and test against. Test suites help find bugs across all browsers and can sometimes identify issues in the spec.

Writing CSS and SVG Tests

There are three different types of tests:

  • Stand-alone tests typically rely on visual verification: if a failure condition occurs, red content will show.
  • Reference tests compare a test against a visual reference that has no dependency on the feature being tested. Note that the test includes a link to the reference test against which is should be compared.
  • CSS Object Model tests depend on a JavaScript test harness; they verify that the object model reflects what static style sheets specify. For instance, this CSS media query test.

W3C’s Doug Schepers covered SVG testing while Adobe’s Rebecca Hauck and Jacob Goldstein provided a test writing tutorial. Peter Linss, CSS Working Group co-chair, offered a deep dive on the CSS testing framework including the test suite build system and management tools such as Shepherd.

Filing Good Bugs

Mozilla’s Elika Etemad then gave attendees advice on what makes a good browser bug report:

  • The issue is specific and reproducible
  • The build and platform are identified
  • You have looked for duplicates
  • It includes steps to reproduce the problem
  • The expected and actual results are described
  • If possible, the issue has been reduced i.e. all HTML, JavaScript and CSS that is not necessary to reproduce the problem has been eliminated from the problem page and the remainder attached to the bug.

Building a Test Suite

Building a test suite is a significant investment. One of the reasons it took a long time for CSS2.1 to reach the Recommendation stage was the size of the specification and the underlying number of requirements to test. The latest version of the test suite contains 9,422 tests.

Microsoft contributed over 7,000 of those tests, and we are continually contributing more tests for other standard specifications.

In IE10, we have added support for a long list of new standard features across CSS, HTML, SVG and the DOM. We have published some of our testcases for these new features on our IE Test Center. We will be submitting more, notably around those features recently unprefixed in the IE10 Release Preview.

How You Can Help

We are excited to be part of the community working towards a more interoperable web. If you want to help move the Web forward, you too can help to drive interoperability higher. You can learn how to contribute tests, or review existing tests. More information for those is available on the CSS WG wiki as well as on the event page.

We will keep you posted on future events.

—Sylvain Galineau, Program Manager, Internet Explorer and
John Jansen, Test Lead, Internet Explorer

Comments (52)

  1. Anonymous says:

    The Standards compliance is great. Please bring the following shortcut to Internet Explorer 10 and Windows Explorer in Windows 8 for the love of keyboard mouse users:

    "While holding the Ctrl button, if you press browser's back or forward button it should open the corresponding (previous or next) page in new tab. This behavior is present in Firefox, Chrome and Safari."

    Please fix this bug as well: http://www.freewebs.com/…/209-print-preview-input.html

  2. Anonymous says:

    HP Pulls out of supporting Windows 8 tablet due to ARM architecture and the fact that it won't run any existing Windows applications (the only feature that windows tablets have of value)

    "The robust and established ecosystem of x86 applications provides the best customer experience at this time and in the immediate future."

    http://www.bloomberg.com/…/hewlett-packard-will-shun-arm-amid-debut-of-microsoft-windows-8.html

    Combine this with the lack of full flash support in Metro IE and you have the beginnings of a failed OS/Device launch.

    Time to talk to the community and listen to what they say!

  3. Anonymous says:

    Ingin membuka situs berkeluarga

  4. Anonymous says:

    Im flm the porno

  5. Anonymous says:

    A windows 7 release for IE-10 might speed up testing the latest features.

  6. Anonymous says:

    It looks like Internet Explorer improved a lot and sometimes I think  I should start using it, but when I try I miss a lot of things, my bookmarks are not in sync between computers, must install many additional software to do that, and all the small little things chrome provides are just missing.

    the developers tools are also much behind Chrome, "inspect element" for a software developer, just does magic

    upgrade, Chrome is updates every 8 weeks or 6 forgot, new API is a there every month, things become a bit faster here and there every month, that is something IE is missing.

  7. Anonymous says:

    Lets be serious… no real testing of this new IE10 browser will occur until it is available to download for our *****EXISTING***** computers which all run Windows 7.

    If you want to release IE10 without full testing go for it, but if you were smart you would make an IE10 release that we can test on our local machines – because right now I can't test anything in IE10.

  8. Anonymous says:

    Just posting here to let you know there are many unanswered questions on the flash post(2 posts ago) that need answering ASAP.

    I do have sites that I expect to work in Metro IE but were not sure if we have enough time for the 2 month review cycle before Win 8 launches!

    More importantly none of us can submit our requests because our content is private and requires a login to view.

    Finally as we all expect this process to be scrapped we don't plan to make any effort that will likely be pointless when this CV list thing is removed.

    Please update the status of Flash in Metro IE.  Is it just going to work? Or are there still a few PMs back there that still don't see all the problems with this plan?

    E.g. Do we need to start working on our graphics now? "this site does not work in IE10 – please use Chrome, Safari, Firefox, Opera for full flash compatibility"

  9. Anonymous says:

    "The quality and correctness of different browsers’ HTML5 engines continue to vary widely" is just such a weird way to frame the whole thing. You are, in effect, trying to get other vendors to step up by saying their engines suck, which is weird diplomacy. When other vendors write, there's a little cattiness and benchmark one-upsmanship (though less today than in the past), but I never lose the sense they're a community working together towards a better web for everybody.

    Microsoft's public posture is essentially that it's the only real browser vendor, standing above all others, and..that's weird. Especially given the quality of the actual work Microsoft's doing, I'd expect to hear more professionalism and optimism, less chest-thumping. It's not my work, I guess.

    The single biggest step Microsoft could take for Web interop would be to get an HTML5 rendering engine on WinXP and push it out as an update by default to the 25% of people still use XP. It could probably only have software rendering. It would probably just be a new MSHTML in the IE8 shell. Some users would refuse or never get the update. It would go beyond Microsoft's promises for support of old OSes and would be expensive to produce. But it would greatly speed up the pace at which Web developers could start relying on HTML5. (And one-off products happen: MS delivered WinFLP to help some corporate customers upgrade old machines from 9x, and that was relatively few seats compared to the number of people out there using XP now.)

  10. Anonymous says:

    Randell, thanks. My thoughts are similar, just didn't take my time to write them.

  11. Anonymous says:

    @Randall

    Today's Windows XP users do not upgrade. Period.

    Microsoft has extended, then extended, then extended the support for XP long enough. People running IE6 on their machines won't upgrade to IE7, then won't upgrade to Firefox, Chrome, or Safari even when prodded with a hot poker. They are the people who are afraid to install security updates because it might break something.

    If Microsoft would make available an IE10 for XP, I can't see many XP users upgrading. The only thing that would make them migrate is when the web breaks in a serious way for them. And then there are more than enough alternatives to move them off IE6. In a way, the new browsers have solved the huge problem for Microsoft regarding backward compatibility.

    But with or without IE10 for XP, those users just won't upgrade.

  12. @GT "the developers tools are also much behind Chrome, "inspect element" for a software developer, just does magic"

    You have that on IE, it's just a bit harder to access. Open developer tools, click on the arrow icon, click on the item to inspect.

    Just two steps if you already have developer tools open and pinned.

    It's also two steps on Chrome: First step – right click, second step – click inspect.

  13. @Microsoft, please fix the blog comment system so that sign-ins and posting works on the first try and comments are not lost randomly.

  14. Anonymous says:

    @alvatrus

    Not at all.  There remains a hard core of Windows XP users on IE6; mostly these are corporate/government.  In web usage figures they vary in number from site to site.  The vast majority of IE-on-XP users, however, are using Internet Explorer 8, and got there via IE7.  (The notable exception here is China; they're upgrading direct from IE6 to IE8, or to other browsers specific to the Chinese market.)

    If IE9 was available on XP there is no reason to think users wouldn't be upgrading to it.  People are far happier about upgrading their browser than their operating system, especially if it gets pushed out as an automatic or recommended update.

  15. Anonymous says:

    I wonder if, due to the "upgrade ceiling" on Windows XP, Internet Explorer 8 will be considered the "new IE6" in a few years time.  (The other candidate is the default Android 2.3 browser.)

  16. Anonymous says:

    @Vlad NC your assessment of the IE dev tools is incorrect.  Ask any developer which toolset is best and IE will always be at the bottom of the heap.

    In chrome you are looking at the element you want to inspect you right click an choose inspect (2 clicks, plus a small scroll)

    In IE you need to use your other hand or release the mouse to press F12… Then you need to move your mouse all the way over to the find by click icon then you have to move back to where you came from and click the element you want (1 key press, 2 clicks, and two big scrolls)

    Not even remotely close to equal.

    Then of course you have the actual DOM tree and tools… Chrome's are LIVE IEs are not, the tree presentation in IE is unhelpful showing garbage nested nodes like "empty text element" IEs tools look and operate with the same ugliness of all 1996 apps with next to no usability enhancements.

    IE has come a long way with their browser but their dev tools are still atrocious an definately the worst of all tools out there.

    If I was forced to develop in IE all day long… I would quit on the spot – it is a horrible experience.

    Neil

    PS this comment form is still broken! Seriously how many Microsoft engineers does is take to comment out the broken Legacy ASP postback and add a real submit button?!?!

  17. Anonymous says:

    @Neil –

    Also, its speed. It is so horribly slow.

    – If you have twenty log messages at the start up of the page, it starts to slowly (very slowly) duplicate them on page reload until it finishes re-initializing and then gets back to normal (also slow, of course).

    – Use the uncompressed (or the prettify option) jQuery, debug code that calls a jQuery method and step into the jQuery code. See where that gets you. 😉

    So bad.

  18. Anonymous says:

    We're still waiting for a response to the proposed CV censorship list dilemma.

    We've pointed out flaw after flaw after flaw with the plan but MSFT has failed to respond to the problems with any sort of solution.

    As a user I fail to see any reason why my desktop experience in Metro should be compromised because Microsoft is trying to use the same OS for their tablets.

    Oh and FOR THE LOVE OF PETE can someone with talent be in charge of the IE Blog… And can they fix the $?!&@!ing comment form on this blog!

  19. Anonymous says:

    Please, seriously, please, release a new Preview/Developer Preview/Beta/Technology Preview (whatever you wan to call it) of IE 10 for Windows 7. Most people do not have the time or ability to work with Windows 8 yet. If you want lots of real world testing, this is the only way you will get it.

  20. Anonymous says:

    It seems to me like the Windows 7 version is being held back solely because one or two features rely on Windows 8 exclusive code, such as the spell check.  I think I speak for everyone when I saw we would be fine with a preview version that did not have those features, though I guess its too late now because the Windows 7 release candidate or whatever it will be called is probably only a couple months away.

  21. Anonymous says:

    I don't think Windows XP will be relevant for web designers much longer, anyone willing to stick with a 10+ year old OS is probably using a non IE browser or is someone who probably wont be visiting your website.

  22. Anonymous says:

    @@Dave H

    I'd like to think so.  Let's all look at our web logs in 2014 and see where we're at!  Obviously the pressures on development differ between different types of website.  An informational government site funded by taxpayers will have different criteria on browser support to a funky online gaming site.  An ecommerce site will gauge additional development costs to lost custom, etc.

  23. Anonymous says:

    Microsofts Tuesday July 3rd Agenda

    1.) Fix the IE Blog Comment system

    2.) Post an update about full flash support in Metro without the censorship list

    3.) Windows 7 version of IE10 beta so that developers can actually test IE10 – something they can't easily do now

    4.) Publish a post regarding Metro support of other browsers – will we be able to upgrade IE10 Metro to a Metro version of Chrome or Firefox?! Especially if that is the only way businesses and users can get complete flash support

    All of this should be accomplish-able by lunchtime.

  24. Anonymous says:

    Microsofts Wednesday July 4th Agenda

    1.) Fix the IE Blog Comment system

    2.) Post an update about full flash support in Metro without the censorship list

    3.) Windows 7 version of IE10 beta so that developers can actually test IE10 – something they can't easily do now

    4.) Publish a post regarding Metro support of other browsers – will we be able to upgrade IE10 Metro to a Metro version of Chrome or Firefox?! Especially if that is the only way businesses and users can get complete flash support

    All of this should be accomplish-able by lunchtime.

  25. Anonymous says:

    an article about Microsoft's Lost Decade

    http://www.vanityfair.com/…/microsoft-downfall-emails-steve-ballmer

    explains a lot about Microsoft's mismanagement and why the company has a very hard time both keeping employes happy as well as innovating and creating good products on time.  it doesn't specifically mention the years of failed Internet Explorer browsers and the OS's known as Windows ME, and Windows Vista but it does elude to the expected downfall of Windows 8 – another case where employees that complained about this mixed OS (tablet/desktop) was a bad idea but where shot down by management that didn't want to take the time to fix it.

  26. Anonymous says:

    Please bring data-uri for HTML, CSS, Javascript and other plain text in Internet Explorer connect.microsoft.com/…/data-uri-seems-to-require-base64-encoding

  27. Anonymous says:

    I'll just put this here: WebGL.

  28. Anonymous says:

    The reason microsoft has a dead fish as the main desktop theme in Windows 8 is because IE is DOA

    No WebGl, Low HTML 5 scores, more web fragmentation by not updating all windows versions.

    True Story!

  29. Anonymous says:

    Microsofts Thursday July 5th Agenda

    1.) Fix the IE Blog Comment system

    2.) Post an update about full flash support in Metro without the censorship list

    3.) Windows 7 version of IE10 beta so that developers can actually test IE10 – something they can't easily do now

    4.) Publish a post regarding Metro support of other browsers – will we be able to upgrade IE10 Metro to a Metro version of Chrome or Firefox?! Especially if that is the only way businesses and users can get complete flash support

    All of this should be accomplish-able by lunchtime.

  30. Anonymous says:

    @ Microsofts Thursday July 5th Agenda

    The IE Blog does not need fixing! It includes an intelligence test that you have obviously failed!

  31. Anonymous says:

    I HOPE IE10 IS MUCH LIGHTER THEN IE9 OR ELSE I AM GOING TO USE FIREFOX OR GOOGLECHROOME.

    I WANT A LIGHT IE10 AND NOT SLOW AND HEAVY

  32. Anonymous says:

    That article is a very interesting read, but why didn't you talked about the HTML5 WebGL tests you've written?

    I'm nervous that HTML5 WebGL support in IE10 might not be as good as in all the other browsers.

    And that would be a farce for a browser that pretends to adopt web standards.

  33. Anonymous says:

    Uh IE9 is really fast…

  34. Anonymous says:

    Microsofts Friday July 6th Agenda

    1.) Fix the IE Blog Comment system

    2.) Post an update about full flash support in Metro without the censorship list

    3.) Windows 7 version of IE10 beta so that developers can actually test IE10 – something they can't easily do now

    4.) Publish a post regarding Metro support of other browsers – will we be able to upgrade IE10 Metro to a Metro version of Chrome or Firefox?! Especially if that is the only way businesses and users can get complete flash support

    All of this should be accomplish-able by lunchtime.

  35. Anonymous says:

    @ie team

    why IE9 for one webpage downloading become freeze ?

    firefox do this work very easily in background ! freeze all tab pages only for one page download ! 🙁

  36. Anonymous says:

    I had a comment and submitted it but alas once again the comment form ate it and this time I forgot to save it first – damn!

    please fix this comment form!

  37. Anonymous says:

    Can we get someone from Microsoft to at least acknowledge that they know the comment form is broken and they are working on it?!

    It's ***EXTREMELY ANOYING*** that Microsoft ignores all our complaints about it… Fine ignore our comments about the stupid features you add to IE (like fake flash support) … But have the decency to address the issues with the broken comment form – your one and only forum for IE feedback!!!

  38. Anonymous says:

    That's Microsoft for you! They haven't changed one bit and won't till people like large stock holders tell them that there unhappy with the

             bull that there trying to pull on the public and they will not do that till sales drop! So stop BUYING! Because they will keep ignoring

             us till there forced by economics to change

  39. Anonymous says:

    Keys are still sticking in Flash games under IE8, IE9 and now IE10.

    http://www.kongregate.com/…/67250-fix-for-keys-sticking-in-games-in-internet-explorer

    Do you want you users to disable Protected Mode just to play Flash games? Do you want you users move to Chrome?

    Web standards are good in IE10. But you forget that your primary customers are users, not developers. Developers won't look at IE if the users leave.

  40. Anonymous says:

    @Who-Cares#,

    What are you a child? What kind of idiotic analogies you are making? People hated MS because developers make the people believe that it doesnt comply with the standards and they need to write *extra* bits of code to make the UI consistent. Now they are complying with standards and developers wont have to write those extra bits of code.. so technically the hate factor is mitigated.. but WTF are you talking about… the stock market? Its a goddamed web-browser for free .. you dont need to BUY it assshoole!

  41. Anonymous says:

    Microsofts Tuesday July 10th Agenda

    1.) Fix the IE Blog Comment system

    2.) Post an update about full flash support in Metro without the censorship list

    3.) Windows 7 version of IE10 beta so that developers can actually test IE10 – something they can't easily do now

    4.) Publish a post regarding Metro support of other browsers – will we be able to upgrade IE10 Metro to a Metro version of Chrome or Firefox?! Especially if that is the only way businesses and users can get complete flash support

    All of this should be accomplish-able by lunchtime.

  42. Anonymous says:

    You typically need to buy four Windows licenses, because you need to test in four versions on Internet Explorer.

  43. Anonymous says:

    @@Zoya, last time I checked Windows 8 RP is free to download and use for 14 months. IE10 GA will come to Windows 7. Moreover, Windows XP, Windows Vista and Windows 7 users will get Windows 8 upgrade for only $39,99 (windowsteamblog.com/…/upgrade-to-windows-8-pro-for-39-99.aspx).

  44. Anonymous says:

    @Zoya

    That is not the argument.

    Some people can´t and will use Windows 8

  45. Anonymous says:

    When was Microsoft going to announce that not only were they crippling Adobe Flash from running in Metro IE but also many standard parts of JavaScript because MSFT was too lazy to implement them?

    Metro IE 10 Apps/Sites will not allow any user interaction using:

    alert();

    confirm();

    prompt();

    print();

    Bug reports here: webbugtrack.blogspot.com

    That means no printing, no prompted input, no confirmation, no alerts!

    How does MetroIE request my permission to share my location without a notification/dialog system?

    How does MetroIE request my permission to increase local storage space without a notification/dialog system?

    Were you going to tell long time IE-only developers that their dialogs will no longer work because you've disabled showMod[eless|al]Dialog(); from working?

    When were you going to tell us that onOffLine and onOnLine were no longer working? what about window.external? or window.clientInformation?

    Pinned Sites?, Behaviors?, Favorites? – all gone!

    webbugtrack.blogspot.com/…/microsoft-drops-support-for-features-in.html

    The only thing you told us was that ActiveX Controls were all gone except for Flash… which you then told us wouldn't really work in Metro IE 10 anyway because you were locking it down to a whitelist of sites that we can't submit to.

    Please tell me that Metro Mode (Windows 8) actually has some sort of notification system where messages can be presented (e.g. like an alert) – and if there (presumably) is one… why can't IE use it?

    Metro IE is turning out to be a fairly useless browser if you need to switch to desktop IE to actually DO ANYTHING! – yeesh!

  46. ieblog says:

    The previous comment is inaccurate. Metro style Internet Explorer 10 does support alert(), confirm(), print(), showModalDialog(), and showModelessDialog(). As noted above, it does not support prompt(). Metro style IE does have updated prompts when user permission is needed. It also supports pinning sites to the Windows 8 Start screen.

  47. Anonymous says:

    @Jens, what argument? This blog post is about IE10.. so we are NOT talking about people who will not be using IE10. IE10 will be available for Windows 7 and Windows 8. If you aren't planning to use Windows 7 or Windows 8 then you don't belong here. In conjunction to that, Windows XP and Vista users who are willing to upgrade will get Windows 8 for 40 bucks!

    @Who-Cares#,  What????? "They haven't changed one bit?" What kind of benchmarks can you provide to support your argument? Are you asserting technically standpoint or is just a regular whining? Your comment is stinkier than your English. You know that right?

  48. Anonymous says:

    @ieblog – please take some time to update MSDN Documentation then please – because your documentation clearly states that these methods are not available in Metro IE 10.

    Quote from MSDN on the confirm method:

    msdn.microsoft.com/…/ms536376%28v=vs.85%29.aspx

    Internet Explorer Developer Center > Docs > Internet Explorer API reference > HTML/XHTML Reference > Methods > confirm

    "This method is not supported for Metro style apps using JavaScript. "

    Seems pretty clear to me that these methods are no longer supported.  Which methods are removed then if the MSDN documentation is inaccurate?

  49. ieblog says:

    @Nolan: The only one that doesn't work on a test page I have here is prompt(). I'm told that's by design.

    I'm testing with Windows 8 Release Preview; perhaps the others didn't work in the Consumer Preview. Anyway, I'll inform MSDN about the need to correct the documentation. Thanks for calling it out.

  50. ieblog says:

    @Nolan: Let me amend my previous replies. The methods (except prompt()) are available in Metro style Internet Explorer but not Metro style apps. The MSDN documentation note "This method is not supported for Metro style apps using JavaScript" specifically refers to Metro style apps (those you build and install from the store), not Web sites viewed in Metro style Internet Explorer. Metro style apps have alternate means of interacting with the user that result in a more consistent Metro style experience.

  51. Anonymous says:

    @ieblog thanks for the clarification.  Needless to say this isn't very clear especially when it's in the IE developer documentation.

    May I suggest that it be rephrased to:

    "This method/property is not available in _compiled_ applications running in Windows Metro UI – however it is still available in IE10 running in Metro mode"

    Some clarity would have avoided all this.

    I do find it funny though that instead of fixing prompt(); to make it work properly Microsoft just decided to get rid of it – very classy.