Test the Web Forward

The quality and correctness of different browsers’ HTML5 engines continue to vary widely. We continue to contribute to the test suites under development at W3C to further the goal of web platform interoperability and same markup. In total, we have submitted 7573 tests that you can view at the IE Test Center as well. As different browsers improve their same-markup support, we can all realize the promise of HTML5.

The title of this post refers to the week-end event hosted by Adobe on June 15 and 16 at their San Francisco office. Dozens of volunteers joined W3C experts and members from Adobe, Google, Mozilla, Apple, HP, and Microsoft to learn about Web standards testing, how to write CSS and SVG tests, how to file good bugs, as well as the tools available for test suite management.

The meeting then turned into a test ‘hack-a-thon’ fueled by free drinks and food throughout the day. Volunteers spent most of their Saturday writing new test cases for the CSS OM, Transforms, Backgrounds & Borders, Exclusions, SVG, and other modules. Participants were then nominated for several prizes.

Testing Web Standards

Adobe’s Alan Stearns introduced the participants to the general principles of W3C testing and the role of testing in moving specifications forward. In particular, establishing browsers’ individual pass rate for a given specification is not a goal of W3C test suites. In order for a specification to become a W3C Recommendation the Working Group must prove it can be implemented. In practice this means:

  • Creating a test case for each requirement in the specification (these are known as normative statements)
  • Verifying that at least two separate implementations pass each test

Note the difference between ‘at least two browsers must pass the entire test suite’ and ‘at least two browsers must pass each test in the test suite’. Browser testers usually describe this phase as ‘testing the spec’.

But an important side-effect of this testing process is to establish a common interoperable baseline that all browsers can develop and test against. Test suites help find bugs across all browsers and can sometimes identify issues in the spec.

Writing CSS and SVG Tests

There are three different types of tests:

  • Stand-alone tests typically rely on visual verification: if a failure condition occurs, red content will show.
  • Reference tests compare a test against a visual reference that has no dependency on the feature being tested. Note that the test includes a link to the reference test against which is should be compared.
  • CSS Object Model tests depend on a JavaScript test harness; they verify that the object model reflects what static style sheets specify. For instance, this CSS media query test.

W3C’s Doug Schepers covered SVG testing while Adobe’s Rebecca Hauck and Jacob Goldstein provided a test writing tutorial. Peter Linss, CSS Working Group co-chair, offered a deep dive on the CSS testing framework including the test suite build system and management tools such as Shepherd.

Filing Good Bugs

Mozilla’s Elika Etemad then gave attendees advice on what makes a good browser bug report:

  • The issue is specific and reproducible
  • The build and platform are identified
  • You have looked for duplicates
  • It includes steps to reproduce the problem
  • The expected and actual results are described
  • If possible, the issue has been reduced i.e. all HTML, JavaScript and CSS that is not necessary to reproduce the problem has been eliminated from the problem page and the remainder attached to the bug.

Building a Test Suite

Building a test suite is a significant investment. One of the reasons it took a long time for CSS2.1 to reach the Recommendation stage was the size of the specification and the underlying number of requirements to test. The latest version of the test suite contains 9,422 tests.

Microsoft contributed over 7,000 of those tests, and we are continually contributing more tests for other standard specifications.

In IE10, we have added support for a long list of new standard features across CSS, HTML, SVG and the DOM. We have published some of our testcases for these new features on our IE Test Center. We will be submitting more, notably around those features recently unprefixed in the IE10 Release Preview.

How You Can Help

We are excited to be part of the community working towards a more interoperable web. If you want to help move the Web forward, you too can help to drive interoperability higher. You can learn how to contribute tests, or review existing tests. More information for those is available on the CSS WG wiki as well as on the event page.

We will keep you posted on future events.

—Sylvain Galineau, Program Manager, Internet Explorer and
—John Jansen, Test Lead, Internet Explorer