Understanding Differences in Hardware Acceleration through Paintball


While most modern browsers now support hardware acceleration, the quality of implementations
varies greatly between browsers. Hardware acceleration is more than just a checkmark.
How a browser chooses to enable hardware acceleration has a direct impact on the performance
and user experience of Web applications.

To highlight the quality and performance differences in hardware acceleration, let’s look closely at the
Paintball
test drive, released with
IE 10 Platform Preview 1
. Paintball uses HTML5 Canvas to simulate shooting
paintballs against a wall. Paintball demonstrates how quickly images compose an entire scene.
Many other HTML5 Canvas performance examples on the web just measure how quickly the same image can be drawn on
the screen.

Paintball draws an entire scene as quickly as possible. At its end, you’ll see two numbers. The first is how long drawing
the entire scene took in seconds; the second is the number of paintballs fired per minute.
In this test, more paintballs in a shorter time is better.

Screen shot of Paintball demo rendered in IE10 Platform Preview 1 src="http://ie.microsoft.com/testdrive/ieblog/2011/Apr/26_UnderstandingDifferencesinHardwareAccelerationthroughPaintball_1.png" />
Screen shot of Paintball demo rendered in IE10 Platform Preview 1

How Paintball Works

We wanted Paintball to be a general purpose Web application for painting any phrase with any color of paintball. To this end, Paintball programmatically
generates paint splatters at runtime based on the specified RGB values. Programmatically
manipulating images at runtime is a common scenario for emerging HTML5 Web applications.

Paintball starts by drawing a solid rectangle which is the color of the paint splatter.

Rendering paint splatter effect - step 1

Paintball uses a sprite sheet to hold the frames of the splatter animation:

Paint splatter effect sprite strip

Paintball decides how large the resulting splatter should be, and draws a frame
from the sprite sheet using the destination-in composition mode. This causes
the pixels being drawn to mask the current contents of the canvas.

Rendering paint splatter effect - step 2

By generating the graphics on the client, Web developers are able to build and download fewer images. This is a great approach for programmatically changing the color of an image.

The paintballs are created in a canvas which is not visible on the screen, and then
drawn to the primary display canvas. Using a secondary, off-screen canvas to compose the effects improves performance quite a bit, in all browsers, by using composition operations
only while the animation is in progress. Once the animation has reached its final frame (paint fully
splattered), we can use a cached view of the secondary canvas when drawing the full scene.

Quality Differences

Paintball uses the same HTML5 markup for all browsers as specified by the HTML5
Canvas specification. Although all modern browsers now support these API’s, the quality
differences are extremely noticeable. Here’s a visual comparison of the rendering
differences:

Visual comparison of Paintball demo in Chrome 12, Firefox 4, IE 10 and Safari 5

You’ll notice the following differences:

  • Chrome 12 does not draw the paint splatters correctly because
    Chrome does not implement
    the HTML5 Canvas composition modes according to the
    HTML5 specification.
  • Firefox 4 renders the scene correctly.
  • Internet Explorer 9 and Internet Explorer 10 render the scene correctly.
  • Safari 5 shares the paint splatters rendering artifacts with Chrome. Additionally,
    Safari does not yet support WOFF fonts.

These differences capture the kinds of problems that developers face building sites that look and work consistently across browsers. Developers want to use the same markup
and experience interoperable results. While these bugs might seem minor (and shouldn’t be hard to fix), they represent the quality issues across browsers that make developers resort to
plug-ins or lower their expectations for sites.

Performance Differences

The performance differences between implementations are even more striking. For our testing we used the same mid-range configuration that
we
have
used throughout the last year. Here’s how quickly these modern browsers,
which all have some degree of hardware acceleration, compose the scene (on Windows 7):


Chrome 12 Firefox 4 IE 10 Safari 5
Seconds
(lower is better)
516.14 100.17 11.35 68.11
Paintballs per minute
(higher is better)
10.11 52.11 459.95 76.64

Graph showing time Paintball demo takes to complete in Chrome 12, Firefox 4, IE 10 and Safari 5
Graph showing time Paintball's fired per minute in Chrome 12, Firefox 4, IE 10 and Safari 5

What’s Happening on the Hardware?

Let’s look at detailed performance traces to see how the different browsers use the
CPU and GPU when animating a single paintball shooting and splattering onto the
screen. The below results were generated using the
Windows Performance Toolkit
.

With Chrome 12 you’ll notice that it takes 3.7 seconds to shoot a single paintball.
We can see that when the paintball first appears on the screen as a ball, there
are lots of updates with moderate GPU and CPU utilization. Chrome is able to maintain
almost real-time animation. Once the splatter effect (described above) starts to
render, we see GPU and CPU utilization spike for several seconds. During this time,
there are no screen updates, so the demo appears to freeze. Finally, the browser
catches up and the scene finishes rendering.

Performance chart of a single paintball being shot in Chrome 12. Shows CPU time, GPU time, and Visual Updates.

Firefox 4 handles this scene quite well in 0.550 seconds. CPU and GPU utilization
is fairly steady throughout the animation although the CPU activity does increase
as the splatter effect is rendered. Screen updates occur predictably, rendering
19 frames during the rendering of the scene.

Performance chart of a single paintball being shot in Firefox 4. Shows CPU time, GPU time, and Visual Updates.

IE 10 renders the scene quite smoothly, in the shortest amount of time (0.475 seconds)
and with the most frames. From the below graph, you can see how CPU and GPU utilization
are lower than with Firefox, and the screen updates occur more frequently and at
more regular intervals.

Performance chart of a single paintball being shot in IE 10. Shows CPU time, GPU time, and Visual Updates.

Performance in Safari 5 can be misleading from the CPU/GPU measurement perspective.
While the GPU utilization is lower, and that screen updates occur frequently at
regular intervals, nothing appears to be happening on the screen. That’s because
Safari is telling the GPU to paint the same image over and over again. Safari only
draws a few images to the screen and then jumps to the final frame without any animation
occurring. Safari feels hung and unresponsive during this time.

Performance chart of a single paintball being shot in Safari 5. Shows CPU time, GPU time, and Visual Updates.

The graphs above show a single paintball splatter – a small slice in the overall
work done to render the entire Paintball demo. While most browsers can handle a
single paintball well, many of the browsers have a difficult time scaling to dozens
of paintballs. For example, Firefox renders a single splatter fairly well (almost
as fast as IE 10). However, when faced with a higher load in the full demo experience,
performance in Firefox slips to about 20% of Internet Explorer 10.

Here’s the total CPU time, GPU time, and visual updates across browsers.

Comparison of CPU time consumed for a single paintball being fired in Chrome 12, Firefox 4, IE 10 and Safari 5.
Comparison of GPU time consumed for a single paintball being fired in Chrome 12, Firefox 4, IE 10 and Safari 5.
Comparison of Visual Updates during a single paintball being fired in Chrome 12, Firefox 4, IE 10 and Safari 5.

Summary

Browsers with full hardware acceleration and fully interoperable implemenations of HTML 5 features deliver a better experience for
Web developers and end users.

We hope this look into the internals of Paintball will help you better understand
the different quality and performance characteristics across browsers, particularly
in relation to HTML5 Canvas and composition modes. It’s an exciting time to be developing
graphically rich Web experiences!

—Seth McLaughlin, Program Manager, Internet Explorer Performance Team

Comments (30)

  1. Jon says:

    If you're comparing browser performance against IE10, why aren't you using Mozilla Aurora, instead of Firefox 4, as it contains a number of hardware acceleration related fixes?

    Somehow I doubt you're unaware of its existence, given the obvious attention you pay to what your competitors are doing, so why was it excluded? Perhaps because the results would not have been as favourable to you?

  2. Zkal says:

    @Jon: Most likely this blog post was written before Mozilla had released any Aurora builds since testing things like this probably takes a bit more time than just one day.

  3. Seth [MSFT] says:

    @Jon: Our goal was to compare the fastest available builds of each browser. In Firefox Aurora, we saw lower scores than Firefox 4 final (about ~164 seconds in FFAurora vs. ~100 seconds in FF4). This is why we used the FF4 numbers. Thanks!

  4. Another Jon says:

    Jon just got pwned…

  5. Roman says:

    Lol, Jon just got so pwned.

  6. Another Another Jon says:

    Creating a "benchmark" that favors your own specific implementation isn't hard.  That's why performance tests are usually performed by an impartial third party.

    Microsoft, please stop trolling.

    Or don't, actually.  Chrome and Firefox developers will soon enough make the tweaks necessary to run this test just as fast, and that's probably not a bad thing.

  7. Björn says:

    Firefox renders the scene differently but still correctly?

  8. Mike Ratcliffe says:

    Firefox is struggling with panning and zooming and this is what lets it down in the test … this is a weakness so now we can work on it and make it better.

    Top marks to the IE devs … more releases = more cakes ;o)

  9. Matthew says:

    Other browsers will make tweaks to run this fast?

    Ha! Right.

    As much as I loathe the continued presence of IE6 and even IE7. As much as I don't like IE8. As much as IE9 wasn't quite catch up in terms of features (IE10 may well be though), if there's one thing that I've not seen is anyone demonstrate a browser acceleration technology as rapid as Microsoft's.

    I think what they're saying about working as directly as possible with the hardware makes perfect sense. I don't think we'll ever see Chrome get as fast because of it's compatibility layering in the API and Chrome is my regular browser (make IE10 look pretty with tabs and I might be tempted to switch). Firefox has decided to jump on board the Direct2D API and on Windows this really shows. Firefox's hardware acceleration is way better than Chrome's, but Firefox is still behind Microsoft.

    I don't expect to see anything beat IE10, other browsers are too focussed on XP and other platform compatibility. A design approach has to compromise somewhere. IE9+ compromised with legacy OS compatibility.

    I know for sure that as a web dev, it's not IE that will be slowing my sites down in the future (in terms of graphics anyway).

    Maybe some dedicated linuxy-browser with a similar direct hardware API to Direct2D could cut the mustard. Give it 5 years or so, you never know.

  10. Jeffrey Gilbert says:

    @Jon just got his player card revoked.

    Better question for you guys at Microsoft. When will we see something like the requestAnimationFrame() so animations wont reduce the browsers speed when tabs are changed? Graphically intense sites shouldn't gobble up resources if they're not being actively viewed, right?

    Keep up the good work.

  11. Kevin Roast says:

    These benchmarks are (as mentioned above) a classic case of being targeted specifically for the strengths of a particularly browser implementation. You only need to run this * independent* HTML5 canvas benchmark to see: http://bit.ly/astbench – IE9 (and IE10) perform amazingly well on the pure bitmap tests (like above) but are very poor at shadows and sub-pixel primitive rendering i.e. lines and polygons – Chrome is much faster at those tests.

    It would be nice if Microsoft released tests that actually covered all the features people are using in canvas, not just the subset they know are well GPU accelerated.

  12. Roman says:

    @Kevin, you seem to be biased by definition against IE. Paintball uses the same markup, it should: 1. Run, 2. Run fast. The first failure indicates not supported standards. The latter indicates poor communication between different components.

    Microsoft repeatedly said that EVERYTHING is well GPU accelerated despite poor protests from Mozilla devs. Here, is clearly shown that MS in fact accelerates whereas Mozilla 'kinda accelerates'.

  13. Kurt Dane says:

    As with FishIE, I imagine Firefox would significantly outperform IE10 with a WebGL implementation (muizelaar.blogspot.com/…/drawing-sprites-canvas-2d-vs-webgl.html). I'd be interested to hear what the roadmap is for adding WebGL support to IE. And while we're talking about roadmaps, what's the schedule for Microsoft to join the WebM CCL (http://www.webm-ccl.org/)?

  14. Zkal says:

    @Another Another Jon: All of the browser makers use benchmarks that work best for their browsers. Kraken for Mozilla, V8 for Chrome for example. You could ask them to stop trolling too.

  15. Björn says:

    @Matthew: Unless GPU produces care more about Linux there won't be much going on.

    @KurtDame: I would guess that the performance numbers don't differ that much in WebGL (be it that they run on top of the same WebGL platform, e.g. OpenGL, ANGLE,  Mesa3D). On the other hand should MS really plan to implement WebGL(which I somehow doubt) then they would surely create their own WebGL-to-DirectX mapper out of their not-implemented-here attitude.

  16. meni says:

    Comparing this post to the canvas-vs-svg one makes me sad. I'll try to enunciate why, and sorry if it comes out confusing.

    What's your message to (your) developers? Is it: IE on windows is the fastest, so make apps to take advantage of that? Is it: Browser speed matters greatly to DEVELOPERS? Then all I have to say is, poor developers, they are being mistreated again. You are (yet again) not giving them the right tools to succeed in today's world. (akin to what Microsoft did to Silverlight developers, who found the rug pulled from under them)

    I really really don't care, as i really find this blog mostly irrelevant (!), but I think your message should be: we like standards, so should you. Make your sites cross-platform and cross-browsers compatible. Learn about standards from whichever source you can find: Yahoo Developer Network (Doug Crockford), Google I/O videos, Mozilla's site etc. Make sure your site works from all browsers/platforms.

    IETeam you should ask yourself, what has Microsoft/IE contribute to the web? is in the field of H/W acceleration?? Is it in the form of Win7 pinned sites??

    It seems to me that Microsoft found itself disrupted very badly, and is struggling to define itself and it's mission (other then to make money).

    com gmail meni.taub

  17. Stilgar says:

    yeah yeah those fancy browsers can paint stuff really fast. Now give me something I can use in my real job like the new HTML5 inputs. I would also like an option to make the text rendering the same as IE8 (or compatibility view for that matter).

  18. Roel says:

    The browser comparison is never a fair one because, to use the same markup everywhere all people in the world should have an updated browser. This Cycle is like 6 years for Internet Explorer (because some people still use IE 6 these days) and all the other browsers will be auto-updated in time. So Internet Explorer will allways slowdown the web, even when there BETA version has much better support of standerds, etc..

  19. Udy says:

    Can you guys compare WebM (VP8) with H264?

    In such a way that the video is high bitrate 1080p (at least 30Mbit) and is encoded in h264 (DXVA compliant) and equivalent WebM.

    The test would load the h264 video for IE9/10 and WebM for the competition.

    As h264 is hardware accelerated, playback of the video would be smooth and CPU-light.

    On the other hand, such quality WebM would bring any non-IE browser on its knees.

  20. @Udy says:

    Laughing Out Loud !!!  Great idea, btw.

    Can't phathom (insert spelling-checker here – I'm not a native English speaker :-( ) the avalanche of comments that would attract, though.

    And while we're at it, can I ask all the WebM affectionados for a small token of compensation for the $5000 equipment (cams, videos, software, etc.) I need to replace with "open" variants that support WebM instead/alongside H264?

  21. Aethec says:

    You gotta love open-source fanboys…when Microsoft are good at their own benchmarks, it's because they wrote them, but when Firefox is good at Kraken and other Mozilla benchmarks, it's because FF is so great…

    …by the way, please *don't* support WebGL. Write specs for a 3D Canvas API if needed, but don't support WebGL. Unless all other browsers start supporting -ms-filter.

  22. Randall says:

    Eh.  I love the product, I'd love other engines to be every bit as good at graphics, but I don't think catfights with other vendors on the official blog is either classy or useful.  Unsubscribed in my RSS reader, but I'll check the IE10 release notes when it comes out.  I'm told that everything  is merely activity, not progress, until it's "site-ready" anyway. ;)

  23. tobi says:

    IE team kicks ass.

  24. hAl says:

    @udy

    If you compare h.264 against WebM you should better try to do it on 720p and a low as reasonable bitrate as mobile devices are most HTML5 capable and they require low bitrates and lower resolutions than 1080p

    Using very high bitrates the quality difference will be significantly less.

    The quality difference in a codec will be visible most when you push it to the edge on lower bitrates. (which is what you want on the internet as low bitrates mean less traffic and less diskspace).

    High bitrates are nice but mostly for blu ray disc versions.

  25. Roman says:

    @hAl, my T-Mobile G2x is capable of 1080p playback. :)

  26. Udy says:

    @hardware Abstraction layer

    High bit-rate 1080p comparison is to emphasize the performance difference between VP8 and H264.

    Technology doesn't stand still – it is reasonable to expect high bit-rate HD +/-3D web video in the near future.

    Comparing whether WebM or h264 has better quality and performance for  +/-3D 1080p video is simply logical.

  27. zzz says:

    Q:Who cares about this "Paintball acceleration" if the IE9/10 text rendering is still more fuzzy than IE8?

    All the improvements in IE9/10 are completely marginalized by the fact that the time wise most used feature of a browser, reading text through it, took a step back to stone age.

    I haven't seen one line of acknowledging the seriousness of this issue from the D2D/IE teams – point it out if you have official announcement somewhere – yet between VS 2010 RC and RTM easily visible improvements were made to VS.

    Q: Why were those changes made to VS if there is nothing serious here to improve? And if there is, why is it not being acknowledged? Is the Strategy of IE team that people using VS move to use different browsers? Does Microsoft now chant the chants of Chinese LCD panel makers (because reports say the cheaper and the worse quality display the less bothering the fuzzy IE9 text rendering is)?

  28. florin says:

    Please consider UI performance in IE10, especially on low end ULV machines, where Chrome feels much faster. For example time tab openings, closings. Also, fix horizontal scrolling speed in zoomed in images (small increments atm) and also consider a vertically larger address bar, another advantage of Chrome once you get used to it. Also, the tiles in the home page sometimes pick up the 2nd or 3rd page of a blog.

  29. florin says:

    Also, if the net connection is bad (icon says ok but no packets get thru) then there is a lenghty delay on loading – 5-6 seconds. This happens on bad wifi access points and it appears to the user as an IE flaw, please fix.

  30. Björn says:

    @zzz: It makes less sense for code, ie. fixed-width font you rarely print, now for the web it makes a lot of sense like explained numerous times and demonstrated in some IE Testdrive demos.