Performance: What Common Benchmarks Measure


Many enthusiasts want a fast browser, and they want to consistently and easily measure the performance of different browsers. There are lots of “benchmarks” in the history of software, and lots of benchmarks around browsers. The challenge with benchmarks is generally understanding what the benchmark measures and deciding whether those measurements are important to customers, developers, and you.

This blog post takes a closer look at some of today’s most common benchmarks. We’ll provide context on their origins, talk about what they test, and show you what browser subsystems each benchmark executes.

Celtic Kane
Celtic Kane is generally credited with being the first JavaScript benchmark. It was originally created in 2006 by a web developer (Sean Patrick Kane) and became popularized after the benchmark was featured on Digg.com. Celtic Kane consists of eight simple test cases which are each run once and measured in milliseconds. Celtic Kane claims to focus on core JavaScript and DOM performance; however the DOM tests inside Celtic Kane don’t actually measure the DOM.

Celtic Kane Test tests the JavaScript engine

WebKit SunSpider
WebKit SunSpider is one of the most commonly cited benchmarks and has become a focus for the industry. Given that SunSpider focuses on JavaScript performance and not real world scenarios that matter to customers this is unfortunate. SunSpider is developed by the Apple WebKit team with the goal of measuring real world JavaScript performance. There are 26 test cases which attempt to measure the performance of individual JavaScript features including arrays, math, and regular expressions. The WebKit SunSpider tests exercise less than 10% of the API’s available from JavaScript and many of the tests loop through the same code thousands of times. This approach is not representative of real world scenarios and favors some JavaScript engine architectures over others.

Webkit Sunspider Test test the JavaScript engine

Google V8
Google’s V8 was designed to help Google tune the V8 JavaScript engine. The benchmark carefully targets the JavaScript subsystem and measures some computationally complex scenarios such as OS thread scheduling, cryptography, and ray tracing in JavaScript. Though well designed, some of the V8 benchmark tests were programmatically generated from Scheme language benchmarks and do not use programming language patterns commonly used by JavaScript developers. While academically interesting these scenarios are not representative of how the JavaScript language is used or fairly represent patterns across the web.

Google V8 Test tests the Javascript engine

Mozilla Dromaeo
Mozilla’s Dromaeo is one of the best engineered JavaScript and DOM benchmarks available today. It was originally written by John Resig and the Mozilla Foundation to support Firefox development and is a very effective tool for browser vendors who need to measure the performance of individual API’s. In addition to pure JavaScript and DOM tests, Dromaeo measures a few additional areas including common jQuery and Prototype features along with CSS Selectors. Dromaeo also provides a test harness for the WebKit SunSpider and Google V8 benchmarks.

Mozilla Dromaeo Test tests the Javascript and DOM engines

Flying Images
The Flying Images demo released with the first Internet Explorer Platform Preview has unexpectedly become a “hardware accelerated graphics” benchmark across the web. Our intent with Flying Images was to show how the GPU can impact the HTML4 websites of today, not create a benchmark. That said it does fairly represent the underlying coding patterns found in today’s common AJAX applications. Flying Images uses standard HTML4, JavaScript, and CSS to programmatically move images around on the screen every 1/60 of a second (considered real-time on a 60hz display). It manipulates the DOM, changes CSS properties, and causes the entire page to go through the layout process.

Flying Images Test test JavaScript, Marshalling, DOM, Formatting, Blocking Building, Layout and Display

FishIE Tank
The FishIE Tank demo released with the third Internet Explorer Platform Preview was treated by some bloggers and technology reviewers as a benchmark because of the visually compelling scenario. FishIE Tank merges both traditional HTML coding patterns with HTML5 Canvas. It uses HTML5 Canvas to animate fish around the screen with a JavaScript based collision detection system, and HTML elements and common HTML coding patterns such as setting innerHTML to display an FPS meter over the tank. This demo is representative of the coding patterns that we’ll see as developers build applications that merge the established HTML and emerging HTML5 constructs.

FishIE Test tests JavaScript, Marshalling, DOM, Formatting, Block Building, Layout and Display

Psychedelic Browsing
The Psychedelic Browsing demo has received over 500,000 page views over the last five weeks since the fourth Internet Explorer Platform Preview was released in August. Psychedelic Browsing spins a color wheel as quickly as possible on the screen and was designed to stress a common HTML5 coding pattern that uses JavaScript and the DOM to draw directly into an HTML5 Canvas while playing an HTML5 Audio element in the background. We have seen this pattern start to quickly emerge over the last six months as Canvas has seen broad adoption. This is representative of where many web applications are headed.

Psychedelic Browsing Test test JavaScript, Marshalling, DOM and Display

We believe that to build a fast real world browser you have to start by understanding the real world patterns and design the right systems around those patterns. You can’t build a fast real world browser by optimizing for benchmarks. And you can’t accurately assess a browsers performance by looking solely at the results from benchmarks. When used for engineering purposes, benchmarks can be an effective tool to help understand the implications of code changes. That’s where the value ends though.

As an industry we have to care be careful that we don’t lose sight of what’s important – the real world customer scenarios of today and the HTML5 scenarios of tomorrow.

Jason Weber
Lead Program Manager, Internet Explorer Performance

Comments (31)

  1. seb says:

    Just one more day till IE 9 beta! This will shame mozilla into finishing JM eventualy.

  2. Asrail says:

    Hi,

    I think that the kind of operations suitable to a GPU or some kind of  graphics library are not real usage tests.

    They are good for building games on HTML, so the psychedelic test has its merits. But what matters is day by day usage.

    I've read this today and, by coincidence, there is a news on this subject.

    See this:

    blog.mozilla.com/…/release-the-kraken

    "We’re pleased to announce the first version of Kraken, a new browser benchmark. More than Sunspider, V8, and Dromaeo, Kraken focuses on realistic workloads and forward-looking applications."

    I would love to know what you think about the project and if Microsoft expects to contribute with it or some kind of tests which represent the performance for the sites the user navigates most.

  3. tim says:

    IE9 (platform preview 4) does perform very well against the above noted demos – but it is important to note that shipping versions of IE (and the last available version of IE for XP) outright FAIL the latter tests, and come in dead last on the first tests.

    Although that speaks volumes for the advancements that IE has made in IE9 and latching on to the GPU to overcome it's performance issues be careful how much you brag about this as your current user base is using a FAIL-ing version of your browser (and a vast quantity of them can never upgrade their IE version – ever)

  4. Alistair says:

    While IE9 is doing very well on these tests that test more subsystems and the implication is that IE9 is overall faster than other browsers that have faster javascript engines, in real world use there are currently virtually no web sites like this.

    The real test of a browser is for pages that people use:

    – How fast it loads

    – How snappy it feels

    A benchmark that tests this would be a better indicator of browser performance as it really matters to users (kraken perhaps?).

    It has been mentioned that as IE is not cross platform there are various speed advantages, but until it can claim to be faster than other (beta) browsers in current real world use, there doesn't seem to be much room for making claims of speed.

    (In 3+ years when "HTML5" is prolific then these tests may be more meaningful)

  5. Alistair says:

    While IE9 is doing very well on these tests that test more subsystems and the implication is that IE9 is overall faster than other browsers that have faster javascript engines, in real world use there are currently virtually no web sites like this.

    The real test of a browser is for pages that people use:

    – How fast it loads

    – How snappy it feels

    A benchmark that tests this would be a better indicator of browser performance as it really matters to users (kraken perhaps?).

    It has been mentioned that as IE is not cross platform there are various speed advantages, but until it can claim to be faster than other (beta) browsers in current real world use, there doesn't seem to be much room for making claims of speed.

    (In 3+ years when "HTML5" is prolific then these tests may be more meaningful)

  6. Miguel Web Developer says:

    You missed the PeaceKeeper Benchmark

    service.futuremark.com/…/run.action

  7. snort says:

    >(kraken perhaps?).

    roflmao… yeah, running FFTs on audio and images is really what "real-world" websites do.

  8. Alvatrus says:

    Good post.

    Measuring (anything) is a science in itself, and I'm sure that you are only scratching the surface here.

    Just thinking of *what* you are exactly measuring, and what you actually measure can be a pain. Weighing the measurements of the subsystems. Graphics vs. DOM, for example. Then there's statistics analysis of your data. And then you have to formulate a result.

    Unfortunately the uninformed internet public can just click any benchmark button, and the one that matches their purpose/preconceived ideas best would be the best and most reliable one. And *obviously* a score of 42 trumps an 18.

    That's an uphill battle you can't win

  9. thenonhacker says:

    As a long-time Firefox user, this is article EPIC PWNAGE.

    I can imagine how the fan boys will react to IE9's improvements — mostly denial stuff. I actually showed one of my officemates the IE9 Previews. He is anti-MS, and as expected, he ignored the laudable benchmarks (FishIE, Psychedelic, Flying Images), and goes on to say like "oh IE will be affected by vulnerabilities" and all that sour graping.

  10. thenonhacker says:

    As a long-time Firefox user, this is article EPIC PWNAGE.

    I can imagine how the fan boys will react to IE9's improvements — mostly denial stuff. I actually showed one of my officemates the IE9 Previews. He is anti-MS, and as expected, he ignored the laudable benchmarks (FishIE, Psychedelic, Flying Images), and goes on to say like "oh IE will be affected by vulnerabilities" and all that sour graping.

  11. thenonhacker says:

    @Alistair:

    "in real world use there are currently virtually no web sites like this"

    You should browse more internets.

    – YouTube has already started with an HTML5-based implementation.

    – Apple favors HTML5 over Adobe Flash. This means that more developers will use HTML5 Canvas.

    – There are many websites using JavaScript Frameworks there that can replace Flash Animation, using jQuery and Prototype.

  12. Asgard says:

    There are quite funny comments about "these test does not matter because they are wrong" but for a couple of years those same tests has been very correct to show the superiority of firefox and chrome. Now when IE takes the lead the tests become not valid anymore :D. And what comes to the new tests that takes advantage of GPU, well we are not there yet but at least we can see that IE9 platform preview has way better hardware acceleration than firefox 4 beta 5.

  13. fr says:

    Some early results from Kraken…

    IE9 PP4 http://tinyurl.com/ie9p4kraken

    Firefox 3.6 http://tinyurl.com/ff36kraken

    Chrome 6 http://tinyurl.com/cr6kraken

    Not a great result for IE, lets hope the beta can do better.

  14. Alistair says:

    @thenonhacker

    Things are certainly moving in the direction of HTML5 but a site could only use the HTML5 doctype and it is still HTML5. Performing well on such a site is going to take the same stuff that performing well on a HTML4 site will require.

    For IE to excel as it does in the psychedelic benchmark in real world use, there would have to be sites that make dramatic and aggressive use of animated canvas. I am not aware of any such sites now and this makes sense since virtually no currently released browsers can handle it.

    It is certainly nice to be able to say, our browser will run next gen web stuff really fast, but it is much less important, right now, than having a browser that runs all the current web pages really fast.

    Not to say IE9 isn't really fast, it is, but the implication that IE9 is overall faster than say Chrome because these tests test many browser subsystems, where as the benchmarks that chrome is faster at only focus on javascript / dom, is false in my opinion. In real world use today IE9 is up there, but it isn't a leader

  15. thenonhacker says:

    @Alistair: Internet Explorer 8 sucked in many areas compared to Chrome and Firefox, but now it's very clear that IE9 is meeting and even exceeding Chrome and Firefox in terms of Performance and Standards Compliance.

    End game would be, these will be the top browsers in the near future: IE9, Firefox, Chrome. Chrome will cannibalize Firefox users. Take note, that I am Firefox user myself, and I am faced with a question of switching to Chrome.

    – Any browser that doesn't support hardware acceleration will die.

    – IE9 performance is so fast, developers will be faced between pushing the limits of a browser, or lowering it to accommodate the slower low-denominator browsers. If this happens, Flash and Silverlight will still prevail over HTML5 Canvas.

    – Developers who do Not care about standards compliance will be forced to comply.

    – There will be a LOT of sour graping once there will be Game Sites that provide DirectX-quality HTML5 Canvas games.

  16. Alistair says:

    @thenonhacker

    Can you name one benchmark of performance or standards compliance that IE9 beats Chrome at that isn't created by Microsoft?

    Peacekeeper (Chrome6: 9433, IE9pp4: 3110 – my pc)

    Kraken

    Dromaeo

    Sunspider

    V8 benchmark

    html5test (Chrome6: 217, IE9pp4 97)

    acid3

    – The driving forces behind the web is business and business wants to reach as many people as possible. Until a significant percentage of people have browsers and computers that can handle the IE9 tests, it doesn't make business sense to develop them. This is a long way off and also, no browser will die while there is enough people using it to make it necessary to cater for it. If this wasn't the case IE6 (and 7) would have been dead long ago and it they have been far surpassed by all other browsers.

    – The web is primarily about information and communication. It is questionable that there is much use for intensive graphics on a website as movement is going to distract the user from reading the information. If IE9 was the only browser in the world there won't be an explosion of intensive graphics on websites because having them doesn't make sense in at least 90% of cases.

    – Quake live provides fully cross browser network gaming via a plugin. Games can be made fully cross browser/platform now without needing a IE9 style browser.

  17. Stilgar says:

    "There will be a LOT of sour graping once there will be Game Sites that provide DirectX-quality HTML5 Canvas games." do you really believe that?

  18. wetran says:

    I wonder why no mention of Futuremark Peacekeeper? Futuremark is one of the most famous, if not the most famous, professional and dedicated benchmark maker. And Peacekeeper is surely one of the most thorough, if not *the* most thorough, all-around browser benchmarks out there.

    Maybe because IE9PP still suck at it?

  19. thenonhacker says:

    @All: Read the article above again. Don't miss his point that a browser should not be just confined to JavaScript Performance. There are more areas than JavaScript that has to be fast. Thanks!

  20. Asrail says:

    @thenonhacker: look at the tests and you will notice that even though the IE tests include DOM and others, the real advantage comes from the hardware acceleration.

    That will only benefit heavy animation. That doesn't mean you will benefit on other areas as well.

    The better comparison right now would be load the list of the 100 most visited sites and comparing timeload of them, without cache.

  21. performance problem with html5 says:

    When viewing youtube videos in html5 mode, there are significant frame drops at the beginning, it's just really slow (unwatchable).

    But after that, everything is super-smooth.

    The first few second uses an interpreted js and after that jitted one is used, correct? Whatever the cause, it's unpleasant.

    It would be ok if it would happen once (for youtube) but this happens for every new movie

  22. Puzzler995 says:

    Ok, so you mentioned the big scripting benchmarks… but what about the 3 ACID tests? They test the basis of the web, HTML and CSS, and yet time and time again Microsoft seems to ignore them. You all do realize that WebKit (Chrome/Safari), Gecko (Firefox), and Presto (Opera) are already at the levels you are trying to attain with IE9, and are focused on implementing HTML 5 and CSS 3. You all better catch up, or else you will fall too far behind to catch up, and web developers will alienate you even worse.

  23. Avindra G says:

    Yah. Despite all the new speed, the IE9 beta doesn't even feel as fluid as my Firefox 3.6.10…

  24. wetran says:

    @Asrail,

    Kraken is just pure JavaScript test, there's no DOM, rendering, graphics, etc. etc. test in Kraken, so I don't think Kraken results will be representative of real world applications. I think Peacekeeper is still a far better overall benchmark than Kraken.

  25. zzz says:

    Most these benchs are totally pointless:

    What matters is deterministic performance regardless of content size at high rate so it appears smooth and fluid.

    The best benchmark of real world usage is to create a really heavy page that uses all technologies supported, then see how smoothly does scrolling & panning, text selection and text search work. The rendering and calculations performed on the page may not stop during scrolling, so any bottleneck will be visible when looking at the framerate when scrolling the page. Separetely, the load of the heavy page should be benchmarked when it's behind https.

    Usability is equally important, readability of text and ability to customize address bar to be long and the space for tabs be plentiful, ie. they should not eat up space from each other.

  26. NoPeace says:

    @wetran

    Here's something interesting about the Peacekeeper benchmark – weblogs.mozillazine.org/…/020664.html

  27. wetran says:

    @NoPeace,

    Yes the Peacekeeper benchmark does have its quirks and weirdness, but it's still arguably the most thorough benchmark that tests various browser subsystems and approximates real world applications best, since unlike Kraken and Sunspider etc. which only shows some text and blocks, Peacekeeper actually renders real pages and shows real graphics drawing. And unlike those Microsoft testdrive demos, it's not trying to exploit specific bugs (or features?) of a specific browser (namely Chrome), ie. the oversampling problem when downscaling images (which is mostly irrelevant to GPU-acceleration as shown by results from Opera). At least it's an independent benchmark that's not constantly sending out absurdly large images for the browser to downscale at the client-side to exploit a specific weakness in a competitor browser (and web design common sense tells you should downscale the images to reasonable sizes at server-side to save bandwidth and loading time).

  28. Andrew says:

    Just a question for the IE team, does using

    <meta http-equiv="X-UA-Compatible" content="IE=IE7" />

    make the webpage rendering slower than if it was using

    <meta http-equiv="X-UA-Compatible" content="IE=IE9" />

    ?

    Is the IE7 mode optimised in IE9 to use GPU rendering etc. and make it faster, or does it go back to the speed of IE7?

  29. EricLaw [MSFT] says:

    @Andrew: Generally speaking, IE9 standards mode is our "fastest" mode. However, we still use GPU rendering, JIT-compiled JavaScript, etc, for IE7-compatibility mode pages.

  30. Andrew says:

    OK, thanks – I didn't realise you replied right after!