Using Hardware to Decode and Load JPG Images up to 45% faster in Internet Explorer 11


Internet Explorer 11 and Windows Store Apps on Windows 8.1 offload parts of the image decoding pipeline to the graphics hardware, resulting in up to 45% faster image load, up to 40% lower memory consumption, and improved battery life. Images on average account for the most bytes downloaded on the Web today. To improve the performance of loading JPG images, IE11 has streamlined its JPG decoding by moving some steps of the decoding process directly to the GPU where it can be done significantly faster and in parallel.

JPG Image Format

Images account for 61% of bytes downloaded on the Web today, and 47% of image requests are for JPG images. By using hardware more efficiently to decode JPG images, IE11 now loads JPG images up to 45% faster and uses up to 40% less memory over previous IE versions.

To understand these performance improvements, let’s start by looking at how JPG images are typically encoded. The first step of JPG encoding is to convert the bitmap from the RGB color space to YCbCr color space.

Image of the JPG Encoding Process

RGB defines color in three color components: red, green, and blue. If we break down this image of the Grand Tetons in to its RGB color channels, you can see that there is a high degree of detail in each of the channels. The amount of memory required by IE for an image in the RGB color model can be calculated by multiplying image width x image height x 32 bits.

Image of the Grand Tetons with its Red, Blue, and Green components

Image of the Grand Tetons with its Red, Blue, and Green components

YCbCr (commonly referred to as YUV color space) defines a color space in terms of one luma (Y) and two chroma (CbCr) components. The luma component represents the brightness information of a color. The chroma components contain the color differences, with Cb containing the blue difference and Cr containing the red difference. The figure below shows the same image of the Grand Tetons broken down into its Y, Cb, and Cr channels.

Image of the Grand Tetons with its Y, Cb, Cr channels

Image of the Grand Tetons with its Y, Cb, Cr channels

The next step in the encoding process is to compress the image size using a lossy compression known as chroma subsampling. The chroma channels can be significantly compressed because the human eye is more sensitive to the brightness of an image and less sensitive to color or hue. For example, looking at the Cb and Cr channels, the last two on the right, you can see that there is very little information contained in these channels relative to the Y channel. The level of subsampling is typically expressed using a three part ratio, e.g., 4:2:0. The first part of the ratio represents a horizontal sampling reference, and the second two parts represent the number of chrominance samples in the first and second rows of that sampling reference. Chroma is typically subsampled at one of three levels:

  • 4:4:4, where the chroma data isn’t subsampled
  • 4:2:2, where the chroma data is downscaled horizontally by 2,
  • 4:2:0, where the chroma data is downscaled horizontally and vertically by 2.

A 4:2:0 subsampled YCbCr JPG image can use up to 62.5% less memory than the original RGB bitmap! Most JPG images used on the Web today are already in 4:2:2 or 4:2:0 chroma subsampling modes. Most image processing tools will automatically subsample to 4:2:2 or 4:2:0 when using the ‘Save for Web’ option.

Once the image has been subsampled, the discrete cosine transformation, quantization, and Huffman encoding processes are also applied to get the final encoded JPG image.

More Efficiently Using Hardware in IE11

To decode a JPG image, you run the same encoding steps in reverse. Traditionally, IE had always decoded JPG images into RGB bitmaps by running all of the below decoding steps directly on the CPU. At render time, IE would copy the RGB bitmap to the GPU for rendering.

Image of the JPG Decoding Process

To more efficiently use the hardware, IE11 now splits the JPG decoding work between the CPU and GPU. IE11 decodes the JPG image into the chroma subsampled YCbCr color space on the CPU, but then does the chroma upsampling and YCbCr to RGB color conversion steps on the GPU at draw time, where it can happen much faster and in parallel. This process frees CPU time to perform other operations, as the CPU is a common bottleneck in modern sites and apps. In addition to the decode time improvements, copying the much smaller YCbCr image to the GPU reduces the amount of memory that is copied and stored on the GPU (a limited resource). Using less CPU and memory also reduces power consumption and increases data locality.

The below CPU chart shows the amount of time it takes to decode and draw a 4:2:0 subsampled YCbCr JPG version of the Grand Tetons image on IE10 on Windows 8. Image decoding took 31.9ms, and total time to draw the image was 81.5ms.

The below CPU chart shows the amount of time it takes to decode and draw a 4:2:0 subsampled YCbCr JPG version of the Grand Tetons image on IE10 on Windows 8. Image decoding took 31.9ms, and total time to draw the image was 81.5ms.

If we load the same JPG image on IE11 on Windows 8.1, you can see that the decode time is now only 17.9ms, which is an improvement of 44%! The total time to draw the image is now 57.5ms, which is 30% faster.

If we load the same JPG image on IE11 on Windows 8.1, you can see that the decode time is now only 17.9ms, which is an improvement of 44%! The total time to draw the image is now 57.5ms, which is 30% faster.

Because most JPG images on the Web today are in YCbCr format, IE11 on Windows 8.1 users will experience these improvements automatically. We recommend that developers ensure their JPG images are compressed in 4:2:2 or 4:2:0 chroma subsampling modes to maximize the performance benefits of hardware accelerating the JPG decoding pipeline.

Summary

By using the hardware more efficiently to decode JPG images, IE11 improves the performance of nearly every page you browse, while also improving the power consumption and battery life of your device. Please install the Windows 8.1 Preview from the Windows Store and try IE11. As always, we welcome your feedback, either through the IE11 Send Feedback tool or on Connect.

Jatinder Mann, Internet Explorer Program Manager

Comments (44)

  1. Arieta says:

    It's nice to see GPU acceleration so prevalent; but are there any chances that you can improve picture resizing as well? It was a long standing problem on many browsers, but fixed somewhat on them. Basically, pictures are very aliased when shrinked down.

    Testcase: bug486918.bugzilla.mozilla.org/attachment.cgi

  2. Wow! While developers of other web browsers are working on cross-platform browser availability and extensibility, Internet Explorer development team is working on making JPEG images load 24 milliseconds faster. And this change will only be available on IE11, which is only available on Windows 8.1. Okay, why am I not excited?

    Oh,  come on, IE team. Do something with a big bang to it.

  3. Arieta says:

    @Fleet Command: Yeah, I can't wait to use all those new cross-platform browser features, like -webkit-mask-image.

    I also can't wait for websites to stop using -webkit-gradient and -webkit-transform and -webkit-transition and -webkit-any-other-css3, so they can be fully functional on my css3-compliant browsers.

  4. Xero says:

    How about you just finish the WebGL 1.0 implementation, so we can view Nokia Here maps! connect.microsoft.com/…/nokia-here-maps-reporting-webgl-issues

    And please support APNG on IE connect.microsoft.com/…/apng-support

    Thank you.

  5. Stifu says:

    @Fleet Command: yeah, like all other browsers are not fighting to shave extra milliseconds here and there. At this point, I think trying to optimize image rendering performances will bring more benefits to users than optimizing JavaScript for certain benchmarks. Not that trying to optimize JS is bad, but there are probably more low-hanging fruits when it comes to the GPU.

  6. @Fleet Command says:

    IE11 is going to be on Windows 7 too, although it might not include this change.

  7. Sardoc says:

    This change is actually nice. That's what graphics cards are for: to deal with graphics.

    Still, I wouldn't mind if the version for Win 7 had aero style restored. This grey scroll bar is hideous and really hard to see on most websites. The person who came up with it should be tarred 'n' feathered.

  8. ZippyV says:

    > "We recommend that developers ensure their JPG images are compressed in 4:2:2 or 4:2:0 chroma subsampling modes"

    How can I check this in Photoshop (or other applications) ?

  9. Steve says:

    I'm actually impressed by this post.  This is the first post I've read on the IE blog where there is a real gain for the end user (and as such the developers).

    No proprietary IE-only code, no non-standards garbage… just pure Engineering Might put to work! – Well Done!

    PS Mozilla, Google, Opera, Apple… *cough*, *cough*… get on this!

  10. @ZippyV: Your answer is in the text: "Most image processing tools will automatically subsample to 4:2:2 or 4:2:0 when using the ‘Save for Web’ option." "Save for Web" is a Photoshop command by the way. I don't remember any other tool with this command.

    @Stifu: Maybe they do. I don't blame them, since they also work on major changes I mentioned earlier. But IE team works exclusively on minor changes. And anyway, those who get these features are very few. Compare Windows 8.1 users (IE11) vs. users of Windows, Mac OS X, Linux and Android. (Firefox, Chrome and Maxthon)

  11. Stifu says:

    @Fleet Command: alright, I'll go ahead and feed the troll. It seems hungry, and I can't let it starve like that.

    "But IE team works exclusively on minor changes"

    So, adding WebGL support is a minor change, for example? The same goes for HTML5, CSS3, SVG, GPU acceleration, JITed JS, and all that, yeah? Your only argument against IE seems to be that it's not cross platform. Yeah, indeed, you got that right. But tell me more about the "major changes" other browser makers are working on, which puts them in a different league.

    "And anyway, those who get these features are very few. Compare Windows 8.1 users (IE11) vs. users of Windows, Mac OS X, Linux and Android. (Firefox, Chrome and Maxthon)"

    Even if IE11 was for Windows 8.1 only (spoiler: it's not), the target OS is completely irrelevent with the discussion here. Sometimes there are things to *** about, but not here nor now. The IE team is working on making their browser better, like other browser makers. The change detailed in this article is nice, and other browser makers should take note of it. End of story.

  12. Blurga says:

    So no such acceleration for 4:4:4 images or will there simply be no difference to IE10?

  13. Arieta says:

    @Blurga: The post says that YCbCr to RGB conversion is offloaded to the GPU, not just upsampling, so it should still work faster. Also with 4:4:4 images you don't need to upsample anything to begin with, anyway… The recommendation to use 4:2:2 is probably to get smaller jpg image sizes.

  14. Yannick says:

    @Fleet Command: All progres is welcome, at least the IE team works on making IE faster instead of adding features that shouldn't be in a browser (by default). Beside, since when is full GPU acceleration, WebGL, HTML5, CSS3, etc. a minor feature. Another big change in IE is the new interface in Internet Explorer9, the Internet Explorer 10 app and the redesigned and improved Internet Explorer 11 app.

  15. Nice to see.

    What you really should do is:

    1. Chrome, WebKit and Firefox are Open Source projects, go there and support for JPEG-XR. It's time to say goodbye to JPEG. We really need 10Bit support and better compression algorithm.

    2. Add APNG support. The GIF is too limiting.

    Thanks.

  16. C Johnson says:

    So, is the image decoded on the GPU and then pulled back as RGB to be repushed as RGB over and over, saving a grand total of what, 500ms off a page load?  or does the JPEG have to be repushed to the GPU with ever render meaning it has to be continually redecoded over, and over and over and over meaning time usage and more battery usage over time?

    Frankly, this seems like a seriously stupid investment of effort no matter how you look at it.

  17. Open Source says:

    libjpeg-turbo.virtualgl.org

    sourceforge.net/…/libjpeg-turbo

    Firefox uses libjpeg-turbo 1.3.0 and is 2x-4x(200%-400%) faster in JPEG decoding. Don't be fooled by MS, they're outgunned and outperformed by superior open source software and code.

  18. Tom Mouton says:

    @ Open Source

    Libjpeg-turbo and what IE11 is doing are two different things. Libjpeg-turbo is a codec while what IE11 is doing is using the GPU to decode JPEGs.

  19. Arieta says:

    @Royi A: According to caniuse.com, IE9/10/11 is, in fact, the ONLY browser to support JPEG XR.

  20. Heath says:

    @Arieta @Royi : Because Microsoft invented it.

  21. Reality says:

    @Royi A @Arieta @Heath

    Firefox developers looked into adding JPEG-XR and were pretty sold on using it. But Microsoft has not released any Open Source JPEGXR codecs that they could use. No one else in the OSS community has bothered to do so either. WebP gets all the attention these days. If Microsoft were ever committed to the stuff they release into the wild, they would make a decoder that OSS projects can use cross platform.

  22. To Recapitulate says:

    After reading these "Not enough!" comments we can conclude that .. Instead of only optimizing performance of something which people didn't asked for, if you guys added support for APNG and related formats, the reaction would have been different from.

    For example, its being a while there is no blog from SVG team. SVG has some unresolved issues since IE 9. Compare this svg tavmjong.free.fr/…/svg_test.svg  in FF, IE and Chrome. There is one significant issue with incorrectly inheritance of fill opacity filter (connect.microsoft.com/…/wrong-inheritance-with-svgs-fill-opacity).

    As IE team has closed this issue as "Won't Fix" and we can't ask for ETA.. I see no value in submitting bugs on connect.

  23. fester says:

    Would like to see Firefox and Chrome head in this direction as well.  I suspect they would.

  24. Stifu says:

    @Royi A: you seem to be stuck in 2008 or something. Read up on WebP.

  25. Blue Ink says:

    @Reality: Microsoft released JPEG XR under a BSD License earlier this year, and an open source library has also been released on CodePlex in April; see http://jxrlib.codeplex.com

  26. PhistucK says:

    @BlueInk –

    Great. JPEG XR support was added to Internet Explorer 9. So, around three years have passed since the first developer preview of Internet Explorer 9, until they released that open source library in April (according to you, I have not checked). I guess they did not care about the format enough to convince developers to use it (I never heard or saw anything like 'move to JPEG XR' or any other promotion of the format), or other browser vendors to implement it.

  27. jader3rd says:

    I love all of the hardware accelerated things that IE has been doing in the last few releases, I really do. It does create a problem though, when an average user goes to a website in IE and the website is "broken". Then they go to the website in "Google" or "FoxFire" (or however they mis-remember their other browser names) and the website works. So of course the problem is that IE is broken. But when a technical user digs into the issue and finds out that the problem is their outdated graphics card driver, and updating it fixes IE, the user would still blame IE. IE needs some sort of visual detection and hinting that graphics card may be the source of problems and hopefully shim out broken combinations.

    There should be more granular options for software rendering instead of hardware accelerated rendering to test why only certain websites are broken in IE, on specific computers.

  28. Phistuck says:

    Apologies for my last comment. After seeing this link: ie.microsoft.com/…/ImageSupport I realized that I am a sick.. seriously. I don't know what's my problem.. I always find myself pulling some sort of rant.. if they don't support some standard, I will b-itch, if they do i'll b-itch about something else..

    All I want is to see IE fail miserably but the recent developments and improvements are against my will. Damm..

    DIE IE DIEEE!!!!

    can't help myself! it has been always like this, it will always be the same.

  29. @Xero @To Recapitulate  

    Thanks for supporting us and sending us feedback. We've reactivated the feedback for consideration in future releases.

    Thanks again,

    Jay Hong [MSFT]

  30. Stifu says:

    @Jay Hong [MSFT]: it's nice to see you care about user feedback, but seriously, APNG is dead in the water. It has been refused for standardization by the PNG group, and is only pushed by Mozilla (they implemented it 5 years ago, and no one followed). APNG couldn't survive that stupid war with MNG. Anyway, WebP does more things and has more backing.

  31. PhistucK says:

    @Phistuck –

    Cute.

  32. Kate says:

    @Jay Hong [MSFT],

    Thanks for WebGL. Please consider WebRTC request.

    connect.microsoft.com/…/ie11-feature-request-support-for-webrtc

    The world is watching and developers are waiting for Microsoft to make a move:

    stackoverflow.com/…/1712065

  33. Real McCoy says:

    @Jay Hong [MSFT], thanks for the consideration. It feels so amazing when we get any kind of response from IE team! 🙂

    By any chance, if you know anyone in IE's printing team, can you please escalate these two bugs with them:

    connect.microsoft.com/…/attribute-value-change-triggers-change-in-ux-behavior

    connect.microsoft.com/…/no-drop-shadow-when-printing-svg

    I know these bugs are not the show stopper but at some point they need to be fixed. The first bug is there like forever. I hope for the vNext of IE, I am not late to the party!!

  34. Christian Stockwell [MSFT] says:

    Thanks Real McCoy.

    There are a few legacy areas of IE that we really want to show some love. One example is that in IE11 we focused on improving our editing capabilities to give developers the platform support they need to more easily create great interoperable editing experiences on IE.

    There are a few other areas that we're spending time modernizing legacy components, but so far printing isn't at the top of our list. When we get to that point we'll definitely mine our Connect feedback to see what issues people have reported and try to address them together. That approach will allow us to fix a larger number of bugs without introducing constant churn to developers over several releases.

  35. Steve says:

    @Christian Stockwell [MSFT]

    While you are at it, please for the love of god fix the IE Blog comment form.  At this point in time Connect is almost usable – however the IE Blog comment form is by far the most unusable blog on the planet due to the broken comment form.

  36. Nic says:

    Sorry for the off-topic question but isn't the "window.external.msReportSafeUrl" insecure? Imagine visiting a malicious site and reporting itself that it is safe to Microsoft and then Smartscreen Filter trusting it.

    Shouldn't the user only be able to use this feature?

    msdn.microsoft.com/…/hh848317(v=vs.85).aspx

    Also why is this API returning a HRESULT? What is a HRESULT anyway? In JavaScript there is no such type. Isn't it?

  37. Christian Stockwell [MSFT] says:

    Thanks for the positive feedback on Connect. The team's been working hard to make it a more effective feedback mechanism, and we know that there's still more for us to do.

    As I've mentioned in some previous posts we're investigating what it will take to fix the blog software. We know that the "lost comment" bug goes away if we disable anonymous posts, but we'd like to avoid that if we can. In the interim, if you sign in you shouldn't hit the bug.

  38. Real McCoy says:

    @Christian Stockwell [MSFT],

    Thanks for the reply. I have certainly observed outstanding progress in web text editors, especially the textareas where we used to struggle with Undo/Redo. Then there are web-based rich text editors, which break horribly in IE10 and lower. Off the top of my head is http://www.cdolivet.com/…/exemple_full.html. IE11 preview has made tons of improvements to solve these issues. Almost 98% there.. A few nagging usability issues left such as; when you make a selection with mouse click (click somewhere and press-and-hold shift key and perform a second click to make selection), it will highlight one extra line in cdolivet editor. But the selection is perfect. I hope this issue will be resolved by the time of IE11 GA.

    Speaking of word selection.. In Microsoft Office Word, there is an option: "When selecting, automatically select entire word" under File > Options > Advanced: Editing options. Deselecting this option precisely pinpoint the selection where user perform second click. Please provide a similar option in Advanced Internet Options so we have the ability to make precise selection. (I reported it on connect. The word "Bug" in the tittle is misleading, its a feature request: connect.microsoft.com/…/747191). Please consider it. Microsoft Office teams have tons of experience in editing and IMO this option can safely be inherited from Word.

  39. Real McCoy says:

    @Nic, its the interoperability feature with other languages such as C/C++. The ActiveX object may have a "raw" return value, which can then be cast to JavaScript object. ActiveX objects are used in HTA (HTML Applications) executable files in Windows. ActiveX may get unpopular in Internet paradigm, but still extensively been used in various industries for LOC HTAs. Mix it with VBScript and PowerShell cmdlets, and you have a powerful tooling to build some serious, industry grade HTML based intranet apps. You can read more on HTAs for better perspective.

    Of course, with the Internet, the security model of IE wouldn't allow you to execute the ActiveX via untrusted source. Even when the source is some known corp, such as Oracle, Cisco or even Microsoft, it takes user consent to run the ActiveX object even if SmartScreen filter approves it. If somehow the malicious stuff gets ignored and user accidentally allow it, the system malware protection (witch is built into Win8) would sniff the questionable execution and prevent the damage as any other software. So, the user is in quite protected with these layered protection model.

  40. Test says:

    Test comment

  41. Nic says:

    @Real McCoy

    So HRESULT is cast into which JavaScript object exactly? String, number, what? And if it is cast to some JavaScript object, then thy doesn't MSDN say that this particular type is the return type instead of talking about HRESULT?

    Secondly, so you are saying that if this API is called by JavaScript code, then the user will be prompted? Are you sure about that? I thought that the API silently reports the current page URL as safe. The MSDN topic didn't say anything about user consent.

  42. Real McCoy says:

    @Nic, HRESULT is a type (implemented with struct data structure) which signals error codes from COM/ATL ActiveX objects. Among statuses, S_OK is the alias for success defined in enum. Other codes can be found here msdn.microsoft.com/…/aa378137(v=vs.85).aspx

    In JavaScript the HRESULT is serialized into integer.

    When an ActiveX object is called. following is how it transpires:

    – Internet Explorer prompts the user to allow ActiveX on the page (Tools>Safety>ActiveX Filtering)

    – When user allow, IE then downloads the ActiveX object and scan with SmartScreen filter.

    – When passes, it shows the vendor signature, like HP, Cisco, Dell and option to install it.

    – If no signature is found, it shows untrusted vendor alert and option to install it on your own risk.

    – Once installed, it runs like any executable: the malware scans the executable with real-time antivirus scanning to prevent any suspicious activity.

    Of course virus/malware is a subjective problem. But the scam or malware risk is pretty much mitigated like that. Also, SmartScreen is sniffing tons of ActiveX objects transferred all the time over the Internet worldwide and maintaining repository. It also has a mechanism to rate and blacklist them in order to prevent the malware to spread.

  43. Martijn says:

    Do go too wild on hardware acceleration. If you're offloading everything to "hardware" then at some point the CPU has nothing to do. Also, please consider that "the hardware" might be slower than the CPU. You don't control that, and the user sure as hell doesn't care.