Under the Covers: Let It Snow…


With one of those rare Seattle snowstorms underway today, I feel this is a great time to publish this description of our Holiday 2011 Test Drive demo “Let It Snow.” —Editor

When a browser effectively uses the underlying hardware, the possibilities are limitless. Over the holidays we released a demo that helps showcase the advantages of a fully hardware-accelerated, touch first browsing experience with Internet Explorer 10. This post takes a closer look at how the Let It Snow demo was created. The patterns in this demo are typical of the HTML5 experiences emerging across the Web today and early Metro Style Apps. We’ll take a look from a functional point of view, rather than visual.

As low power mobile devices and touch first user experiences become mainstream, customer expectations around browser performance are quickly expanding. Browser performance now includes how smoothly the Web site moves under your finger, how quickly the Web site responds to touch interactions, and how efficiently the browser consumes battery. With Internet Explorer 10 and Windows 8 we set out to build the world’s fastest browser for real world scenarios.

Building Let It Snow

We start with a themed postcard which contains a designated area for the holiday greeting.

Holiday Postcard

We start with a themed postcard which contains a designated area for the holiday greeting. The postcard is generated at runtime using HTML5 canvas and uses many common drawing techniques including drawing images, composing gradients, manipulating opacity and skew, and more.

We next add lots of falling snowflakes to create a snowstorm.

Animated Snowstorm

We next add lots of falling snowflakes to create a snowstorm. We do this using a second canvas that���s layered over the postcard. Each snowflake is represented as an object in a JavaScript collection which holds the state of the snowflake, such as current location, velocity, and the image to draw. The canvas is cleared each frame and the scene recreated from the underlying model.

Over time snowflakes will collect on the sign.

Snowflakes Collecting

Over time snowflakes will collect on the sign. Using hit testing techniques we determine when the snowflake is over the sign and then randomly assign the snowflake a target on the sign. As the snowflake approaches the target we use HTML5 Canvas composite modes to compose the snowflake on the canvas.

Snow can be brushed off the postcard using your finger or mouse.

Brushing Away the Snow

Snow can be brushed off the postcard using your finger or mouse. We track the user input through the new MSPointer event which enables a single consistent API across different pointer models. This allows us to provide a great multitouch experience in Internet Explorer 10 running on Windows 8. We use these captured points to erase portions of the canvas, creating the illusion of the snow being brushed away.

When the layers are combined they create an interactive holiday scene.

Complete scene

When the layers are combined they create an interactive holiday scene. The patterns used in this demo represent techniques commonly found in games such as Angry Birds or Cut The Rope, including script based animations, sophisticated user input, physics based game logic, and more.

We’re excited about the fast and fluid interactive experiences that can be achieved in Internet Explorer 10 and Windows 8 with a fully hardware-accelerated and touch enabled HTML5 platform. There’s no better way to experience these benefits than first hand through the Windows 8 Developer Preview.

—Bogdan Brinza‎, Program Manager, Internet Explorer

Comments (37)

  1. John Heston says:

    Let It Snow uses MP3 audio. I'd be interesting to know what process happened under the covers to access Let It Snow's liability for the MP3 licensing fees (http://www.scirra.com/…/why-you-shouldnt-use-mp3-in-your-html5-games). How do you know that the MP3 track doesn't require licensing fees to be paid in this case, or were licensing fees in fact paid? You mention Cut the Rope in your blog post. Cut the Rope features Ogg Vorbis audio and works well in Firefox 9 without Flash. Why doesn't Let It Snow similarly feature Vorbis audio?

    At present the biggest challenge with HTML5 audio is Internet Explorer”s lack of support for Web standards compatible royalty-free audio. Web developers don”t want to have to worry about paying licensing fees to patent cartels. They just want to get on with developing for the Web.

    Steve Ballmer is on record as being pro-developer (See http://www.youtube.com/watch or http://www.youtube.com/watch if you prefer the remix). He's even on record as being pro-web developer (http://www.youtube.com/watch). When then will Microsoft and the IE team address this pain point for Web developers? When will Internet Explorer offer out of the box support for Ogg Vorbis in the HTML5 audio tag?

  2. John says:

    Not supporting OGG and other open formats is a complete deal-breaker for most of us, they're either supported or a browser doesn't support the audio and video elements period. This "if the user installs it" move with IE9 wasn't acceptable either, time to drop MPEG-LA games and support standards without holding back because if you don't win us web developers over you'll only continue to see us migrate users away from your software and I would imagine Microsoft values Bing succeeding more than the MPEG-LA license/patent pool.

  3. atomic1fire says:

    The funny thing is that when bing runs videos, they use mpeg and then webm as a fallback for chrome and firefox.

  4. Enough says:

    @John: OGG is container format, that can hold audio tracks with format like these: Vorbis, Flac, Speex. Which one of them do you think to be used? Only one of them is standardized and ironically it is the less used one. The OGG container format is also NOT standardized.

  5. Jimmy says:

    @Enough: @John made it clear they meant the Vorbis format in the Ogg container, he mentioned Vorbis several times in his post.

  6. @John Heston

    Actually MP3 licensing does not apply to HTML5 games because such games do not contain codec or encoder software elements.

    All te MP3 patemt licensing applies to mp3 coding en decoding and transmitting technology.

    The files format themselves are not the patented technology.

    The patented technology is in the codec or encoder software. So in producing or playing mp3 files.

    In HTML5 audio case the browser plays the adio and thus contains the patented technology and the codec for that has been payed by te browsermaker.

    If you have a game which has th mp3 player contained in it then you have to pay a licening fee.

    Still I would advise everybody to use AAC which is a much better fomat for audio

  7. Moderator says:

    – Apple Safari supports H.264 and requires plugin to run ogg/vorbis

    – Mozilla Firefox supports ogg/vorbis and requires plugin to support H.264

    – MSIE supports H.264 and requires just a "codec" installed on the system (VLC/K-lite) to run ogg/vorbis

    – Google Chrome supports both.. threatened several time to drop the H.264 support.. but haven't dropped it yet (even in the final revision of Chrome)! FF seems to be influenced by their approach but unable to sense Google's play in all the staged war. When FF supporters don't mind installing flash plugin, why don't they install H.264 plugin/add-on^ ?

    ^http://www.interoperabilitybridges.com/html5-extension-for-wmp-plugin

  8. John Heston says:

    @A_Zune: "Actually MP3 licensing does not apply to HTML5 games because such games do not contain codec or encoder software elements."

    You should probably read the Scirra blog post I linked to and the MP3 Licensing website. Royalties apply to the use of MP3 encoders and decoders as well as to the streaming and distribution of MP3 files. From the MP3 Licensing FAQ (http://www.mp3licensing.com/…/index.html):

    "4) Do I need a license to stream mp3 or mp3surround encoded content over the Internet?

    Yes. A license is needed for commercial (i.e., revenue-generating) use of mp3/mp3PRO in broadcast systems (terrestrial, satellite, cable and/or other distribution channels), streaming applications (via Internet, intranets and/or other networks), other content distribution systems (pay-audio or audio-on-demand applications and the like) or for use of mp3/mp3PRO on physical media (compact discs, digital versatile discs, semiconductor chips, hard drives, memory cards and the like)."

    And

    "5) Do I need a license to distribute mp3 or mp3surround encoded content?

    Yes. A license is needed for commercial (i.e., revenue-generating) use of mp3/mp3PRO in broadcast systems (terrestrial, satellite, cable and/or other distribution channels), streaming applications (via Internet, intranets and/or other networks), other content distribution systems (pay-audio or audio-on-demand applications and the like) or for use of mp3/mp3PRO on physical media (compact discs, digital versatile discs, semiconductor chips, hard drives, memory cards and the like)."

    Note also that the royalties section of MP3 Licensing's website (http://www.mp3licensing.com/royalty) lists the royalty rates that apply to software that incorporates MP3 decoders and encoders (http://www.mp3licensing.com/…/software.html) and the "running royalty" that applies to software that merely uses MP3 audio, such as HTML5 games (http://www.mp3licensing.com/…/games.html).

    Games are defined as being "Interactive software products intended to interact with a user for entertainment (action, role play, strategy, adventure, simulations, racing, sport, arcade, card and board games) and/or education, that may be published for multiple machines, platforms or media." I'd say Let It Snow qualifies under that definition.

    When all the other protocols and formats that make up the web are royalty-free, it's a shame that Internet Explorer only supports royalty-bearing audio. It's especially a shame when there are high quality royalty-free (and worry-free) formats available such as Ogg Vorbis. Who knows? Maybe IE10 will support Ogg Vorbis. I hope so.

  9. Aethec says:

    @John Heston: How are these HTML5 games and demos commercial? Did you even read what you copy/pasted?

  10. Enough says:

    @Jimmy

    Too bad that Vorbis is not a standardized format.

  11. Hey says:

    @John Heston

    Does AAC have royalty fees too?

  12. @John Heston

    Actually streaming mp3s does probably could fall under patents because streaming is a form of transmitting the mp3's.

    It depend on how you stream the mp3 if you actually infringe on suc a patent. Even with streaming music it is likely possible to avoid transmission patents on mp3 but that would depend on you implementation

    However serving HTML5 does not use streaming but uses mp3 webserver file distribution on the server side and webserver file distribution technologie is not in any way mp3 patent related.

    You can look up the patents on the mp3 licensing site and see they all describe methods or processes in encoding, decoding and transmitting mp3s.

    They do not describe no static files of filedistribution which is what HTML5 is using.

    Of course the browser playing the HTML5 audio does decode and likely requires licenses or requires a media framework that has the licenses. For instance on Windows mp3 is always present in the media framework that all the browsers can use.

    And of course the site serving he HTML5 game might have to pay licensing for copyrighs to the music rights holder.

  13. John Heston says:

    @Aethec: "How are these HTML5 games and demos commercial? Did you even read what you copy/pasted?"

    I never said they were or they weren't. I said the running royalty for games probably applies. The running royalty for games makes no distinction between commercial and non-commercial games. Here is the link again: http://www.mp3licensing.com/…/games.html

  14. John Heston says:

    @A_Zune: "They do not describe no static files of filedistribution which is what HTML5 is using."

    The licensing for distribution of files is separate from the issue of patents. Whoever controls the patents controls how the format is licensed. Note that the license conditions explicitly talk about the distribution of MP3 files on physical media like CDs. There's no distribution more "static" that files on a write once disc.

    And again, in the case of HTML5 games it is the running royalty that is the core problem: http://www.mp3licensing.com/…/games.html If the MP3 files are "distributed" 5,000 times, by whatever means, you must pay the licensing fee.

  15. Sean says:

    @A_Zune using mp3 files inside HTML5 games most certainly does require licensing fees if you exceed the download count noted here: http://www.mp3licensing.com/…/games.html

    There's a reason why ABSOLUTELY NO ONE building HTML5 games will even consider using mp3 files due to this specific issue.

    Luckily IE does support AAC files so there is a *temporary* alternative – however longer term Microsoft seriously needs to get their act together and support at least 1 open audio and 1 open video format or they will loose developer support very quickly.

    MSFT made great strides in IE9 and IE10 betas to support standards and use the same markup that works in other browsers.  Currently Microsoft's lack of support for free and open audio/video formats are really the only thing holding them back from being able to claim support for HTML5.  So close – but certainly at the moment not there yet.

    In the mean time we will continue to build apps that support open audio and video formats… and detect IE on load to redirect users to download a browser that does support open formats.  It's harsh – but currently we have no choice but to force IE users to upgrade their browser to a Standards based one.  🙁

  16. Moderator says:

    guys let W3C come up with the final decision about the "standard" codec… then expect the support… both are equal candidates.. just like SVG vs VML back in 90's when SVG won !

  17. A_Zune says:

    Just because the mp3 licensing site claims somthing might require a license does not make it so.

    I have looked at ALL the patent they have listed on the site (and thus claim to license)

    and none of them apply purely to existing mp3 files (only to the processes or methods of encoding en decoding them) and none of the patents apply to distributing mp3 as a file.

    The mp3 sites encourages people to take a license even if they do not need it.

    Broadly seen games using mp3 require licenses because most videogames contain playing and decoding software.

    But HTML5 games normally do not contains any audio decoding or playing technolgy as a html game with an mp3 audio file itself does not code, encode, play or transmit the3 mp3.

    Decoding and playing is done by the browser (or underlying media framework).

    Delivery of the mp3 file is usually done by a webserver as a complete file and thus the mp3 is not transmitted as a signal either (in streaming audio it might be).

    The mp3 licensing site is trying to exaggerating the scope of their patent by avoid to state when their patents do not apply. They even avoid to use the word 'file' on the entire site because not of their patents apply to them.

    They are in the business of selling licenses. The more the better.

  18. hAl says:

    MP4 => h.264+AAC

    These are the codecs for the near future at least untill h.265 arrives on the scene in the next few years.

  19. alvatrus says:

    John Heston wrote:

    I never said they were or they weren't. I said the running royalty for games probably applies.

    So you admit you're spreading FUD?

  20. @Moderator says:

    W3C isn't going to bother specifying a standard codec.  They explicitly designed the video and audio tags to support multiple codecs pointing to multiple sources depending on browser support and state within the standard draft that user agents can support any codecs or containers that they want.  They might make a recommendation at some point, but their notes state that no current codec exists that satisfies all parties and concerns.  In any case it is absolutely silly for all browsers to not support whatever codecs as provided by the platform as at least a fallback mechanism.  Even if there is a "standard" codec that is well rounded there are numerous situations where a specialized codec might make more sense, like for replaying screen captures.

  21. John Heston says:

    @alvatrus: "So you admit you're spreading FUD?"

    FUD? What's doubtful about the fact that the MP3 licensing is hugely problematic for anyone trying to develop for the web? What's uncertain about the fact that a licensing fee applies to "Interactive software products intended to interact with a user for entertainment"? It seems necessary to link you to the Scrira blog post again: http://www.scirra.com/…/why-you-shouldnt-use-mp3-in-your-html5-games

  22. Sceptical says:

    I don't get it…  Gazillions of Flash based applications have been using MP3 and H.264 for at least *TEN* years now. And… Suddenly a bunch of "patent and licensing expert" jumps out of nowhere and claim that devs should not use them, instead use the non-standardized and even non-defacto-standard foss formats. It seems really really odd for me…

  23. Hanan Ashraf says:

    Let it snow is great but I have a good idea for IE-10 which is key shortcuts ie:- by long pressing keyboard keys user defined websites would open this will ease up browsing.

  24. Jimmy says:

    @Sceptical: read the follow up post: http://www.scirra.com/…/even-more-about-audio-licenses-on-the-web – Flash has always been exempt, only HTML5 has a problem with MP3 distribution licenses.

  25. Nelson says:

    @A_Zune – Microsoft has already indicated that they need to pay the licensing fees for using mp3 files in their HTML5 based games.

    The reality is that using mp3's puts developers in a position where they may have to pay licensing fees… and using them in HTML5 games that become remotely popular will DEFINITELY require payments.

    Developers are well advised to use AAC format audio files when publishing games that will also play in Internet Explorer.

    Hopefully Microsoft will realize that supporting a fully open and free format like Ogg Vorbis is actually the only realistic option going forward in the future.

    HTML5 Game developers will be speaking loud and clear on the matter over the next year or so – don't expect this issue to die anytime soon.

    @A_Zune – in the mean time take a few minutes to familiarize yourself with the status quo and the many, many posts on the subject.  You are very wrong in your current understanding of the mp3 licensing issues with HTML5, and you do a great dis-service to the entire developer community by continuing to spread false information about a subject you are not fully versed in.  I'm sorry that you are so naive about the issue – but a little research and an open mind will solve your mis-understanding.  If you accidentally released game titles using mp3's change them ASAP before you exceed the threshold that REQUIRES you make licensing payments!

  26. @Nelson

    Where has Microsoft indicated that?

    What mp3 technolgy would be involved uin distributing standard files trough a webserver?

  27. no fraud in this blog says:

    All the scirra.com links are trying to do is sell the product „Conxxxct 2” – and they are at least succeeding in getting back-links, which is good for their search engine ratings. The rest is FUD!

  28. Don says:

    @Microsoft – please step out of the shadows and come forth and clear up the air about mp3 usage in HTML5 games!!!!!!

    The vast majority of developers are aware of the licensing issues (except A_Zune) but clarification from Microsoft is necessary to continue.

    Is Microsoft exclusively recommending AAC audio only? Did Microsoft make the payments they were required to for mp3 licensing on their HTML5 games?

    Please tell us you led by example and paid the fees accordingly and on time.  Microsoft fought to get annoying DRM and license encumbered audio and video formats pushed into IE – we'd like to hear that you felt the payments were worth the effort.

    Oh and by the way your comment form doesn't save posts properly, readers must save their post comments and re-post on every comment submission – please fix this!!!

  29. @Don

    Why don't you ask the Mp3 licensing site what mp3 patents would apply to webserver serving files to a browser.

  30. dex3703 says:

    Boy I love "web development".

  31. Trish says:

    @A_Zune why don't you explain how a company that owns a patent on licensing wouldn't do every last thing they could to get licensing fees from developers that use the technology?  Ever hear of a company called Lodsys?  Patent Trolls are a reality – and everyone that has been developing for at least 10 years is well aware that this is a slippery slope when you assume you are ok to use a file-format or technology that is patented/licensed.

    There's a good reason why the Web is Open… and why HTML5 Video and Audio must also be open.  Just because you feel taking a gamble with your financial security is fun doesn't mean the rest of us do.

    HTML5 Video via h.264 is dead, HTML5 Audio via mp3 is dead.  The only current solutions are Ogg, AAC, and WebM (and even WebM has some strings attached that need to be clarified)

    Unfortunately IE backed the wrong horses – they have a change to rectify this in IE10 – the question is will they?!

  32. @Trish says:

    Patents trolls are a reality, and Google has yet to offer any indemnification to those who implement WebM.  If Microsoft were to embed and distribute WebM with native support in IE and a patent holder does surface they can sue Microsoft.  However, if IE defers to fallback to installed codecs, and the user has a WebM codec installed, they are effectively off the hook.  What reason does MS have to stick their neck out there to support a non-standard proprietary (Google owns it, that's called "proprietary") video format over an industry standard (officially, ISO/IEC 14496-10) with well defined licensing that Microsoft already bought and paid for?  Placating those few who side with the increasing irrelevant Mozilla on their political agenda?  Safari and Chrome both play h.264 out of the box just fine.  Even Opera on Linux will play h.264 if the appropriate GStreamer codec is installed.  Virtually every piece of hardware uses h.264 and virtually every piece of video software works with h.264.  The first *four* examples of source tags for HTML5 video in the current HTML5 draft specification and h.264.  It's not going anywhere.  

  33. Those patentholders on googles proprietary video format VP8 already surfaced 7 month ago.

    fosspatents.blogspot.com/…/googles-webm-vp8-allegedly-infringes.html

    "Thus far, 12 parties have been found to have patents essential to the VP8 standard."

  34. Randy says:

    @@Trish – **PRECISELY!!!*** as you quoted: "h.264.  It's not going anywhere." it may be a great video format, but on the Web its useless due to being a patent encumbered format.

    The Web is open, HTML is open, CSS is open… HTML5Video and HTML5Audio tags are part of HTML – thus they too must be open… this is where IE fails.

  35. @Randy says:

    Reread the post again, slowly.  You might not like h.264 because of the liability it puts on you but at least Microsoft already has licensed those standards and is free to implement them in IE without fear of patent lawsuits.  Until Google officially backs their own proprietary standard with patent indemnification if Microsoft were to implement or even just to ship an existing implementation of it they would be liable to any patent lawsuits that may arise.  If you want WebM to be as open as you feel the web must be then your pressure should be on Google to offer such programs.

    Just because a company buys another company's video encoding tool and throws it onto the Internet it does not make it "open."  There is no officially recognized standard nor is there any legal backing which alleviates the burden of liability.  WebM is a ticking timebomb wrapped up in OSI good feelings and hype.

  36. hAl says:

    @Randy

    Wer already know from years of existing experience that h.264 works very well on the internet. It is the most efficient format.

    Every other format will require a lot of extra costs (more than h.264 licenses) and will cause a lot of extra wasted energy. Other less efficient formats are bad for the environment.

    And nearly everybody already uses it for webvideo. Even Google with their competing format has been using h.264 for many years now and supports it in android, supports it in Chrome and Chromebooks and supports it on Youtube.

  37. Very interesting. I now know how one of my favorite Internet Explorer demos works.