EtchMark Under the Hood: Building a Website that Handles Touch, Mouse, and Pen – and Device Shakes


EtchMark is a new take on the classic Etch-A-Sketch drawing toy, showcasing IE11’s improved support for touch and emerging Web standards (including Pointer Events and Device Orientation). In this post, we’ll walk through several features that you can easily add to your own sites to build an experience that feels smooth and natural with touch, mouse, pen, and keyboard – and even responds to device shaking.

Structure of the Demo

EtchMark enables you to draw anything you want on the screen using touch, mouse, pen, or the arrow keys.  The drawing surface is an HTML5 canvas element that we update any time a knob turns.  In benchmark mode, we use the requestAnimationFrame API, which provides a smooth 60 frames per second animation loop and longer battery life.  The drop shadows for the knobs are created using SVG filters. IE11’s hardware acceleration moves much of this work to the GPU, which leads to a blazing fast experience.  Check out the video below to see these features in action, and then we’ll dive in and see how it’s built.

EtchMark uses HTML5 canvas, requestAnimationFrame, SVG filters, Pointer Events and Device Orientation APIs to create a new take on a classic toy

Touch, Mouse, Keyboard, and Pen Using Pointer Events

Pointer Events enable you to build experiences that work equally well with mouse, keyboard, pen, and touch – all by coding against a single API. Pointer Events are supported across the full range of Windows devices and are coming soon to other browsers as well.  The Pointer Events specification is now a Candidate Recommendation at the W3C, and IE11 supports an un-prefixed version of the standard.

To get started, the first thing we need to do is wire up our pointer events in Knob.js. First we check for the standard un-prefixed version, and if that check fails we fall back to the prefixed version needed to enable IE10 support.  In the example below, hitTarget is a simple div containing the knob image, sized to be a bit bigger so the user has space to land her finger easily: 

    if (navigator.pointerEnabled)

    {

        this.hitTarget.addEventListener(“pointerdown”,   pointerDown.bind(this));

        this.hitTarget.addEventListener(“pointerup”,     pointerUp.bind(this));

        this.hitTarget.addEventListener(“pointercancel”, pointerCancel.bind(this));

        this.hitTarget.addEventListener(“pointermove”,   pointerMove.bind(this));

    }

    else if (navigator.msPointerEnabled)

    {

        this.hitTarget.addEventListener(“MSPointerDown”,   pointerDown.bind(this));

        this.hitTarget.addEventListener(“MSPointerUp”,     pointerUp.bind(this));

        this.hitTarget.addEventListener(“MSPointerCancel”, pointerCancel.bind(this));

        this.hitTarget.addEventListener(“MSPointerMove”,   pointerMove.bind(this));

    }

Similarly, we add the correct fallback for setPointerCapture to Element.prototype to ensure it also works on IE10:

    Element.prototype.setPointerCapture = Element.prototype.setPointerCapture || Element.prototype.msSetPointerCapture;

Next let’s handle our pointerDown event.  The first thing we do is call setPointerCapture on this.hitTarget.  We want to capture the pointer so that all subsequent pointer events are handled by this element. It also ensures that other elements don’t fire events even if the pointer moves into their bounds.  Without this, we’d run into problems when the user’s finger is on the edge of the image and the containing div: sometimes the image would get the pointer event and other times the div would.  This would result in a jagged experience where the knob jumps around. Capturing the pointer is an easy way to solve this.

Pointer capture also works nicely when the user puts her finger down on the knob and then gradually moves off the hit target while continuing to rotate.  Even if the finger isn’t lifted until it has moved several inches off the hit target, the rotation still feels smooth and natural.

The last thing to note about setPointerCapture is that we pass in the event’s pointerId property.  This enables us to support multiple pointers, meaning the user can use a finger on each knob simultaneously without interfering with the other knob’s events. The support for multiple knobs means that when the user rotates both knobs at once, she gets a freeform drawing rather than only seeing vertical and horizontal lines.

We also want to set two flags on this, which points to our Knob object (the flags are per-knob):

  • pointerEventInProgress – tells us whether or not the pointer is down
  • firstContact – tells us if the user has just put their finger down

    function pointerDown(evt)

    {

        this.hitTarget.setPointerCapture(evt.pointerId);

        this.pointerEventInProgress = true;

        this.firstContact = true;

    }

Finally we want to reset the pointerEventInProgress flag when the user lifts up her finger (or mouse/pen):

    function pointerUp(evt)

    {

        this.pointerEventInProgress = false;

    }

 

    function pointerCancel(evt)

    {

        this.pointerEventInProgress = false;

    }

PointerCancel can occur in two different ways. The first is when the system has determined that a pointer is unlikely to continue producing events (for example, due to a hardware event). The event also fires if the pointerDown event has already occurred and then the pointer is used to manipulate the page viewport (for example by panning or zooming).  For completeness it is always recommended to implement both pointerUp and pointerCancel.

With up, down, and cancel events wired up, we’re now ready to implement support for pointerMove.  We use the firstContact flag so that we don’t over-rotate when the user first touches her finger down. After firstContact is cleared, we just calculate the movement deltas of the finger.  We use trigonometry to turn our start and end co-ordinates into a rotation angle, which we then pass along to our drawing function:

    function pointerMove(evt)

    {

        //centerX and centerY are the centers of the hit target (div containing the knob)

        evt.x -= this.centerX;

        evt.y -= this.centerY;

 

        if (this.pointerEventInProgress)

        {

            //Trigonometry calculations to figure out rotation angle

 

            var startXDiff = this.pointerEventInitialX – this.centerX;

            var startYDiff = this.pointerEventInitialY – this.centerY;

 

            var endXDiff = evt.x – this.centerX;

            var endYDiff = evt.y – this.centerY;

 

            var s1 = startYDiff / startXDiff;

            var s2 = endYDiff / endXDiff;

 

            var smoothnessFactor = 2;

            var rotationAngle = -Math.atan((s1 – s2) / (1 + s1 * s2)) / smoothnessFactor;

 

            if (!isNaN(rotationAngle) && rotationAngle !== 0 && !this.firstContact)

            {

                //it’s a real rotation value, so rotate the knob and draw to the screen

                this.doRotate({ rotation: rotationAngle, nonGesture: true });

            }

 

            //current x and y values become initial x and y values for the next event

            this.pointerEventInitialX = evt.x;

            this.pointerEventInitialY = evt.y;

            this.firstContact = false;

        }

    }

By implementing four simple event handlers, we have now created a touch experience that feels natural and sticks to your finger.  It supports multiple pointers and enables the user to manipulate both knobs simultaneously to produce a freeform drawing.  Best of all, because we used Pointer Events, the same code also works for mouse, pen, and keyboard.

Getting more fingers in the game: Adding Gesture Support

The Pointer Events code that we wrote above works great if the user rotates the knob using one finger, but what if she rotates it using two fingers?  We had to use trigonometry to calculate the rotation angle, and calculating the correct angle with a second moving finger gets even more complex.  Rather than trying to write this complex code ourselves, we take advantage of IE11’s MSGesture support.

    if (window.MSGesture)

    {

        var gesture = new MSGesture();

        gesture.target = this.hitTarget;

 

        this.hitTarget.addEventListener(“MSGestureChange”, handleGesture.bind(this));

        this.hitTarget.addEventListener(“MSPointerDown”, function (evt)

        {

            // adds the current mouse, pen, or touch contact for gesture recognition

            gesture.addPointer(evt.pointerId);

        });

    }

With the events wired up, we can now handle gesture events:

    function handleGesture(evt)

    {

        if (evt.rotation !== 0)

        {

            //evt.nonGesture is a flag we defined in the pointerMove method above.

            //It will be true when we’re handling a pointer event, and false when

            //we’re handling an MSGestureChange event

            if (!evt.nonGesture)

            {

                //set to false if we got here via Gesture so flag is in correct state

                this.pointerEventInProgress = false;

            }

 

            var angleInDegrees = evt.rotation * 180 / Math.PI;

 

            //rotate the knob visually

            this.rotate(angleInDegrees);

 

            //draw based on how much we rotated

            this.imageSketcher.draw(this.elementName, angleInDegrees);

        }

    }

As you can see, MSGesture gives us a simple rotation property which represents the angle in radians, so we don’t have to do all the math ourselves manually.  This now gives us support for two-finger rotation that feels natural and sticks to your finger.

Device Motion: Adding a Little Shake

IE11 supports the W3C DeviceOrientation Event Specification, which enables us to access information about a device’s physical orientation and movement.  When a device is being moved or rotated (or more accurately, accelerated), the devicemotion event is fired at the window and provides acceleration (both with and without the effects of gravitational acceleration on the device, expressed in meters/second2) in the x, y, and z axis.  It also provides the rate of change in the alpha, beta, and gamma rotation angles in degrees/second.

In this case, we want to erase the screen anytime the user shakes the device.  To do this, the first thing we do is wire up the devicemotion event (in this case we’re using jQuery):

    $(window).on(“devicemotion”, detectShaking);

Next, we detect if the user has moved the device in any direction with an acceleration greater than our threshold value.  Because we need to detect shaking, we have a counter to ensure that there are two such fast movements in a row.  If we detect two fast movements, we erase the screen:

    var nAccelerationsInARow = 0;

 

    var detectShaking = function (evt)

    {

        var accl = evt.originalEvent.acceleration;

 

        var threshold = 6;

        if (accl.x > threshold || accl.y > threshold || accl.z > threshold)

        {

            nAccelerationsInARow++;

            if (nAccelerationsInARow > 1)

            {

                eraseScreen();

                nAccelerationsInARow = 0;

            }

        }

        else

        {

            nAccelerationsInARow = 0;

        }

    }

For more information on device orientation and motion, please see this post on the IE Blog.

Orientation Lock

IE11 also introduces support for the Screen Orientation API and features like Orientation Lock.  Since EtchMark is also a performance benchmark, we want to keep our canvas size the same across screen resolutions so that we’re doing the same amount of work on every device.  This can make things really tight on smaller screens, particularly in portrait mode.  To enable the best experience, we simply lock the orientation to landscape:

    window.screen.setOrientationLock(“landscape”);

This way, no matter which way the user rotates the device, she always sees it in landscape mode.  You can also use screen.unlockOrientation to remove the orientation lock.

Looking Ahead

Interoperable and standards-based techniques like Pointer Events and device orientation events enable new exciting possibilities for your web sites. IE11’s excellent touch support delivers a smooth stick-to-your-finger experience and interactivity. You can go even further with IE11 and MSGesture, making scenarios like calculating two-finger rotation angles as simple as accessing a property. Give these techniques a try on your own site, and we look forward to your feedback.

Jon Aneja
Program Manager, Internet Explorer

Comments (13)

  1. Anonymous says:

    When will you stop pushing pointerevents and get real?

    No browser other than IE supports this.

  2. Jacob Rossi [MSFT] says:

    @Mark

    Actually, Pointer Events is now a W3C Candidate Recommendation specification created in collaboration with Google, Mozilla, Opera, jQuery, Nokia, and many others (and reached Candidate Recommendation faster than most specs at the W3C).  Early open source prototype code exists for Chromium [1], written by Google, MS Open Tech, and others. Firefox is also in the process of implementation [2] with an early developer build made public recently [3].  For sites today, a high quality polyfill has been built by Google's Polymer project [4]. Many JS frameworks have already added support for pointer events as well and tons of top sites are using them.

    [1] code.google.com/…/detail

    [2] bugzilla.mozilla.org/show_bug.cgi

    [3] tbpl.mozilla.org

    [4] http://www.polymer-project.org/…/pointer-events.html

  3. Anonymous says:

    Why don't you go complain to them instead of b*tching here?

    @ieblog

    Thanks for the informative post.

  4. Anonymous says:

    Wohooo, new IE for new XBox!

    Remote debugging would be really nice, as "To develop and debug your website for Internet Explorer for Xbox, you can use Internet Explorer 10 on Windows 8." (msdn.microsoft.com/…/dn532261%28v=vs.85%29.aspx) is useless to debug IE10 for XBox One specific performance and user interaction issues. Also, a script that behaves nicely across all available IE9-11 on Windows 7/8/8.1 combinations runs away on IEXBO, so the provided means of debugging might be completely useless and misleading.

    If the timeline for bringing JIT to IEXBO is not overly top secret, I'd appreaciate If you'd share soon. Right now some computational tasks are about twice as fast in webkit, blink or gecko on previous-gen phones (half the number of CPU cores and signifficantly slower RAM). And we have to somehow explain that to our IEXBO users, as they otherwise might think our app is rubbish or even broken. Or lock them out. Please don't make planning the future too difficult for us developers. We've already enough worries, like that we have to support mouse, touch and pointer events, but Apple doesn't. It doesn't really matter that I like Pointer Events the best (especially the freedom of choice provided by pointer capture as opposed to Touch Events).

  5. Anonymous says:

    Will the F12 developer tool Emulation feature ever get fixed? It's completely broken when I use Windows Phone mode.

    I get residual "graphics droppings" every time I change a resolution, it shows up as multiple layered scrollbars.

    When I resize the window a little, the emulation viewport suddenly expands to 100% of the window.

    The emulation tool in IE10 was much better.

  6. Anonymous says:

    Please go on Channel 9 and make a presentation of IE11. I would like to know more about current Javascript 6 support.

  7. Anonymous says:

    The one thing that's still missing is a gesture (preferably) or some UI control, in Metro IE10/11, that allows you to quickly move the top or bottom of a very long page.  This is a seriously usability nightmare for me, and is one of the main reasons I use Desktop IE over Metro IE on Surface tablets.

    I would think semantic zoom would work great for this (after some point in "pinching in", it should zoom way out to just show the entire page (so far rendered) in a long column strip, and then let you touch any point … top, middle, bottom… to quickly navigate to that position on the page).  But if not, at least let there be something else… a button or control, a two finger swipe, something analogous for a trackpad, that lets you at least quickly jump to the top or bottom in scenarios where you don't have a keyboard, or where the keyboard is inconvenient.

    This is especially necessary on the Windows Phone version of IE, but definitely on any touch-screen tablet implementation.

  8. Anonymous says:

    Really wish you guys would post clean code to this blog and follow modern JavaScript coding standards! (no one uses C style coding conventions outside of C!)

    Here's your first sample cleaned up:

    var el = this.hitTarget;

    if(navigator.pointerEnabled){

     el.addEventListener("pointerdown", pointerDown.bind(this));

     el.addEventListener("pointerup", pointerUp.bind(this));

     el.addEventListener("pointercancel", pointerCancel.bind(this));

     el.addEventListener("pointermove", pointerMove.bind(this));

    } else if(navigator.msPointerEnabled){

     el.addEventListener("MSPointerDown", pointerDown.bind(this));

     el.addEventListener("MSPointerUp", pointerUp.bind(this));

     el.addEventListener("MSPointerCancel", pointerCancel.bind(this));

     el.addEventListener("MSPointerMove", pointerMove.bind(this));

    }

  9. Anonymous says:

    Hi, please stop adding features that we'll need in next ages and focus missing ones for past ages. While upgrading IE to standard based you have accidently removed (weirdo) XSLT and XPath support. Currently there is no way to perform an xslt or executing xpath query.

    For example you have added to DOMParser which is great but how on earth have you forget XSLTProcessor ? I cannot use native dom object with Msxml2.XSLTemplate activex class.

    For xpath example see ; stackoverflow.com/…/xpath-in-internet-explorer-10-gone

    Our customers also is your customers and currently we are telling them to "do not use IE"

  10. Anonymous says:

    @ed:

    Please post a link to modern JavaScript coding standards.  Recommendations, but not standards, are easy to find from Google, jquery, ……..

  11. Anonymous says:

    I second Ed's assertion. I guess IETeam members a very basic shortcut of VS, Ctrl+k, Ctrl+d and VS will refactor the tons of code in half a millisecond!

    VS2013 with Web Essentials 2013 (github.com/…/webEssentials2013) FTW! !!

  12. Anonymous says:

    In addition to a necessary gesture for getting to the top or bottom of a page quickly…

    I'm finding that using IE Desktop on high-resolution laptops (like the Yoga Pro 2) is very cumbersome.  Can the UI team please add a "touch mode" to Desktop IE?  Like they do in Office?

    In particular:  It's nearly impossible to select the last tab without accidentally opening a new tab, or to close a tab rather than selecting it… and the "favorites bar" targets are just WAY too small (and way too thin vertically).  It's VERY difficult to grab the scroll-bar thumb on long pages (it gets way too small).  ALL of these things would be relatively easy to do if IE-Desktop either had a touch-mode you could manually select (like Office 2013), or at least had more customizable options, or even if it just adapted to a high-DPI screen a bit.

  13. Anonymous says:

    IE 11 crashes on several GitHub pages and also the customized tracking protection list is broken.

    I have it set to manually block content and to show content that is at least on 3 pages… it shows the facebook tracker and a sourceforge tracker, and always keeps saying "not set, allow" even if i disallow and press ok and reopen it… it is back on "not set" again…. also it never refreshes and never shows new pages.

    why doesn't microsoft open a bug tracker / submission page like all other vendors?

    i have found lots of bugs in IE11. and even in IE10 that never got fixed.