Handling Multi-touch and Mouse Input in All Browsers


Touch interaction with Web sites and apps has the opportunity to improve their usability and ubiquity as the Web and Windows 8 Metro style apps play a key role on tomorrow’s touch-enabled devices.

This post explains how Web developers can use the new IE10 pointer event model along with the iOS touch event model and the W3C mouse event model (as extended) to create a cross-browser, common-code handler for pointer, touch, and mouse input.

A little background: I’m fortunate to have a Samsung 700T Windows Developer Preview tablet PC. With it, I’ve been able to enjoy the IE Test Drive multi-touch demos Touch Effects and Lasso Birds. Like me, you may have noticed that Lasso Birds works across different devices and browsers in addition to IE10. For example, its multi-touch works on iOS devices. In this post, I’ve taken some of the patterns from Lasso Birds and generalized and extended them to include older versions of browsers.

The result of my experimentation is below. It should work in your browser. A discussion of the coding patterns and lessons learned follows below the demo.

The Code

The basic algorithm for drawing with the mouse model is straightforward:

var drawingStarted = false;

function DoEvent(eventObject) {

if (eventObject.type == "mousedown") {

drawingStarted = true;

startDraw(eventObject.pageX, eventObject.pageY);

}

else if (eventObject.type == "mousemove") {

if (drawingStarted) {

extendDraw(eventObject.pageX, eventObject.pageY);

}

}

else if (eventObject.type == "mouseup") {

drawingStarted = false;

endDraw();

}

}

The only change needed to make this work with IE10’s pointer events is to add awareness that multiple pointers can be down at the same time, each identified by a different pointerId value. The IE10 pointer model fires separate MSPointerDown, MSPointerMove, and MSPointerUp events for each pointer that changes state.

var drawingStarted = {};

function DoEvent(eventObject) {

// the following is needed in developer preview to stop implicit pan/zoom

if (eventObject.preventManipulation)

eventObject.preventManipulation();

 

var pointerId = eventObject.pointerId;

if (eventObject.type == "MSPointerDown") {

drawingStarted[pointerId] = true;

startDraw(pointerId, eventObject.pageX, eventObject.pageY);

}

else if (eventObject.type == "MSPointerMove") {

if (drawingStarted[pointerId]) {

extendDraw(pointerId, eventObject.pageX, eventObject.pageY);

}

}

else if (eventObject.type == "MSPointerUp") {

delete drawingStarted[pointerId];

endDraw(pointerId);

}

}

Adapting the original mouse model to Apple’s iOS touch event model requires that you iterate through the list of changedTouches for each touchstart, touchmove, and touchend event because, in the iOS model, state changes that occur at the same time are bundled into one event. Like the IE10 pointer model, a unique identifier identifies each touch point.

var drawingStarted = {};

function DoEvent(eventObject) {

eventObject.preventDefault(); // without this, instead of drawing, you pan

for (var i = 0; i < eventObject.changedTouches.length; ++i) {

var touchPoint = eventObject.changedTouches[i];

var touchPointId = touchPoint.identifier;

if (eventObject.type == "touchstart") {

drawingStarted[touchPointId] = true;

startDraw(touchPointId, touchPoint.pageX, touchPoint.pageY);

}

else if (eventObject.type == "touchmove") {

if (drawingStarted[touchPointId]) {

extendDraw(touchPointId, touchPoint.pageX, touchPoint.pageY);

}

}

else if (eventObject.type == "touchend") {

delete drawingStarted[touchPointId];

endDraw(touchPointId);

}

}

}

Merging these three individual algorithms requires noting the differences between the event names and the unique pointer identifier attribute names and accounting for the lack of an identifier in the mouse model.

In the merged model, below, I also add a check that the “move” position has actually changed because the IE10 pointer model streams MSPointerMove events with the same x, y position when a touch point is held down but not moved. By filtering out these redundant moves, I eliminate calls to extendDraw() that do nothing. I implemented this check by storing the last x,y position from a start or move in the lastXY object and, by checking that a lastXY entry exists for a particular id, lastXY replaces the drawingStarted object used in the previous two examples.

var lastXY = {};

function DoEvent(eventObject) {

// stop panning and zooming so we can draw

if (eventObject.preventManipulation)

eventObject.preventManipulation();

 

// we are handling this event

if (eventObject.preventDefault)

eventObject.preventDefault();

 

// if we have an array of changedTouches, use it, else create an array of one with our eventObject

var touchPoints = (typeof eventObject.changedTouches != 'undefined') ? eventObject.changedTouches : [eventObject];

for (var i = 0; i < touchPoints.length; ++i) {

var touchPoint = touchPoints[i];

// pick up the unique touchPoint id if we have one or use 1 as the default

var touchPointId = (typeof touchPoint.identifier != 'undefined') ? touchPoint.identifier : (typeof touchPoint.pointerId != 'undefined') ? touchPoint.pointerId : 1;

 

if (eventObject.type.match(/(down|start)$/i)) {

// process mousedown, MSPointerDown, and touchstart

lastXY[touchPointId] = { x: touchPoint.pageX, y: touchPoint.pageY };

startDraw(touchPointId, touchPoint.pageX, touchPoint.pageY);

}

else if (eventObject.type.match(/move$/i)) {

// process mousemove, MSPointerMove, and touchmove

if (lastXY[touchPointId] && !(lastXY[touchPointId].x == touchPoint.pageX && lastXY[touchPointId].y == touchPoint.pageY)) {

lastXY[touchPointId] = { x: touchPoint.pageX, y: touchPoint.pageY };

extendDraw(touchPointId, touchPoint.pageX, touchPoint.pageY);

}

}

else if (eventObject.type.match(/(up|end)$/i)) {

// process mouseup, MSPointerUp, and touchend

delete lastXY[touchPointId];

endDraw(touchPointId);

}

}

}

The examples above specifically ignore the issues of registering to receive the events or ensuring that they apply to the drawing target. Making this work for real and with all browsers—including versions of Internet Explorer before IE9—requires a bit more work. Interested parties can peruse the final version of my multi-browser, multi-touch drawing class here.

Of particular note is a change between Windows Developer Preview and Windows Consumer Preview. To improve the performance of touch interactions, it is now required that you add to the intended touch target a CSS property that disables the default touch actions. The CSS syntax is:

#touchTarget {

-ms-touch-action: none;

}

In the code that powers the demo above, this property is set in JavaScript using code similar to:

if (typeof touchTarget.style.msTouchAction != 'undefined')

touchTarget.style.msTouchAction = "none";

A future blog post will describe the new -ms-touch-action property in more detail.

Code for All Input

By coding for touch alongside mouse, Web developers can assure their sites work with all browsers—whether desktop, tablet, or phone.

—Ted Johnson, Graphics Program Manager Lead, Internet Explorer

2 April 2012—
Updated this post to reflect changes between Windows Developer Preview and Windows Consumer Preview. I thank reader Ike Starnes for reminding me that this post needed updating.
—Ted Johnson

Comments (19)

  1. Anonymous says:

    When can we expect an IE10 preview that runs on Win7? Or will the next build be released with the Win8beta only?

    You're scoffing your developers (again) by not answering this question.

  2. Anonymous says:

    @alvatrus You can get the IE10 preview here: ie.microsoft.com/…/Default.html It's been available for many months now.

  3. Anonymous says:

    @Juliana

    Sorry I wasn't clear. I meant the *latest* Platform Preview, of course. The one that includes the features discussed in this and other posts here on this blog.

  4. Anonymous says:

    If someone want to write about some missing features in IE, please check this post social.msdn.microsoft.com/…/463dbc6d-8bc5-4aca-9b70-b8084ed268cf

  5. Anonymous says:

    @ieblog : you really should try your stuff before releasing them…

    With IE8, just doubleclick in the drawing area and you'll get tons of javascript errors until the browser crashes.

  6. Anonymous says:

    @oliver: Guess what? SVG is an HTML5 feature, so of "course" it doesn't work on IE8.

  7. @Olivier: Thank you for the bug report. As I wrote in the post, "Making this work for real and with all browsers—including versions of Internet Explorer before IE9—requires a bit more work." Apparently, even more work is required for old versions of IE.

    @Leigh: The demo uses VML in IE versions < 9. @Olivier's bug is valid.

  8. Anonymous says:

    @Ted Johnson: When will the latest IE10 platform preview be available to Win7?

  9. Anonymous says:

    How about releasing beta very soon?

    Also the IE9 mobile version seems to have a bug with urls that conain a "#" for positioning within a page. Often the mobile version stays on the beginning of a webpage where it expected to jump to a certain spot lower on the webpage (which the desktop version of IE9 does fine).

  10. ieblog says:

    To those asking about the next IE10 platform preview release:

    We’ll continue to release platform previews on Windows 8 about every 8-12 weeks. We will release an IE10 Beta and Release Candidate on Windows 7 prior to IE10’s general availability.

    Web developers interested in working with the new features of IE10 are encouraged to download the Windows 8 Developer Preview at msdn.microsoft.com/…/home.

  11. Anonymous says:

    Just for curiosity; I have a tablet PC with both pen and touch – in IE9, with the pen I can draw in the field but with my finger I can only scroll the page. Is that expected behaviour?

  12. @SnarkMaiden: Unfortunately, yes, that's the expected behavior in IE9 and document mode 9 in IE10. IE9 has no way to override the default panning gesture. IE10's 10 mode has a new CSS property, "-ms-content-zooming: none" that disables panning and zooming on the target element.

    BTW, this blog runs in document mode 9 in IE10. So even IE10 users with touch are also seeing this behavior.

  13. Anonymous says:

    thanks for the info, ieblog.

    At least now we know that there will be no "delayed" release of PP3 for Win7, but that the next one will be PP4 and will be made available conform the regular release cycle.

  14. Anonymous says:

    Well I hope the Beta isn't too far away then because it is impossible for me to install Windows 8…

  15. @Olivier: Thanks again for pointing out the bug in my demo on IE8. I believe I’ve fixed it. The double-click bug was related to an interaction between VML, the blog’s analytics, and jQuery. I also fixed some unrelated drawing bugs on IE8 and 7.

  16. Anonymous says:

    @Ted Johnson [MSFT] : thanks, it's fixed 😉

  17. Anonymous says:

    So – this is very cool, but you're letting yourselves down if this does not make it into Windows Phone soon – hopefully in the Tango update perhaps. I'm currently trying to build a number of Windows Phone web apps, but every single attempt has been cut off at the neck by the lack of Touch events in IE9 mobile. I was actually shocked by this. Make it happen guys, the mobile web is the future and you're already lagging behind.

  18. Anonymous says:

    Why go down the proprietary route when there's a perfectly decent "touch events" W3C spec covering all of this?

    http://www.w3.org/…/touch-events

    I thought the days of MS doing their own thing and ignoring web standards were in the past…