Behind the scenes

Many people work together make an effective theatre production. Some are actors, others directors, but many work backstage. These guys and gals work hard to make sure the play goes smoothly, both for those on the stage and those in the audience. You probably have seen our “Ease of Access Panel” or our other applets in Windows, but often as an end user, you don’t know what a large portion of our team does here at Microsoft “behind the scenes” for accessibility.

You may have never heard of “UI Automation” prior to this blog, and for the most part, assistive technology (AT) companies and Microsoft try to make their software “just work”. You may know that Microsoft provides “something” to facilitate communication between your AT and the applications you work in, but you might not know what that something is. That something is what we in the ‘biz call programmatic access. It’s a way for assistive technology to communicate to the application so they can present it in a different way. They make the application bigger, allow the user to speak to the application, or provide other cool alternatives. Think of programmatic access as a different way to interact with an application than originally intended by the developer.

Programmatic access is kind of an abstract concept because its platform dependent. Whether you’re at an ATM, on a P.C., or on a phone, the platform should provide AT the ability to “plug in” to that UI and allow you to interact with any application. Here’s where Microsoft comes in! UI Automation is a platform independent technology that allows your AT to talk to any application. UI Automation was introduced in Windows Vista, greatly improved in Windows 7, and is even being developed on Linux by Novell.

As you probably guessed (or already know) we had an existing programmatic access technology called Microsoft Active Accessibility (MSAA), which we introduced in Windows 98/95. Microsoft was the first company to provide a standard API that allowed vendors to communicate to software on the Windows desktop. MSAA was great for a lot of your basic applications with buttons, check boxes, edit boxes, etc. As you know, software progresses, and we found quite a few limitations in MSAA. These limitations were found by AT developers as well as end users.

The AT vendors have their own requests, so let’s stay focused on end users. The Accessibility team learned a lot from end users. Let me point out a few things we heard:

· Why is INSERT_MY_FAVORITE_AT so slow!

· Why doesn’t INSERT_MY_FAVORITE_APPLICATION work with my AT?

· I wish I could do THIS with my AT, why isn’t it possible!

There have been many other complaints recommendations, but these are three occur often. Let me try to summarize how UI Automation addresses some of these problems.

Why is INSERT_MY_FAVORITE_AT so slow!

Sometimes your AT is slower than you are. This was a major request from end users, but also vendors asking us to provide a way to speed up their software. UI Automation provides “caching” for AT, something that was not provided in MSAA. This means, for things applications and parts of those applications you use a lot, the AT can have it ready to serve to you right away.

Why doesn’t INSERT_MY_FAVORITE_APPLICATION work with my AT?

This is a hard question to answer, and many ways you can’t blame Microsoft or your favorite AT for different applications that don’t work. Programmatic Access requires three key players. The AT, the application that’s talking with the AT, and the language for the two to talk. UI Automation is the language; we facilitate the communication with our technology. Why does the person that makes software need to provide programmatic access? Because both pieces of software need to communicate to each other. Unfortunately the AT doesn’t always know what the application is trying to do! Imagine you are learning a new language, and you say (or sign) an incoherent phrase. When miscommunication occurs, others will have no idea what you are trying to say. This happens all the time with AT and your favorite software. Whether its miscommunication or no communication at all, the software doesn’t tell the AT what to do correctly and as a result, your experience isn’t as great as it could be.

UI Automation can’t solve all these problems (boy we’d love too!) but we do a few things to help facilitate this communication among developers of applications and AT on all operating systems.

· Windows Applications use UI Automation – Windows Explorer, Internet Explorer 8, XPS viewer, and many controls you see in Windows use UI Automation. We’re not perfect, but we work hard to make sure our own software uses UI Automation

· Make it easier to implement UI Automation – Without going into the details, UI Automation makes it easier for AT and other software to implement UI Automation. The API’s are easier to use, applications have richer ways to describe and provide interaction to their UI, and we now have native Win32 and managed .NET UI Automation API’s.

· UI Automation is Open – I’m sure it’s hard to believe, but Microsoft invested in making sure our licensing included ways for developers of all operating systems to develop UI Automation. We’re putting our money where our mouth is on this one. We are actively working with Novell who is creating UI Automation implementation for the Mono project on Linux. We also recently provided a grant with NVDA who creates an open source screen reader for Windows.

· UI Automation provides a suite of testing tools – We provide tools that allow AT and other developers to investigate their UI’s to understand how well they implement UI Automation. Did you know we (yeah Microsoft!) has open source tools on our website as www.codeplex.com. One of our major UI Automation tools, UIA Verify can be found on Codeplex. For another blog on our test tools, check out MY_OTHER_BLOG

I wish I could do THIS with my AT, why isn’t it possible!

UI Automation opens new doors for developers to provide you, with new ways to interact with your computer. Much of this advancement comes through the use of “control patterns.” We could spend an entire blog just talking about this cool technology, but the point is, control patterns enable AT to interact with UI by directly linking into the methods that occur with what the developer expected as the “typical input/output” Control patterns are the “actions” that AT vendors can enable you to use. In MSAA, AT could implement very few actions; for example, clicking a button, or inserting text. With UI Automation, we allow many more actions as well as provide applications the ability to create their own actions we couldn’t think of. We’re excited to see how AT developers use control patterns to make your (and my) experience even richer!

So…now what do I do after learning all of this?

Great question! As we mentioned, end users don’t use UI Automation directly. We’re working hard to get UI Automation adopted in various platforms, but technology adoption takes a lot of work. Microsoft has to keep improving UI Automation, AT developers and application developers need to adopt it, and end users need to understand that it benefits them. If you’re interested if your software uses UI Automation, head to their web page, email them, or ask. We’re not asking you to start a crusade or even demand UI Automation be included in the software, but there isn’t any harm to learning a little more about your favorite assistive technology.

In addition, whether you, friends, or coworkers develop software, make sure your applications you use with your assistive technology provide programmatic access. We’d love for it to be provided with UI Automation.

Plays can often be called off for different reasons, but many times, technical problems occur canceling shows, sometimes during the opening act. The audience is ready, the actors are acting, but the show is cancelled because something brakes backstage. We’re working behind the scenes to make sure you don’t get stuck in the same position as the audience. Our Windows Accessibility team invests in many technologies you see on your desktop, but even though you won’t see us, many more of us are working so your software works in faster, secure, and robust ways.