Making the connection between HTML and UI Automation Patterns


This post discusses how your HTML UI can automatically support various UI Automation patterns, and how you can verify the results by using the Inspect SDK tool.

 

Introduction

The Narrator screen reader uses the Windows UI Automation (UIA) API to access UI presented by your app. So Narrator will access UIA properties relating to elements shown in your UI, and this means it can announce helpful information such as an element’s Name and ControlType. For example, your customer might hear “Save, Button” as they move the keyboard focus over to a button whose purpose is to save something.

Narrator can also access information about the UIA patterns supported by an element. These patterns describe the behaviors of the element, and allow Narrator to programmatically control the element in response to your customer’s actions. For example, your customer might move Narrator to an element which supports the UIA ExpandCollapse pattern, and Narrator will inform your customer whether the element is currently expanded or collapsed. If your customer is using a keyboard, they might choose to change the expanded state through the regular keyboard action. If your customer has no keyboard, and is interacting with the device through touch, they might make a touch gesture which results in Narrator programmatically changing the expanded state through the UIA ExpandCollapse pattern.

 

I’ve recently seen a few devs get told that their UI doesn’t support all the UIA patterns that it should. So this then raises a few questions:

1. Who says my UI should support those patterns?

2. How could this have been avoided?

3. How do I fix this?

4. How could I have learnt about the problem myself when I first built the UI?

 

Below is my take on all this.

 

Who says my UI should support those patterns?

When you design your UI, you don’t think of some big array of pixels on the screen, where some pixels show different colors from others. And you don’t think of shapes and lines and text characters which happen to appear near each other on the screen. So you don’t say to yourself, “I’m going to put a small box here, and nearby I’ll put some text, and if my customer clicks or taps somewhere on all that, something great will happen”. Instead you say, “I’m going to put a really helpful checkbox here”.

Everything in your UI has meaning. By creating UI elements with meaning, you already have an expectation on how that element will behave, and very importantly, so does your customer. Your customer will have expectations around how the element will react to mouse, touch and keyboard input, and how a screen reader will interact with the element. To provide a predictable experience to your customer is essential.

So what’s going to set your customer’s expectations around an element’s behavior when Narrator reaches the element? Those expectations will be set through the element’s UIA ControlType property. If Narrator announces that it’s reached an element with a ControlType of CheckBox, (and doesn’t announce that the checkbox is disabled,) then your customer will expect that they can toggle it. Similarly, if they reach an element with a ControlType of ComboBox, they’ll expect that they can expand it to reveal its dropdown list.

By the way, when I refer to ControlTypes of CheckBox and ComboBox, the actual value of the ControlType properties are UIA_CheckBoxControlTypeId and UIA_ComboBoxControlTypeId respectively.

And from a Narrator perspective, if an element can be programmatically toggled, then it must support the UIA Toggle pattern. If an element can be programmatically expanded, then it must support the UIA ExpandCollapse pattern. A full list of patterns can be found at UI Automation Control Patterns Overview. (Technically, I should be referring to patterns as “control patterns” throughout this, but I’m just so used to using the term patterns…)

 

The first part of the answer to the question of “Who says my UI should support those patterns?”, is really around what you would expect an intuitive experience to be. Many sighted devs who look at some UI, would intuitively know how they expect the UI to behave, based on their past experiences with UI. If they see something that looks like a checkbox, they’ll expect that it can be toggled. If they see something that looks like a combobox, they’ll expect that it can be expanded and collapsed. Your customers who use screen readers have the same expectations in response to learning of the element’s ControlType. In order to meet your customer’s expectations, the elements will need to support the relevant UIA patterns, such as the Toggle pattern and ExpandCollapse pattern.

Some more prescriptive details can be found at Control Pattern Mapping for UI Automation Clients. A table at that page lists UIA ControlTypes, along with the UIA patterns that each ControlType is expected to always support, and the UIA patterns which may be supported in some situations.

I’d recommend taking a look at that table, as I do think it’s pretty interesting. For example, the Button ControlType is not required to support the UIA Invoke pattern. That might come as a surprise, because all buttons can be invoked right? Well, no, not all.

As I type this in Word 2016, I consider the buttons that exist on the Word ribbon. If I point the Inspect SDK tool to the “Decrease Font Size” button, Inspect tells me that the button supports the UIA Invoke pattern. But if I point Inspect to the “Bold” button, Inspect tells me that button doesn’t support the Invoke pattern, but it does support the Toggle pattern. This all makes sense given the purpose of the buttons. The “Bold” button has a current state which the “Decrease Font Size” button doesn’t.

 

The Inspect SDK tool reporting that the “Bold” button on the Word 2016 ribbon supports the UIA Toggle pattern.

Figure 1: The Inspect SDK tool reporting that the “Bold” button on the Word 2016 ribbon supports the UIA Toggle pattern.

 

We’ll talk more about the Inspect SDK tool a little later, and highlight how useful it can be when verifying that the UIA patterns are really supported when an element claims that it supports them.

Note: The table shown at the MSDN page referenced above is created with all sorts of useful HTML such as <th>, <tr> and <td>, which gives the UI meaning. With all that great information, Narrator can help your customer efficiently interact with the data contained in the table. More details on how your customer can interact with tables in your UI is at How a table at MSDN became accessible.

Ok, so you now feel that it’s fair enough that your UIA should support some particular UIA pattern, and the bug assigned to you is justified. Your next thought is, “Why do I have to be dealing with this? Shouldn’t the UI platform have supplied my element with the required UIA pattern support by default?”

 

How could this have been avoided?

You’re asking the right question. Avoiding accessibility bugs in your UI is a far better thing than having to spend time investigating and fixing bugs.

In many situations, the bugs can be avoided by using standard controls which are provided by the UI framework.

 

For example, say my HTML is to contain a combobox with a dropdown. A standard way to define such UI is through use of the “select” tag. When the minimal HTML shown below is loaded up in Edge, a combobox is shown visually, and I can use the mouse, touch or the keyboard to expand and collapse the combobox.

 

<label for="birdBox">Birds

<select id="birdBox">

    <option>Towhee</option>

    <option>Steller's Jay</option>

    <option>Chickadee</option>

</select>

 

But what’s more, I can point the Inspect SDK tool at the UI, and learn that the UIA ExpandPattern is supported by default.

 

The Inspect SDK tool reporting that a combobox defined through use of the “select” tag supports the UIA ExpandCollapse pattern by default.

Figure 2: The Inspect SDK tool reporting that a combobox defined through use of the “select” tag supports the UIA ExpandCollapse pattern by default.

 

Inspect also shows me that Edge is exposing all sorts of other helpful information about the element through UIA. We’ll take a look at some of those later.

 

SO: The message here is that if at all possible, use a standard control for your UI. By doing this, in many cases, the UI framework will provide the UI with a lot of accessibility by default. If you’re presenting an interactable element that shows all sorts of fun custom visuals, go crazy with styling a standard control rather than building a fully custom control.

 

How do I fix this?

Ok, so say by now, you’ve found that for some of your bugs you could replace some div, (that you’d previously marked up to behave in a particular way,) with a standard control, and that resolved the bugs. But in some other cases, you really feel that you needed to stick with something like a div for your control. The next question is, how can you convey the full meaning of that UI to your customer?

Let’s look at an example where you’ve decided to stick with using a div to define an element whose purpose is a checkbox. If Narrator’s going to be able to inform your customer that they’ve encountered a checkbox, and is to be able to programmatically toggle the state of that checkbox through the UIA Toggle pattern, then the element better have a UIA ControlType of CheckBox.

In order to achieve this, you can add an HTML role to the div. Your next question is, how do you know what role you should use, such that the element gets exposed through UIA as having a ControlType of CheckBox. This is where you can visit Mapping ARIA Roles, States, and Properties to UI Automation, and find the role you’re after. And not too surprisingly, the role you need in this case is checkbox.

So let’s say I have the following HTML hosted in Edge, and let’s ignore the fact that this doesn’t look like a checkbox at all. (If you were doing this for real, presumably you’d have all sorts of styling to make your UI look like a checkbox.)

 

<div role="checkbox">Birdwatching required</div>

 

If I point the Inspect SDK tool to that element, I’m told that the element has a ControlType of CheckBox, and that it supports the UIA Toggle pattern, and that its current toggle state is “off”. Wow! All that, just by setting the role!

 

The Inspect SDK tool reporting that the UIA Toggle pattern is supported by a div, and its current toggled state is “off”.

Figure 3: The Inspect SDK tool reporting that the UIA Toggle pattern is supported by a div, and its current toggled state is “off”.

 

But let’s pause for a moment to think about this. Yes, we did get some very important changes to the programmatic accessibility of the div by adding a role of checkbox. But that’s only the start of the work that’s required to make the element fully accessible. You’ll need to react to the element being programmatically checked or unchecked through the UIA Toggle pattern, and always expose its current state through the aria-checked attribute.

And then going beyond programmatic accessibility, you need to make sure the element is fully keyboard accessible. So your customer can tab to it, and your sighted customer is informed through visual feedback that the element has keyboard focus, (including through appropriate system colors when a high contrast theme is active), and they can change the toggle state of the element through the keyboard.

 

Important: Simply assigning a role to an element, or applying some aria tag, does not impact how the element responds to regular keyboard input. To have some custom UI respond to keyboard input in a similar way to how a standard control responds, will require you to add the necessary javascript.

 

So while it is sometimes necessary to set a role on some custom UI as part of making the UI accessible, the additional work required to make the UI fully accessible can be far from trivial. So please do consider first whether you really need that custom UI. If some standard control can provide all or most of the accessibility that your customer needs – go for it!

 

How could I have learnt about the problem myself when I first built the UI?

After careful consideration and some changes to use standard controls or roles, you’ve fixed your bugs. But you’re left thinking, how could you have determined that these bugs were lurking in your UI when you were first building it? It’s time consuming for multiple people to have been involved with logging and discussing bugs, and to be code-reviewing fixes. It would have been great if the bug had never been checked-in in the first place.

This is where I find the Inspect SDK tool so very, very helpful.

Say I’ve built my combobox UI in HTML, using the “select” tag. I believe that by doing this, my customer will be informed that the ControlType of the element is ComboBox, and that they can determine its current expanded state, and can use Narrator to programmatically change the expanded state. But now I want to verify that.

So I point Inspect to the element, and first check out the UIA properties exposed by Edge for that UI.

For this discussion, the first property I verify is that the ControlType is indeed ComboBox, and then I check that the IsExpandCollapsePatternAvailable is true. So far so good.

But an IsExpandCollapsePatternAvailable property of true only means that the element claims to support the UIA ExpandCollapse pattern, and doesn’t necessarily mean that the element really supports it. So what I can do next is check the UIA properties that are exposed through the ExpandCollapse pattern. (Many UIA patterns contain properties which are relevant only to that pattern, and are not available on elements which don’t support the pattern.)

In the case of the ExpandCollapse pattern, it only has one property; the ExpandCollapseState property. So I can use Inspect to verify that that property matches the visual state of the UI.

 

Inspect reporting the current expanded state of my combobox.

Figure 4: Inspect reporting the current expanded state of my combobox.

 

And this is where a powerful, but lesser known feature of Inspect becomes interesting. Not only can I use Inspect to verify the properties associated with a particular pattern, I can also have Inspect programmatically call methods available through that pattern. In the case of the ExpandCollapse pattern, it has the methods of Expand() and Collapse().

To call those methods, I point Inspect to the combobox element, and then go to Inspect’s Action menu. That menu is dynamically populated to include the methods available off the patterns that the element claims to support. All I then need to do is pick the methods shown in the menu to call the Expand() and Collapse() methods. As I do this, I make sure that the combobox UI shown in Edge responds as expected, and that the ExpandCollapseState property reported by Inspect gets updated to reflect the new state.

 

Inspect’s Action menu allowing me to programmatically expand my combobox.

Figure 5: Inspect’s Action menu allowing me to programmatically expand my combobox.

 

Once I’ve done this, I can examine lots of other UIA properties being exposed for the combobox, including:

 

1. The Name of the combobox, which tells my customer what the combobox relates to.

2. Its BoundingRectangle, which is leveraged by my customers who find Narrator’s multi-modal output of audio and visual highlight helpful. And is also leveraged by my customers using Narrator with touch, if they want to learn about the spatial relationship of the UI shown visually. And is also leveraged by my customers using Magnifier, when the entire element is to be shown in the magnified view

3. The current value of the combobox as exposed through the UIA Value pattern. The Value property should be the same as the currently selected item in the combobox’s visual UI.

4. Loads of other helpful UIA properties.

 

My combobox expanded in response to the ExpandCollapse pattern’s Expand() method being called. Inspect shows the current expanded state of the combobox, as exposed through the ExpandCollapse pattern and the current value exposed through the Value pattern.

Figure 6: My combobox expanded in response to the ExpandCollapse pattern’s Expand() method being called. Inspect shows the current expanded state of the combobox, as exposed through the ExpandCollapse pattern and the current value exposed through the Value pattern.

 

One way or another, whether I use the “select” tag or instead build custom UI, I need to get to a point where Inspect reports everything I’d expect relating to the programmatic accessibility of the combobox.

 

A note on UWP XAML and WPF

An HTML dev doesn’t directly add markup describing an element’s UIA pattern support, but UWP XAML and WPF devs have more direct control over such support. That’s because there’s a fairly close match between patterns that can be added to custom AutomationPeers and the desired UIA patterns.

For example, you might build a custom AutomationPeer that implements the XAML Windows.UI.Xaml.Automation.Provider.IExpandCollapseProvider interface, in order for the UIA ExpandCollapse pattern to be supported.

The “Adding support for other UIA patterns to a custom AutomationPeer” section at More tips on building accessible Windows apps include snippets showing how to add support for a bunch of UIA patterns by updating a custom AutomationPeer.

Important: By default you’ll want to use standard UWP XAML or WPF controls which already fully support the UIA patterns that your customer needs. Only add pattern support through custom AutomationPeers when absolutely necessary.

 

Summary

When a bug comes your way which says that your HTML doesn’t support some required UIA pattern, consider the following:

1. Consider whether it is appropriate for the UI to support that pattern. For example, if the element can be invoked, toggled, or expanded through input mechanisms such as touch, mouse or keyboard, then it will need to support the relevant UIA patterns so that the same functionality can be accessed programmatically.

2. If practical, replace the custom UI which exhibits the bug with a standard control which fully supports the required UIA pattern by default.

3. If use of a standard control is not practical, set a role on the element such that Edge will add partial support for the UIA pattern. You will then need to enhance the element’s functionality to react to programmatic calls into the pattern’s methods, and update as required the exposed properties accessed through the pattern.

4. Once you believe the bug is fixed, used the Inspect SDK tool to verify that all properties and methods contained in the pattern are fully functional.

 

As always, thanks for considering how all your customers can efficiently access all the great functionality in your app!

Guy


Comments (0)

Skip to main content