A case study in investigating why Narrator’s not announcing a change in UI state


This post describes the approach taken when I recently investigated why the Narrator screen reader wasn't announcing a change in checked state of menu item UI in an in-development product.

Apology up-front: When I uploaded this post to the blog site, the images did not get uploaded with the alt text that I'd set on them. So any images are followed by a title.

 

Introduction

A few days ago I was contacted by a developer who had a bug assigned to them, relating to Narrator not announcing a change in state of their checkable menu item. I found the subsequent investigation really interesting, as I don't think I've encountered a situation quite like this before. So I thought it'd be worth sharing the steps I took, in case anyone else discovers a similar bug with their UI.

Spoiler alert: The root cause seems to be due to the behavior of a UI library being hosted by the product. If that really is the case, then it would seem possible that any product hosting this library will exhibit the same accessibility bug.

 

The UI in question related to a menu in a dropdown. At any given time only one menu item could be checked, and when it was checked, its background changed from white to grey. The following images are my own simulation of the UI. The first image shows a menu containing three items, and the middle item has a grey background, indicating that it's checked, and a black border, indicating that it has keyboard focus.

 

Figure 1: A dropdown menu containing the three menu items of "Chickadee", "Towhee", and "Grosbeak". The menu descends from a button showing a bird icon.

 

When the down arrow key is pressed while the menu is in the above state, keyboard focus moves to the next item in the menu, as shown in the following image.

Figure 2: The second menu item remains grey, indicating that it's checked, and keyboard focus has moved to the third menu item.

 

And finally, if the spacebar is then pressed, the third item becomes checked and the second item unchecked, as shown in the following image.

 

Figure 3: The second menu item is now unchecked, and the third menu item is now checked and has keyboard focus.

 

The bug is that when the menu item becomes checked, Narrator says nothing, (other than "Space" if the keyboard echo setting is on). The change in checked state is not announced by Narrator when the state of the menu item changes, and so the customer is not made aware of the change. They'll no longer know what state the UI is in, and that's not acceptable.

So the steps below describe one approach to investigating this bug. It's important to remember that Narrator only cares about the programmatic representation of the UI, as exposed through the UI Automation (UIA) API. The fact that the background of some menu item happens to be grey, means nothing to me in this investigation.

Tip: For a very quick introduction to UIA, check out UIA at a glance. For a more detailed introduction, check out Introduction to UIA: Microsoft's Accessibility API.

 

 

Step 1: What state is the UI really in after it was meant to change?

The first thing I'm interested in is whether the checked menu item is really checked from a UIA perspective. Semantically, if a UI element can have a state of checked or unchecked, then it would support the UIA Toggle pattern. By using the Inspect SDK tool, I can verify whether the Toggle pattern's ToggleState value is "On" for the checked menu item, and that it's "Off" for the other items in the menu. Sure enough, the ToggleState values were all as I'd expect them to be, so that's good.

Note that in some cases, we might find a bug like this is due to the checkable UI not supporting the UIA Toggle pattern at all, and we'd need to figure out why it doesn't support the pattern. But that's not the case here.

Figure 4: The Inspect SDK tool reporting that the second menu item has a UIA Value property of "Towhee", and a ToggleState property of "On".

 

And for completeness here, now that I know that the UI seems to be in the expected programmatic state, I can arrow away from the checked menu item, and back to it, and verify that Narrator does announce the new checked state. When I did that, Narrator's announcement included the new state of the menu item just fine.

 

Step 2: Remove Narrator from the equation

So, if the Narrator experience is not as it should be, then it must be a Narrator bug right?

Wrong.

Sure, Narrator can be improved, just like any product can, and occasionally some customer experience issue crops up which is caused by Narrator itself. But Narrator's only one of many components that are involved with delivering the customer experience.

This is the stack that I usually care about:

  • Narrator, the UIA client app.
  • UI Automation itself.
  • The UI Framework, which implements the UIA provider API on behalf of the product.
  • The UI implemented by the product developer

 

My first step when investigating the problem with the menu item interaction, is to try to remove Narrator from the equation. If I can do that, then that helps pinpoint where the root cause of the bug may lie.

Narrator is a UIA client app, and so interacts with the product UI of interest through the UIA client API. So if I point another UIA client app at the product UI, and that other UIA client app behaves exactly as expected, then that might suggest the problem lies with Narrator. But if instead, the other UIA client app also struggles at the product UI, then that might suggest I should focus on the product UI, rather than on Narrator.

The bug relates to how Narrator reacts to a change in state of the product UI. In order for narrator to react to a change in the state of UI, the UI must raise a UIA event to make Narrator aware of the change. If no event is raised, then Narrator isn't made aware of the change, and can't make your customer aware of the change. For a one-minute introduction into UIA events, check out Part 4: UIA Change Notifications.

So the next question is: Did the appropriate UIA event get raised when the menu item was checked?

The AccEvent SDK tool is a UIA client app, and can report details of UIA events being raised by UIA. So I'll point the tool at the UI, and check whether a ToggleStatePropertyChanged event is being raised when I select the last item in the menu.

The following image shows what I would have expected to have been reported in AccEvent if everything was working as it should.

 

Figure 5: The AccEvent SDK tool reporting a UIA ToggleStatePropertyChanged event being raised by a menu item.

 

However, it turned out that there was no property changed event reported by AccEvent when the menu item was checked. As such, this seemed to suggest that this bug isn't caused by Narrator. Rather the UI isn't making Narrator aware of the change.

 

Step 3: Why isn't the UIA event being raised?

Now, this is where my investigation became a real learning experience for me.

Traditionally I've found that if AccEvent doesn't report a UIA event, then the event wasn't being raised. So I then ask the product teams for details on exactly how the UI is implemented. In desktop UI, if a standard control is being used from the Win32, WinForms, WPF or UWP XAML frameworks, then a UIA ToggleStatePropertyChanged event would be raised by the UI framework as required. And for web UI hosted in Edge, if the UI is defined using semantic, industry-standard HTML, (including in this case making sure it has a role of "menuitemradio" and uses "aria-checked"), then when an element is checked, Edge would raise the expected UIA ToggleStatePropertyChanged event.

And often during a bug investigation such as this with web UI, we'd examine how the UI was defined and find it didn't use semantic, industry-standard HTML. Rather the UI was built to react visually to customer interaction, but not programmatically. Consequently, the product team would update their UI to use semantic, industry-standard HTML, and the bug would be resolved. Jolly good.

In this particular case the UI was defined in HTML, but by a UI library not created by the product team. So it wasn't quite as straightforward for us to investigate what might be happening.

 

Step 4: Did the UI change far more than it seemed when its state change?

I'm going to confess that at this point, I was stuck. I flailed around somewhat, trying to think of any way to make progress on this. But after a while, I did discover one critically important piece of information, which at least unblocked the product team. So I'll jump to the useful ending of the story here…

In my experience, if AccEvent doesn't report a UIA event, then the event wasn't raised. (That's assuming I set the AccEvent to report the events of the appropriate type and scope, in its Settings UI.) But what if AccEvent did receive the event from the UI, yet didn't report it? When AccEvent receives an event, it goes back to the sender of the event to learn more about the sender. AccEvent will gather up details of the sender, and display those details in its UI. Perhaps if AccEvent were to have problems gathering up the sender's details after receiving the event, then the event won't get reported?

So, say the menu item raised the expected ToggleStatePropertyChanged event. Having done that, the product then destroys that menu item. It then creates a new menu item, in the checked state, and inserts the new menu item into the menu, in the same place that the previous menu item was. While this is happening, a UIA client app listening for events, (such as AccEvent or Narrator,) receives the event, and returns to the source element to gather details about it. The attempt to get those details fails, because the sender's been destroyed. As such AccEvent and Narrator both discontinue the attempt to react to the event. And given that the new menu item was always checked once inserted into the UIA tree, no ToggleStatePropertyChanged event was raised by that new menu item.

That hypothesis seemed rather unlikely to say the least, but it would match the results that we were experiencing.

So to pursue that line of thought further, we needed to know whether the UIA element representing the menu item of interest was the same UIA element both before and after the change in checked state. Comparing UIA properties such as the Name or AutomationId wouldn't help here, as they're very likely to be the unchanged throughout. But if the UIA RuntimeId property has changed, then it's not the same element. (I mentioned something about RuntimeIds a while back, at Don't use the UIA RuntimeId property in your Find condition.)

So I pointed the Inspect SDK tool at the menu item when the item was not checked, and then again once it had become checked. And having done so, I found the RuntimeIds were different. For example, in one run-through, the RuntimeId changed from "2A.80A44.4.816C", to "2A.80A44.4.81C5".

 

Figure 6: The Inspect SDK tool reporting the UIA RuntimeId property for a menu item.

 

At this point, it seemed that we knew enough to follow up with the owners of the library which created the menu UI. I strongly suspect that the action being taken to replace the menu item UI during the interaction, is leading to UIA clients like Narrator not being able to make the customer aware of the change in state at the time the change happens. Whether the ultimate resolution is to have the library updated to not recreate UI in this way during customer interaction, or to replace the use of the library with semantic, industry-standard HTML which simply checks or unchecks an item which lives through the interaction, I don't know. But at least the product team is unblocked.

 

Summary

This investigation has been a reminder of a couple of things:

  • The visual representation of UI is no indication of the programmatic representation. Our customers require both representations to be a good match for the meaning of the UI, but depending on how the UI is implemented, that requirement might not be met by default, particularly for web UI. So be sure you're familiar with both representations.

     

  • Before considering leveraging any library to present UI in your product, where the action taken by the library is outside of your control, always be sure that the UI that the library provides is fully accessible. Whether that UI is a menu item or a full-blown complex chart, you don't want to risk shipping UI that's inaccessible. Your customers won't care that it's some library UI hosted in your product that's blocking them. All they care about is that they can't use your product.

     

And by the way, while this particular investigation has involved web UI, the principles apply to any UI. For example, say you want your UWP XAML app to make your customers that are using a screen reader aware of some important change in the app's UI. You call FrameworkElementAutomationPeer.FromElement() to get the AutomationPeer associated with a control, then call that AutomationPeer's RaisePropertyChangedEvent(), and then for some reason destroy the control. If Narrator or any other UIA client quickly comes back to the app to learn more about the UI that raised the event, it's not going to be able to do much that's helpful to your customer if that UI's already been destroyed.

 

'Til next time.

Guy


Comments (0)

Skip to main content