An investigation into why standard UI wasn’t raising a UIA event needed by Narrator


This post describes the approach taken when investigating why the Narrator screen reader was not reacting as expected to a change in state of some UI. While in this case the UI was a Win32 checkbox, some of the steps described could be useful when investigating Narrator behavior regardless of the control type being used.

 

Introduction

A few weeks ago I was told that the Narrator screen reader wasn’t reacting as expected when a checkbox in a particular UI was toggled. The expected behavior is that when Narrator’s cursor is located at a checkbox, and the toggle state of the checkbox changes, Narrator should make an announcement related to the toggle state change. Instead in the case of this particular UI, Narrator said nothing.

The following sections describe the investigations that took place in order to solve this mystery.

 

The Investigation

1. Reproducing the problem

One of the first things I like to do in a situation like this is reproduce the problem on my own device. Once I can do that, I can relax in the knowledge that I can spend a while pointing SDK tools to the UI, and not feel pressured that I may lose access to the device which reproduces the problem.

By reproducing the problem myself, I can also be sure that there are no required steps or software/hardware configuration that I hadn’t been made aware of. For example, when someone says they “clicked” a button, it might be critical as to exactly what input mechanism was used. For example, did they use a mouse, touch, keyboard, or invoke it programmatically through Narrator touch input or speech input?

Sure enough, I ran the feature on my own device and pointed Narrator to the UI checkbox of interest, and Narrator said nothing as I changed the toggle state of the checkbox. So it repro’d fine for me.

 

2. Considering how much of a problem this is to the customer

Having repro’d the problem on my device, I then took some time to consider how severe this problem is to the customer. After all, we all have a long list of things that we need to work on, and we need to feel confident that we’re working on the things that have most impact to our customers.

As a test, having changed the toggle state of the checkbox, I pressed Tab and Shift+Tab to move Narrator away from, and back to the checkbox. When Narrator returned to the checkbox, the current toggle state of the checkbox was announced as expected. Similarly when the Narrator cursor was on the checkbox, I could press the CapsLock+D keyboard shortcut to have Narrator announce details of the UI, and again the current toggle state of the checkbox was announced.

This means that there is a way for the customer who’s using Narrator to learn of the toggle state of the checkbox, despite an announcement not being made at the moment when the toggle state changes. Maybe that’s ok?

By default, an issue like that is not ok.

A sighted customer who changes the toggle state of the checkbox is informed immediately through visual feedback that the state has changed. That customer doesn’t have to take additional action such as tabbing or clicking away from the checkbox and then back to it, before being informed that their action at the checkbox was successful. So why would it be ok for a customer who’s using Narrator to have to take such action?

Perhaps if after further investigation, we discover that for some unexpected reason it would actually be a huge pile of work to fix the Narrator experience at this particular checkbox, and so further discussions around the impact of the issue might be justified. But until we have more information, we assume that we will be fixing this.

 

3. Is this checkbox really the issue?

Ok, so I know at this point that I can repro the problem with Narrator at the checkbox on my device. But what if the problem isn’t the checkbox, but rather Narrator? Or perhaps something between the two? For example, the UI framework associated with the checkbox, or maybe Windows UI Automation (UIA).

Actually, on a side note, in general for an issue like this, UIA is rarely a part of the problem. UIA channels the data associated with the checkbox UI from the “provider”, (in this case whatever exe is hosting the checkbox,) to the “client”, (in this case Narrator,) and typically works exactly as expected. So the other three areas of the control, the UI framework, and Narrator are always initially where I’m most interested.

So in order to get a feel for whether the issue really seems to lie with the checkbox in question, I want to point Narrator to another “similar” checkbox. Maybe I’ll find that all such checkboxes exhibit the unexpected behavior.

The question of what a “similar” checkbox is here is really important. Often a big proportion of the accessibility of a control is provided by the UI framework being used to present the control. This UI framework might be XAML, Edge, Win32, or something else. To involve multiple UI frameworks in my investigation at this point, will complicate things unnecessarily. Instead, let’s just limit things to one UI framework for the moment.

I had a pretty good idea what UI framework was being used for the checkbox of interest, but I never like to make assumptions which could result in me losing time if I assumed wrong. So I pointed the Inspect SDK tool at the checkbox, and took a look at the FrameworkId and ProviderDescription properties. The FrameworkId was reported to be “Win32”, and the very complicated ProviderDescription string contained the text “MSAA Proxy”. From this, I could tell that the checkbox was built with Win32.

A few comments on the ProviderDescription property associated with UI built with other UI frameworks are in the section “A tip on learning where UIA data is coming from” of Steps for customizing the accessible name of a standard control, for five UI frameworks.

Having established that the checkbox was built with Win32, I then looked for another Win32 checkbox on my device. I picked one from the classic Mouse Properties control panel, and verified using the Inspect SDK tool that its FrameworkId and ProviderDescription were similar to that of the original checkbox.

I then pointed Narrator at the control panel’s checkbox and toggled the state of the checkbox. When I did that, Narrator announced the toggle state change just fine. While that doesn’t prove that the issue I’m investigating lies with the original checkbox rather than the UI framework or Narrator, it does make me feel that it’s appropriate for me to continue focusing for the time being on the original checkbox itself.

 

The Inspect SDK tool reporting the FrameworkId property of a checkbox in the classic Mouse Properties control panel.

Figure 1: The Inspect SDK tool reporting the FrameworkId property of a checkbox in the classic Mouse Properties control panel.

 

Bonus: Check out some of the helpful properties in the classic Mouse Properties control panel. Settings such as “Display pointer trails” and “Show location of pointer when I press CTRL key” can really help draw your attention to where the mouse cursor is.

 

4. Is the checkbox letting Narrator know that something needs to be announced?

When something changes in UI, Narrator needs to be informed of the change. Otherwise Narrator can’t react by making a helpful announcement to the customer. Narrator’s not going to constantly poll for changes in the UI, rather it will react to UIA events being raised by the UI. One of the most fundamental events is the FocusChanged event. This is raised whenever keyboard focus moves from one element to another, and so Narrator can react to that event to let the customer know what element has gained keyboard focus.

So the next question is, when the toggle state of the checkbox of interest changes, is a UIA event raised to let Narrator know about the change?

In order to learn the answer to that, we can run the AccEvent SDK tool. This tool is a UIA client app, (like Narrator is,) and can report what UIA events are being raised by UI. In this case, we’re interested in an event that lets us know that the ToggleState property, (from the UIA Toggle pattern,) has changed.

Note: UIA Control pattern property ids often involve a concatenation of the pattern name followed by the property name, so that can result in some duplication in the ids’ names. For example, the id of the ExpandCollapseState property from the ExpandCollapse pattern is “UIA_ExpandCollapseExpandCollapseStatePropertyId”. Technically that makes sense, but it can seem pretty weird when you first encounter it.

The id of the property we’re interested in with the original checkbox is the UIA_ToggleToggleStatePropertyId, because it’s the ToggleState property from the Toggle pattern. So I ran the AccEvent SDK tool, and set it up to report all ToggleToggleState property changes. I also set it up to report FocusChanged events, given how much impact that event has on the Narrator experience, and I could verify that FocusChanged events were being raised as expected when I tab to and away from the checkbox.

 

The AccEvent SDK tool’s setup window, showing that FocusChanged events and ToggleToggleState property changed events will be reported.

Figure 2: The AccEvent SDK tool’s setup window, showing that FocusChanged events and ToggleToggleState property changed events will be reported.

 

With the AccEvent SDK tool now reporting the events I’m interested in, I return to the checkbox, tab to it, and press the spacebar a few times to see what events are raised.

Crucially, as I toggle the state of the checkbox with the keyboard, the visuals on the screen change to reflect the new state of the checkbox, but no UIA ToggleToggleState property changed events are raised. This explains why Narrator doesn’t react to the change in toggle state at this checkbox.

 

5. Compare the AccEvent SDK tool output with a working checkbox

Ok, based on what I just did, I now believe that the checkbox of interest isn’t raising the UIA ToggleToggleState property changed event. But to be really thorough, I want to check what happens if I repeat the AccEvent test with a checkbox where Narrator works as expected. After all, maybe I botched the AccEvent setup, and it wasn’t really set up to report the event I’m interested in.

So I pointed the AccEvent SDK tool to the checkbox in the classic Mouse Properties control panel. When I then changed the toggle state of that checkbox, AccEvent did report the ToggleToggleState property changed events. This explains why Narrator does react to the change in toggle state at this checkbox.

 

The AccEvent SDK tool reporting ToggleToggleState property changed events being raised by a checkbox.

Figure 3: The AccEvent SDK tool reporting ToggleToggleState property changed events being raised by a checkbox.

 

6. Comparing the original checkbox with a working checkbox

Once we reach a point where we find one control works as expected with Narrator, and another control doesn’t, it’s always interesting to try to identify some difference between the controls which might relate to the different Narrator experiences. We already know the two checkboxes that we’ve examined so far are both Win32 controls, so what else could be different?

So I next pointed the Inspect SDK tool to the two checkboxes, to see if anything in the long list of UIA properties seemed different. I found no interesting differences. Many important properties, such as those listed below, were identical across the two controls.

 

ControlType: UIA_CheckBoxControlTypeId

IsEnabled: true

IsOffscreen: false

IsKeyboardFocusable: true

HasKeyboardFocus: true

FrameworkId: “Win32”

ClassName: “Button”

IsTogglePatternAvailable: true

 

This was a little surprising to me, as often there’s some difference found in the list of properties when exploring why Narrator behaves differently at two controls.

Given that the Win32 checkbox has its own hwnd, I then decided to run the Spy++ tool available through Visual Studio’s Tools menu. Who knows, maybe there’d be some difference in the window styles which might be related. Overall the checkboxes’ styles were mostly the same. I did notice one had BS_CHECKBOX while the other had BS_AUTOCHECKBOX, and wondered if that might be related to the difference in the UIA events raised by the checkboxes. But that seemed really unlikely, otherwise Narrator would be hitting this problem more frequently than it apparently is.

 

7. Experimenting with my own checkbox

At this point, I felt I needed to experiment with my own brand-new checkbox, to see if I could break it in the same way that the checkbox of interest is apparently broken. It seemed unlikely that simply comparing further the two checkboxes that I’ve been working with so far, would help me figure out the problem. But by creating my own checkbox, I could modify the checkbox button styles in the resource file however I wanted, and see if I could eventually replicate the problem with the Narrator experience.

And to be clear, this isn’t a Narrator problem as such. What I’m really trying to do is get to a point where my own checkbox stops raising the UIA ToggleToggleState property change event when I toggle it.

Note: The step of creating the simplest standalone app which repro’s a problem is something that I expect to have to do periodically. If I’ve spent months updating an app, and then find something I leverage from another team doesn’t work as expected in my app, it’s not helpful if I go to the other team and say “Your stuff doesn’t work”. It’s far more helpful if I create a tiny standalone app, and repro the interaction issue with the other team’s feature with that tiny app. The other team can then more efficiently investigate the issue without wondering how the months of updates I put in my original app might be related to the issue.

Whenever I need to create the simplest app I can, of any UI framework, I go to Visual Studio, and do New->Project, and pick the framework I’m interested in. In this case I picked “Visual C++”, “Win32 Project”. I then modified the Help dialog box in my app, such that it showed a checkbox.

 

A dialog box showing a test checkbox.

Figure 4: A dialog box showing a test checkbox.

 

Now that I have my own experimental checkbox, I need to verify that the UIA event of interest is raised as expected when I toggle the state of the checkbox. Once I’ve done that verification, I can then start working on incremental changes to see if I can break the checkbox. To my great surprise when I pointed the AccEvent SDK tool at my new checkbox, I found that no ToggleToggleState property changed event was raised when I toggled the checkbox. So my brand-new shiny checkbox was broken from the start, and any customer using Narrator wouldn’t be notified when they change the state of the checkbox in the app.

This was most unexpected.

After this discovery, I was no longer focusing on the original problem checkbox. Instead I’d focus on my own checkbox, (on which I’d done no customization,) which wasn’t raising the expected UIA event, and on the checkbox in the classic Mouse Properties control panel, which did raise the event.

My first step was to make sure the button styles, (such as BS_AUTOCHECKBOX,) were identical between the two checkboxes. Yet nothing I could do with the button styles got my checkbox to raise the event I needed.

My next thought was perhaps the flags passed into the InitCommonControlsEx() function might be different between the apps hosting the two checkboxes. After all, a Win32 checkbox is a common control. It turned out that my new app wasn’t making a call to InitCommonControlsEx(), so I added the call, and tested it with a variety of flags. Still I couldn’t get the required UIA event to be raised.

 

8. The solution

There was only one more thing I could think of trying now. When using Win32 common controls, it’s possible to specify which version of the common controls you want to use. Whenever I specify that I want a particular version used, I always say I want version “6”. I’ve never looked into the full set of differences between versions of the Win32 common controls, but no doubt Common Control Versions contains some very interesting information.

I took the standard steps for adding a manifest to my Win32 app, and updated it with the XML shown below, (copied from Enabling Visual Styles).

 

<dependency>
    <dependentAssembly>
        <assemblyIdentity
            type=”win32″
            name=”Microsoft.Windows.Common-Controls”
            version=”6.0.0.0″
            processorArchitecture=”*”
            publicKeyToken=”6595b64144ccf1df”
            language=”*”
        />
    </dependentAssembly>
</dependency>

 

 

When I then ran the updated app, and set up the AccEvent SDK tool again to report events being raised, I found the ToggleToggleState property changed event was now raised as required. And having verified that the event was being raised, I pointed Narrator to the checkbox, and found Narrator now did announce the change in toggle state as I interacted with the checkbox.

This meant that I could get back to the team who owned the UI containing the original problematic checkbox, verify that they weren’t using version 6 of the Win32 common controls, and let them know that if they update their UI to use version 6, it’s very likely that Narrator will announce changes to the checkbox toggle state as required. And it so happens, they did just that and all was well.

 

9. Why the solution works

This investigation had been an interesting journey for me, and I was curious as to why the version of the common controls made a difference to which UIA events were raised by the UI.

The Win32 common controls do not natively support the Windows UI Automation API. (More recently-introduced UI frameworks such as XAML and Edge do natively support the API.) So the question is, how can a UIA client app like Narrator interact with the Win32 common controls? In general, this is made possible by UIA itself. UIA will recognize that a Win32 checkbox doesn’t natively support the UIA API, and will interact with the UI through other means, including the use of WinEvents. WinEvents are an old technology available only on the Windows desktop platform, (so not on Windows Phone, X-Box or Hololens). Modern features can leverage UIA’s rich set of events supported on many platforms, rather than using WinEvents, but in some cases when UIA’s APIs for raising events aren’t being called by UI, UIA can react to WinEvent-related functions instead, and in response will go on to call UIA clients’ event handlers.

However, this isn’t practical in the case of a Win32 checkbox being toggled. When a Win32 checkbox is toggled, the checkbox raises an event to say something changed, but isn’t specific about what changed. This means it isn’t practical for UIA to make a one-to-one mapping to a UIA event like the ToggleToggleState property changed event. Because of this, long ago the checkbox common control was updated to explicitly raise a UIA ToggleToggleState property changed event. That update is included in version 6 of the common controls, and so any checkbox built with an earlier version of the Win32 common controls will not raise the event that the customer using Narrator needs when interacting with the checkbox.

 

10. How to raise the required event yourself

Say for some reason it wasn’t practical for the UI with the original problematic checkbox to be changed to use version 6 of the common controls. Our goal of delivering a high quality Narrator experience with this UI still stands, so what else can we do to fix the problem?

The core problem here is that the expected UIA event isn’t being raised. So if the UI framework that we’re using won’t do it for us, can we do it ourselves? With this in mind, I returned to my experimental new app, and tried to raise the UIA event myself whenever the state of the checkbox changed. Before trying this, I needed to make sure that the app was in its original broken state, and I use AccEvent to verify that the required event wasn’t being raised as I toggled the state of the checkbox.

 

The first change I made was to include the main UIA header file in the app’s existing stdafx.h.

 

#include <objbase.h>
#include <UIAutomation.h>

 

 

I then added the following to the existing WM_COMMAND handler for the About dialog box in the app.

 

case WM_COMMAND:

    if ((HIWORD(wParam) == BN_CLICKED) &&
        (LOWORD(wParam) == IDC_MYCHECKBOX))
    {
        NotifyWinEvent(
            UIA_ToggleToggleStatePropertyId,
            (HWND)lParam,
            OBJID_CLIENT,
            CHILDID_SELF);

        return 0;
    }

 

I have to say, it really is pretty handy that I can pass the required UIA property id into the NotifyWinEvent() function, even though that function was introduced long before UIA existed. Note that I’m not including any details in the call about the new toggle state of the checkbox. Instead I’m just raising an event to say the toggle state has changed, and UIA will react by raising the required UIA event and will include the new state of the checkbox in the call that ultimately gets made to the client’s property changed event handler.

Note for XAML devs: A standard XAML Checkbox control will automatically raise ToggleToggleState events as your customers interact with your checkbox. If for some reason you’ve built custom UI which the user can toggle, but which doesn’t raise the event by default, you’d get an AutomationPeer for the control, (perhaps using FrameworkElementAutomationPeer.FromElement), and then call RaisePropertyChangedEvent(). In general it’s preferable to base any custom UI which can be toggled, on a XAML control which automatically raises all related UIA events for you.

 

Summary

If Narrator doesn’t announce a change in your UI as expected, consider the steps listed below during your investigation.

 

– Make sure you can reproduce the problem yourself.

– Compare the Narrator experience at other similar UI.

– Examine the UIA events raised by the UI where the problem occurs.

– Examine the UIA events raised by similar UI where the problem does not occur.

– Use the Inspect SDK tool to compare the UIA properties of UI where the problem does occur, with UI where it does not.

– Build a minimal UI of the same UI framework, and try to modify it to either fail or not fail in a similar way to the UIs you’ve examined so far.

 

In some cases by running through the above steps, an implementation detail might be revealed which when included in your own UI, will lead to the desired Narrator behavior.

And if you’re presenting Win32 checkboxes, use version 6 of the common controls if that’s practical. Otherwise you’ll want to raise the ToggleToggleState property changed event yourself in order to deliver a high quality experience to your customers who use Narrator.

Guy


Comments (0)

Skip to main content