Narrator announcing an item’s status

Article
01/02/2017

This post discusses ways to use the UIA ItemStatus property in your app’s UI to have some current status exposed by an element. The details included below reflect the behavior of the Windows platform as it stands at the time this post is written.

Introduction

A couple of weeks ago I was chatting with a dev who wanted to have the Narrator screen reader announce a UI element’s status. This status was not represented through the element’s UIA Name property, but was rather a dynamic status affected by some current criteria. So in this situation, it seemed that the UIA_ItemStatusPropertyId might be handy. As MSDN says:

“ItemStatus enables a client to ascertain whether an element is conveying status about an item as well as what the status is. For example, an item associated with a contact in a messaging application might be "Busy" or "Connected".”

This discussion was all the more interesting to me, because it got me thinking how in retrospect, I probably should have used this UIA property when building the UI for the app described at https://github.com/MSREnable/SightSign/blob/master/docs/Accessibility.md. That document details some accessibility-related considerations in the app, and shows how I incorporated connection status as part of the HelpText property of an element. Looking back on things, I really should have used the ItemStatus when conveying the connection status of the robot being driven by the app. Something to remember for next time…

Narrator and the ItemStatus

Narrator considers an element’s ItemStatus to be a part of the element’s “advanced” information. So when your customer issues the command to have the element’s advanced information announced, the announcement will include the ItemStatus.

Similarly, if your customer moves Narrator to an element and leaves it there for a few seconds, then Narrator will automatically announce the advanced info which will include the ItemStatus.

And for some control types, (for example, an Image,) Narrator will include the ItemStatus in the announcement made as it moves to an element.

So there are a few ways of having an element’s ItemStatus announced by Narrator.

Setting the ItemStatus on an element

If the UIA ItemStatus property does seem to have potential to be useful to your customer, below are some ways of setting it on an element.

UWP XAML and WPF

The steps for setting the ItemStatus property in code-behind in a UWP XAML app and a WPF app are identical. So if I have an element whose name is WeatherStatusButton, and I want to set its ItemStatus to “Sunny”, I could do the following*:

AutomationProperties.SetItemStatus(WeatherStatusButton, “Sunny”);

Where AutomationProperties lives in Windows.UI.Xaml.Automation or System.Windows.Automation for UWP XAML or UWP respectively.

*Important: Just as you localize an element’s accessible name, you always localize the ItemStatus.

What’s more, for a UWP XAML app, an initial localized ItemStatus could easily be set in XAML just as the element’s localized Name can be set. That is, simply set the x:Uid in the XAML, and then add the required strings in the localized string resource file. For example:

XAML file

String resource file

<data name="WeatherStatusButton.AutomationProperties.Name" xml:space="preserve">
<value>Weather status</value>
</data>
<data name="WeatherStatusButton.AutomationProperties.ItemStatus" xml:space="preserve">
<value>Rainy</value>
</data>

Having set the initial ItemStatus as above, I can point the Inspect SDK tool to the UI element, and verify that the button’s ItemStatus is “Rainy”.

Figure 1: The Inspect SDK tool showing that an element which visually shows an umbrella, has a UIA Name property of “Weather status” and a UIA ItemStatus property of “Rainy”.

Win32 app

For a Win32 app, you can set the UIA ItemStatus property on an element with an hwnd, by calling the very handy SetHwndPropStr(). (I’ve mentioned this function before at Steps for customizing the accessible name of a standard control, for five UI frameworks and How to have important changes in your Win32 UI announced by Narrator.)

Set the property using the steps below:

At the top of the file:

#include <initguid.h>
#include "objbase.h"
#include "uiautomation.h"
IAccPropServices* _pAccPropServices = NULL;

Somewhere before the property is set:

HRESULT hr = CoCreateInstance(
    CLSID_AccPropServices,
    nullptr,
    CLSCTX_INPROC,
    IID_PPV_ARGS(&_pAccPropServices));

To set the property, (using example values here for string and control ids):

WCHAR szItemStatus[MAX_LOADSTRING];
LoadString(
    hInst,
    IDS_ROBOT_CONNECTED,
    szItemStatus,
    ARRAYSIZE(szItemStatus));

hr = _pAccPropServices->SetHwndPropStr(
    GetDlgItem(hDlg, IDC_BUTTON_ROBOT_CONNECTION_STATUS),
    OBJID_CLIENT,
    CHILDID_SELF,
    ItemStatus_Property_GUID,
    szItemStatus);

When freeing up resources later:

if (_pAccPropServices != nullptr)
{
    // We only added the one property to the hwnd.
    MSAAPROPID props[] = { ItemStatus_Property_GUID };
    _pAccPropServices->ClearHwndProps(
        GetDlgItem(hDlg, IDC_BUTTON_ROBOT_CONNECTION_STATUS),
        OBJID_CLIENT,
        CHILDID_SELF,
        props,
        ARRAYSIZE(props));

_pAccPropServices->Release();
_pAccPropServices = NULL;
}

Having a change in ItemStatus automatically announced

Now, there may be situations when you’d like Narrator to automatically announce a change in an element’s ItemStatus property. This is possible in some scenarios, but not in others. Below are some reasons why you might find that Narrator does not announce a change in Itemstatus.

Where’s keyboard focus?

The UIA ItemStatus property is a property that Narrator will monitor, but only on the element which has keyboard focus. The reason for that is that Narrator tries to minimize the distractions that your customer encounters. So if a bunch of elements are raising ItemStatus property changed events, the user probably mostly cares about the status of the element which they’re currently interacting with.

So if Narrator receives an ItemStatus property change event, and the element doesn’t have keyboard focus, (or more specifically, if its UIA HasKeyboardFocus property is not true,) then the new ItemStatus will not be announced.

Note that this is not the same as Narrator checking whether the element raising the event is the element where the Narrator cursor is, but rather whether the element has keyboard focus. The Narrator cursor can lie at an element that does not have keyboard focus.

Is a UIA ItemStatus property changed event being raised?

Say I move keyboard focus to an element, and while that element has keyboard focus, its ItemStatus property changes. If Narrator does not announce the new ItemStatus, I need to figure out if no UIA event was raised by the app, (and so Narrator was not made aware of the change), or the event was raised and Narrator chose not to react to it.

In that situation, I always point the AccEvent SDK tool to the UI. By pointing the AccEvent and Inspect SDK tools to UI, I can verify that the programmatic interface seems to be behaving as expected, without adding Narrator to the mix.

When I tried this with the UWP XAML and the WPF app, I found no ItemStatus property changed event was raised. So I could address that by adding the code below, and explicitly having the app raise the event itself. I would always raise the event after I’d set the new ItemStatus property on the element.

AutomationPeer peer =
    FrameworkElementAutomationPeer.FromElement(WeatherStatusButton);
if (peer != null)
{
    // Important: Localize these statuses!
    peer.RaisePropertyChangedEvent(
        AutomationElementIdentifiers.ItemStatusProperty,
        “Sunny”,
        “Rainy”);
}

When I tried this out in my test app, I set a timer to change the status every 3 seconds. The screenshots below show the AccEvent settings window being set up to have AccEvent report the ItemStatus property changed events, and the output in the AccEvent main window with the ItemStatus events being reported.

Figure 2: The AccEvent SDK tool’s Settings window, with the ItemStatus property list item highlighted.

Figure 3: The AccEvent SDK tool showing the new ItemStatus associated with ItemStatus property change events.

For Win32 apps, I could get the ItemStatus changed event raised using the old NotifyWinEvent() function. This is an interesting function in that not only can it be used to raise the WinEvents it was originally designed for, but also UIA events which were introduced long after the NotifyWinEvent() function came into existence.

There’s a lot of Information at Event Constants and Allocation of WinEvent IDs around what ranges of events ids can be passed into NotifyWinEvent(). The UIA_ItemStatusPropertyId property id falls into the valid range of UIA property ids, so the following call can be made to raise the id.

NotifyWinEvent(UIA_ItemStatusPropertyId, hwndControl, OBJID_CLIENT, CHILDID_SELF);

Note: It seems that a table up at one of the pages I referenced above is inaccessible. I’ll look into what can be done about that.

Is the status change announcement being interrupted?

Sometimes Narrator might start preparing to announce an ItemStatus property change event, but then another event arrives whose related announcement interrupts the ItemStatus announcement. This interruption might happen so quickly that the ItemStatus announcement never even started to be made. So if you’ve used the AccEvent SDK tool to verify that the ItemStatus property changed event has been raised, it’s worth then checking what other events are being raised soon after the ItemStatus event.

Another important potential cause of the announcement interruption is key echo. Your customer might well have key echo enabled in Narrator, and so if an announcement is about to be made and your customer presses a keyboard key, the key echo announcement could mean that the earlier announcement is never heard. This is particularly relevant for situations where the element raising the event is an editable text control. Your customer might be typing into a field, and the app decides the status of the field has changed based on the contents of the field, so the edit field element raises the ItemStatus property changed event. It’s quite possible that your customer’s typing so quickly that the key echo announcement for the next key press prevents the ItemStatus change from being heard.

Summary

If you have an element that exposes some status visually, consider whether the UIA ItemStatus property is the most appropriate way of exposing that status programmatically. I’m pretty sure I’ll be using that property more in the future.

Guy