So how will you help people work with text? Part 3: The UIA Provider

This series of posts describes how you can use UI Automation (UIA) as part of your solution to help people who find some aspect of working with text to be a challenge.

Introduction

A UIA provider is a process that supports some of the UIA provider interfaces, and so allows its UI to be programmatically consumed and controlled through the UIA API. And so for the purposes of this series of posts, the UIA provider is the source of the text that ultimately needs to reach the customer. For example, Word 2013 and the RichEdit control are UIA providers which support the Text pattern provider interface.

Most devs do not have to implement the UIA provider interfaces associated with the Text pattern. Instead, devs are more commonly app builders using a UI framework which implements the UIA provider interfaces on their behalf, or they’re builders of a UIA client apps which access the text through the UIA Text pattern client API.

But where an app or component does manage all the display of the text it shows, it may have to implement the UIA provider interfaces itself. Somewhere along the way someone did that for Word 2013 and the RichEdit control.

Where to start when building a UIA provider

If you need to implement UIA provider interfaces relating to the Text pattern, it would be good to get familiar with other more fundamental interfaces first. And the best way to do that is to step through a couple of UIA provider SDK samples in a debugger. Links to the useful samples are below.

It’s also worth calling out a great series of posts describing how to implement UIA providers, starting at Custom UI Automation Providers in Depth: Part 1. Unfortunately the sample projects referenced in these posts were lost at some point, but the posts are still very helpful regardless.

UI Automation simple provider sample

This sample introduces the most fundamental of all UIA provider interfaces, IRawElementProviderSimple. That interface allows you to customize the accessibility of a specific element being exposed through UIA. The screenshot below show how a colored square in the app UI is being exposed programmatically as a button.

Figure 1: Inspect reporting that the programmatically accessible Name property for the custom area in the sample app is “ColorButton”, and its ControlType is “UIA_ButtonControlTypeId”.

Thanks to the work being done by the sample app, if a screen reader is pointed to that square in the app UI, the customer would hear something like “ColorButton, button”. (Typically the Name property in a shipping app would be something a little more helpful than “ColorButton”. For example, “Set color”.)

Without the work done by the app with the IRawElementProviderSimple interface, the custom area would have no useful Name or Control Type, and UIA would fall back to providing general default values. As a result, the customer using a screen reader wouldn’t know what the purpose of the UI is. The screenshot below shows the Inspect SDK tool reporting those default properties being exposed by the UI.

Figure 2: Inspect reporting that the programmatically accessible Name property for the custom area in the sample app is a meaningless “SimpleProvider”, and its ControlType is “UIA_PaneControlTypeId”.

The sample is additionally helpful in that it shows how the UIA provider API can be used to expose the behaviors of UI programmatically. Just as a UIA client app access UI behaviors through UIA patterns, UIA providers expose behaviors through UIA patterns. The simplest pattern is the Invoke pattern, which allows an element to be programmatically invoked. In this sample app, the custom area in the UI is to behave like a button, so the sample implements the Invoke pattern.

Having the provider support the pattern requires the following steps:

1. The provider inherits from the relevant pattern provider interface, in this case, IInvokeProvider.

2. The provider returns itself when QueryInterface’d for the pattern provider interface.

3. The provider returns itself when its IRawElementProviderSimple::GetPatternProvider() is called for the pattern provider interface.

4. The provider implements all members of the pattern. In this case, that means implementing IInvokeProvider::Invoke().

UI Automation fragment provider sample

Having got familiar with the IRawElementProviderSimple interface, the next step is to learn about “fragments”.  Fragments allow multiple UIA elements to be built into a logical hierarchy in the UIA tree of elements. The previous sample was fine for having a single element contained within the hwnd for the custom control, but using IRawElementProviderSimple alone couldn’t have built a UIA tree of elements, all beneath that one custom control hwnd. For example, say I wanted to build some really cool UI with four distinct, invokable elements within the same hwnd, like this:

Figure 3: Example UI where implementing the IRawElementProviderSimple alone cannot make the UI accessible.

For the example UI above, I’d have to provide the relationship between the UIA elements representing the four squares in the UI. That can be done through the use of UIA fragments.

The fragment provider sample shows how the UIA elements can be built into a hierarchy using the IRawElementProviderFragment interface. One of the most interesting methods on that interface is Navigate(). That allows a UIA client to move between nearby elements in the element hierarchy. For example, to move to the logically previous or next element.

Another important interface used in this sample app is IRawElementProviderFragmentRoot. At the top of each fragment of UIA elements, is the “fragment root”. That element is responsible for allowing UIA to hit-test at a point on the hierarchy and to return the appropriate element.

The UI Automation fragment provider sample builds up a fragment of UIA elements, such that the elements are exposed programmatically as a list, in a similar way to how they’re exposed visually.

Figure 4: The Inspect SDK tool showing the hierarchical relationship between UIA elements exposed in the UIA fragment SDK sample.

Implementing the Text provider

Ok, having got familiar with those three fundamental UIA provider interfaces of IRawElementProviderSimple, IRawElementProviderFragment and IRawElementProviderFragmentRoot, you now want to start experimenting with a Text provider. And a good way to start doing that is to step through another SDK sample, UI Automation document content provider sample.

This sample demonstrates three text-related UIA provider interfaces:

ITextProvider2
ITextRangeProvider
IAnnotationProvider

It also demonstrates the raising of the UIA_Text_TextSelectionChangedEventId event, which lets UIA client apps know when the selection’s changed or the caret’s moved in the text.

Note: When I downloaded the sample, I found I couldn’t build it in VS 2015. I don’t know if that was because my dev machine was in some unexpected state, but in order to build the sample, I created a new VS 2015 app and added the sample code to it. If that’s a genuine issue with the sample, hopefully it’ll get sorted out before too long.

Having built and run the sample, I could use Inspect’s Text Explorer tool to learn how the app’s using the Text pattern and TextRange interfaces to expose the text in the app. I used the tool to access the TextRange beneath the mouse cursor, and then expanded the range to include the word at that point. The screenshot below shows the results when I did that with the mouse cursor hovering over the big word “Section”. That all worked fine, and I could also see the attributes of the text being exposed through the TextRange.

Figure 5: The document provider SDK sample app being examined with Inspect’s Text Explorer tool. 

In fact, having seen the result in Text Explorer, I just pointed my simple app that I described in So how will you help people work with text? Part 2: The UIA Client to see if I could get that app to speak the word beneath the mouse cursor. Sure enough, the simple app said “Section” as expected. This is another example of how I’d not try running my own UIA client code without taking a look at the provider UI with the SDK tools first.

As it happens, I’m not convinced that the document provider SDK sample is bug-free, as I found some text navigation didn’t work as I expected. For example, I could move the Text Explorer word-by-word through the text, but not line-by-line.  But even if there are some bugs to be fixed, I strongly recommend that if you need to implement the UIA Text provider interfaces yourself, to do some debugging through this sample first.

One more interface to spotlight

While there are a few more UIA provider interfaces, there’s really one more that’s important to mention here,  IAnnotationProvider. This interface allows access to properties of annotations, such as the Annotation Type, (eg comment or spelling error), author and date/time. It also allows access to the TextRange with which the annotation is associated. So this is a whole load of useful information about an annotation.

And what’s more, the TextRange’s IUIAutomationTextRange::FindAttribute() allows a UIA client app to search for annotations by type in a TextRange, (by supplying an attribute of UIA_AnnotationTypesAttributeId). So it’s straightforward for a UIA client app to find all (say) comment annotations in a document.

The screenshot below shows the Inspect SDK tool reporting that the UIA elements representing the annotations shown in the sample app have an IsAnnotationPatternAvailable property of true, meaning that the elements claim to support the UIA Annotation pattern. Inspect also reports the properties being accessed through that pattern, and so the sample app is indeed implementing the UIA IAnnotationProvider interface as expected.

Figure 6: The Inspect SDK tool reporting the annotation-related properties exposed by an annotation in the SDK sample app.

Summary

If your feature presents text, and that text is not automatically being exposed through the UIA Text provider API by the UI framework that you use, you’ll want to consider implementing the related UIA provider interfaces yourself. Debugging the related SDK samples is a great way to get a head start on implementing UIA provider interfaces.

Once you’ve made your text programmatically accessible through UIA, your customers who use UIA client apps will be able to efficiently consume and interact with your text. For example, your customers who use assistive technology UIA client apps will be able to selectively move through the text paragraph-by-paragraph or line-by-line, and have the area in which the text lies highlighted in some way by the assistive technology app.

Figure 7: The Narrator screen reader leveraging the Text pattern support in Word 2013 to move through the text in the manner chosen by the customer, and to highlight the text while being spoken.

Devs writing UIA client apps can do all sorts of creative things to access text that’s shown in apps, and to present that text in a way that’s most helpful to their customers. You’ll want your text to be able to fully participate in this information exchange, and to be as helpful as possible to your customers regardless of how they interact with your text.

Guy

Posts in this series:

So how will you help people work with text? Part 1: Introduction

So how will you help people work with text? Part 2: The UIA Client

So how will you help people work with text? Part 3: The UIA Provider