A real-world example of quickly building a simple assistive technology app with Windows UI Automation


This post describes one example of how the Windows UI Automation API helped when building a simple tool for people with low vision.

 

Introduction

I said recently at Can UIA help you build a tool for someone you know? that the Windows UI Automation (UIA) API can help you build a solution which helps people who have specific challenges using a computer. Since uploading that post, I was contacted by someone in the Netherlands who works with people with low vision, and he asked whether I could build an app which highlights where keyboard focus is. He was after a free, standalone app, and felt that there was nothing available to him which exactly matched what he was looking for.

This definitely seemed worth investigating.

So I grabbed a couple of hours early the next morning to build a V1.0 of the app, and made it available to the person who contacted me to get his feedback. Within a week, I’d made some updates based on that feedback, and completed a V1.2. The app’s now at a stage where more people can try it out, and over the coming months I can keep tweaking it to make it as useful as I can. The app has some known constraints which would take me a while to address, so I’ll hold off updating the app until I know exactly what’s most important to the people using it.

Overall it’s been a fun week, building a new tool which has potential to help a lot of people. And UIA has made this possible.

 

The Herbi HocusFocus app

Figure 1: The Herbi HocusFocus app highlighting the element with keyboard focus.

 

Building the new app

The new app is a C# WinForms app. (It’s not a Universal Windows Platform app, because Universal Windows Platform apps can’t use the UIA client API). I chose WinForms for the UI, because it’s really quick for me to build UI with WinForms. So it didn’t take much time to create the app UI, and Visual Studio made it easy for me to build Dutch and German versions of the app. And of course, I mustn’t forget to add localized accessible names to the comboboxes shown in the app.

 

Using Visual Studio to add an accessible name to a combobox.

Figure 2: Adding an accessible name to a combobox shown in the new app.

 

The highlight presented by the app is simply another form, whose position and size is set based on the bounding rectangle of the element with keyboard focus. The form has a transparency key color set to make the insides of the form transparent. The customer only sees a rectangle at the form’s border, and they can customize the rectangle’s color, thickness, margin and style to whatever works best for them.

I also added functionality to optionally have the accessible name of the element that gets keyboard focus spoken. This provides a very helpful feature with a tiny amount of code. In fact I’ve added the following code to a bunch of apps over the years:

 

using System.Speech.Synthesis;

private SpeechSynthesizer _synth;

_synth = new SpeechSynthesizer();

_synth.SpeakAsyncCancelAll();

_synth.SpeakAsync(strName);

 

The UIA stuff

Now we get to the stuff of most interest to this blog, and that’s how to use UIA to know what’s happening to keyboard focus as the customer moves focus around the screen.

As I usually do, I’m using a managed wrapper around the native Windows UIA API, and the wrapper is generated using the tlbimp.exe tool. Some notes on generating the wrapper are at So how will you help people work with text? Part 2: The UIA Client.

I created the following class to perform all interaction with UIA. Hopefully the comments adequately describe what it does.

 

using System.Diagnostics;
using System.Drawing;

// This namespace is available through the managed wrapper generated by the
// tlbimp.exe tool.

using interop.UIAutomationCore;

namespace HerbiHocusFocus
{
    // Add support for IUIAutomationFocusChangedEventHandler in order to get
    // notifications when keyboard focus changes.

    internal class FocusHandler : IUIAutomationFocusChangedEventHandler
    {
        private IUIAutomation _automation;

        // Define a few properties using the values in UIAutomationClient.h.
        // (The managed wrapper generated by tlbimp.exe doesn't expose these
        // in an easy-to-use way by default.)

        private int _propertyIdBoundingRectangle = 30001;
        private int _propertyIdName = 30005;

        private bool _fAddedEventHandler = false;

        private HerbiHocusFocusForm _mainForm;

        // Use a delegate so that we can have the app take its highlighting
        // action on the UI thread, regardless of what thread the FocusChanged
        // event handler is called on below.

        private HerbiHocusFocusForm.EventHandlerDelegate _eventHandlerDelegate;

 

        public FocusHandler(
            HerbiHocusFocusForm mainForm,
            HerbiHocusFocusForm.EventHandlerDelegate eventHandlerDelegate)
        {
            this._mainForm = mainForm;

            this._eventHandlerDelegate = eventHandlerDelegate;
        }

 

        public void Initialize()
        {
            this._automation = new CUIAutomation();

            RegisterFocusChangedListener();
        }

 

        public void Uninitialize()
        {
            UnregisterFocusChangedListener();
        }

 

        private void RegisterFocusChangedListener()
        {
            // Use a cache here, so that the name and bounding rectangle of the element
            // with focus is cached with the FocusChanged event. This means that when
            // the app gets notified of the focus change, it doesn't then have to go back
            // to the element that raised the event to get its name and bounding rect.

            IUIAutomationCacheRequest cacheRequest = _automation.CreateCacheRequest();

            cacheRequest.AddProperty(_propertyIdName);
            cacheRequest.AddProperty(_propertyIdBoundingRectangle);

            // The above properties are all we'll need, so we have have no need for a
            // reference  to the source element when we receive the event.

            cacheRequest.AutomationElementMode =
                AutomationElementMode.AutomationElementMode_None;

            // Now register for the FocusChanged events.

            _automation.AddFocusChangedEventHandler(cacheRequest, this);

            _fAddedEventHandler = true;
        }

 

        private void UnregisterFocusChangedListener()
        {
            if (_fAddedEventHandler)
            {
                _fAddedEventHandler = false;

                _automation.RemoveFocusChangedEventHandler(this);
            }
        }

 

        public void HandleFocusChangedEvent(IUIAutomationElement sender)
        {
            // UIA is notifying us that keyboard focus has moved. If we're in the middle
            // of closing down, don't do anything here.
            if (!_fAddedEventHandler)
            {
                return;
            }

            // Get the cached bounding rect so we know where we should highlight.

            tagRECT rect = sender.CachedBoundingRectangle;

            Rectangle rectBounds = new Rectangle(
                rect.left, rect.top,
                rect.right - rect.left, rect.bottom - rect.top);

            // Get the cached name so that it can be spoken if that's what the customer wants.

            string name = sender.CachedName;

            Debug.WriteLine("Focus now on: " + name);

            // Now have the UI thread highlight the focused UI. This will return immediately.
            _mainForm.BeginInvoke(_eventHandlerDelegate, name, rectBounds);
        }
    }
}

 

And so thanks to those few lines of UIA client code, it’s much more practical for my customers to know where keyboard focus is in a bunch of apps.

 

The Herbi HocusFocus app highlighting a menu item in the Edge browser.

Figure 3: A menu item in Edge being highlighted by the new app.

 

 

Conclusion

Thanks to the power of .NET and UIA, in a few hours over the course of one week, I built the first version of a tool which someone had specifically requested. Based on customer feedback I can update the app to make it more useful in practice. The person who contacted me had said that he’d unsuccessfully searched for such a tool for several years, and so this is a reminder of the difference we can all make in plugging gaps in what’s available to people. And sometimes – it takes very little time for us to do that.

I’d usually say here that I’ll upload the app’s source code somewhere public, but I keep saying that, and then don’t. So I’ll hold off on saying it here. If I ever make the code public, I’ll update this post.

It goes without saying that I’m very grateful to the person who contacted me, querying whether it’d be practical for me to build this tool. And if you know of someone who might find the tool useful, or might find it useful yourself, it’s freely available at Herbi HocusFocus.

 

Good luck with your own assistive technology projects!

Guy


Comments (16)

  1. John Smith says:

    Hi! It seems that you are one of those rare people from MS who are working with Automation API. Since there is no way to get help from MS in conventional manner I will try to ask here.
    Probably there is a bug in IUIAutomation.CompareElements method: it may return false for pointers of the same UI element until switching focus between client and other applications. In other words I am observing finishing cycle for code like this:
    "while (!CompareElements(e1, e2)) {
    }"
    after several iterations before switching focus. Pointers are obtained using a raw walker.
    Is this known problem and is there a workaround?

    (BTW, it seems that automation API is still in experimental state. Too much bugs and unexpected errors)

    1. Hi John, is this happening on Windows 10? If so, could you give me steps for me to try to reproduce the problem myself please? If I can also consistently hit the problem, I'll try to investigate.

      Thanks,

      Guy

      1. John Smith says:

        Hi Guy, it is Win 7 Prof SP1, latest updates are installed, UAC is switched off, etc.
        It will be a little bit difficult to provide the isolated test case immediately - I am using Java testing harness with the bridge to MS Automation API. I will try to provide the isolated c++ sample asap.
        BTW, compared pointers even return the same RuntimeID's...
        Thanks!

      2. John Smith says:

        Hi Guy, it is Win 7 Prof SP1, latest updates are installed, UAC is switched off, tested application is using controls from MS Control Library.
        It will be a little bit difficult to provide the isolated test case immediately - I am using Java testing harness with the bridge to MS Automation API. I will try to provide the isolated c++ sample asap.
        BTW, compared pointers even return the same RuntimeID's...
        Thanks!

      3. John Smith says:

        Hi Gui,
        Sorry for delay. Here is the standalone c++ sample to reproduce the problem. It uses well known TotalCommander as tested application. In my case the main cycle is executed until changing focus from the test app.

        #include "stdafx.h"

        CComPtr createCacheRequest(CComPtr automation_ptr) {
        CComPtr cache;
        HRESULT hr = automation_ptr->CreateCacheRequest(&cache);
        hr = cache->put_TreeScope((TreeScope)(TreeScope_Children | TreeScope_Element));
        CComPtr true_condition_ptr;
        hr = automation_ptr->CreateTrueCondition(&true_condition_ptr);
        cache->put_TreeFilter(true_condition_ptr);
        cache->put_AutomationElementMode(AutomationElementMode_Full);
        return cache;
        }

        int _tmain(int argc, _TCHAR* argv[]) {
        CoInitializeEx(NULL, COINIT_MULTITHREADED);
        int iteration = 0;

        while (true) {
        iteration++;
        CComPtr automation_ptr;
        HRESULT hr = automation_ptr.CoCreateInstance(__uuidof(CUIAutomation), NULL, CLSCTX_INPROC_SERVER);

        CComPtr root_element_ptr;
        hr = automation_ptr->GetRootElement(&root_element_ptr);

        VARIANT prop;
        prop.vt = VT_BSTR;
        prop.bstrVal = SysAllocString(L"TTOTAL_CMD");
        CComPtr class_name_condition_ptr;
        hr = automation_ptr->CreatePropertyCondition(UIA_ClassNamePropertyId, prop, &class_name_condition_ptr);
        CComPtr app_ptr = NULL;
        hr = root_element_ptr->FindFirst(TreeScope_Children, class_name_condition_ptr, &app_ptr);

        prop.vt = VT_I4;
        prop.lVal = UIA_MenuBarControlTypeId;
        CComPtr type_condition_ptr;
        hr = automation_ptr->CreatePropertyCondition(UIA_ControlTypePropertyId, prop, &type_condition_ptr);
        CComPtr menu_ptr = NULL;
        hr = app_ptr->FindFirst(TreeScope_Children, type_condition_ptr, &menu_ptr);

        CComPtr array_ptr1 = NULL;
        CComPtr cache = createCacheRequest(automation_ptr);
        CComPtr updated_menu_ptr = NULL;
        hr = menu_ptr->BuildUpdatedCache(cache, &updated_menu_ptr);
        hr = updated_menu_ptr->GetCachedChildren(&array_ptr1);

        CComPtr file_item_ptr1 = NULL;
        array_ptr1->GetElement(0, &file_item_ptr1);

        CComPtr walker_ptr = NULL;
        automation_ptr->get_RawViewWalker(&walker_ptr);

        CComPtr menu_ptr2;
        walker_ptr->GetParentElement(file_item_ptr1, &menu_ptr2);

        CComPtr file_item_ptr2 = NULL;
        walker_ptr->GetFirstChildElement(menu_ptr2, &file_item_ptr2);

        /*CComPtr menu_ptr2;
        hr = file_item_ptr1->GetCachedParent(&menu_ptr2);

        CComPtr array_ptr2 = NULL;
        CComPtr updated_menu_ptr2;
        hr = menu_ptr2->BuildUpdatedCache(cache, &updated_menu_ptr2);
        hr = updated_menu_ptr2->GetCachedChildren(&array_ptr2);
        CComPtr file_item_ptr2 = NULL;
        array_ptr2->GetElement(0, &file_item_ptr2);*/

        BOOL same;
        hr = automation_ptr->CompareElements(file_item_ptr1, file_item_ptr2, &same);

        printf("%d\n", iteration);
        if (!same) break;
        }
        return 0;
        }

        1. Hi John,

          Thanks for supplying the standalone code. I don't think I'll be able to provide any help in this case, as I'm working with Windows 10 and don't have the TotalCommander installed. I ran your code with Notepad on my machine, and it worked fine. You could be hitting a bug in UI Automation, but my experiment doesn't help us learn whether it's been fixed in some later version of UI Automation.

          While I don't think this will make any difference to the results, I would point out one thing which I've not done myself. The TreeFilter used in the cache request that gets the children relates to what view of the UIA tree is of interest. UIA has 3 views, but I only ever consider 2 of them. These are the Control view, (which should contain everything of interest to the user,) and the Raw view, (which contains everything being exposed through UIA.) So whenever I've set the TreeFilter, I've created a condition which specifies a view. In fact I tend to not set it at all, because I think by default the results come from the Control view, and that's usually what I'm interested in. I doubt using a true condition as you're doing is having any negative effect on your results here.

          Even though you might be hitting a bug in UIA, I wonder if it might be possible to try a couple of things to see if it can be worked around, or to at least learn more about why it's failing. The CompareElements() call will be using the elements' RuntimeIds as part of the comparison. So it might be interesting to learn what those RuntimeIds are.

          One way to learn about the first element's RuntimeId, would be to add it to the CacheRequest in createCacheRequest()...

          hr = cache->AddProperty(UIA_RuntimeIdPropertyId);

          ...and then retrieve it from the cache later. Alternatively we could get the RuntimeId directly from the element with a cross-proc call when we have the element.

          SAFEARRAY* runtimeId1;
          hr = file_item_ptr1->GetRuntimeId(&runtimeId1);

          printf("%d\n", runtimeId1->cbElements);
          for (int i = 0; i cbElements; i++)
          {
          printf("%d\n", *((int*)(runtimeId1->pvData) + i));
          }

          SAFEARRAY* runtimeId2;
          hr = file_item_ptr1->GetRuntimeId(&runtimeId2);

          printf("%d\n", runtimeId2->cbElements);
          for (int i = 0; i cbElements; i++)
          {
          printf("%d\n", *((int*)(runtimeId2->pvData) + i));
          }

          And then see whether UIA considers the RuntimeIds to be the same.

          BOOL sameRuntimeIds;
          hr = automation_ptr->CompareRuntimeIds(runtimeId1, runtimeId2, &sameRuntimeIds);

          The above test might reveal whether one of the runtimeIds isn't accessed as expected before the target app window has been made active and has focus. The actual meaning of the values in the runtimeId is opaque to clients, and the runtimeId is only meant to be used for comparison, (and only during the lifetime of the instance of the app). I know me saying the runtimeId is only meant to be used for comparison isn't useful when you've found a case when the comparison doesn't work.

          I did also notice something unexpected in my own test of this with Notepad, in that the cbElements is 4 for both runtimeIds, yet when I point the Inspect SDK tool to the File menu, it shows a runtimeId with 7 values. In fact it's only the 7th value that's different between the File menu element and the Edit menu element. So as another test I accessed the runtimeIds of the File menu element and the Edit menu elements and called CompareRuntimeIds() with them. That returned saying the runtimeIds are not the same, so it must be looking at all the values in SAFEARRAY, despite cbElements being less that the full count.

          Periodically I have seen some apps' UIA data change depending on what action has occurred at the app. These tend to be apps that do a lot of the work to implement the UIA provider interface themselves. The idea is that they don't do the work unless they think it's necessary. That means it can be possible for something to be not set up as expected if the app hasn't realized it needed to do some UIA-related work. I really doubt that's the sort of thing that's going on in the case you're hitting. That is, the runtimeId on some element isn't being set up until the app is made active or got focus. (In theory the related UI framework might be missing out on taking some action that it should before the app's been made active.) But maybe looking explicitly at the runtimeIds might shed some light on what's going on.

          Thanks,

          Guy

          1. John Smith says:

            Hi Guy,
            Thanks for help.

  2. P says:

    Hey, you mentioned you are considering to make the source public. I was wondering if you could?
    Thanks!

    1. Hi,

      I would still like to make the source public at some point, but last week I did something that's probably going to delay me doing this for a while. I had a request to update the app to highlight where the text cursor is, (aka caret,) while the user's typing. So I had my Surface Pro 2 out while at 38,000 feet over the Atlantic last week, and had a go.

      The end result is that the app can do caret highlighting in some useful places, and I'm hoping that it’s sufficiently useful to some people. But the code's a real mess. I started out using winevents to track caret movement, and that works fairly well in many places. (And I've never written C# interop code for caret winevent handling, so that was interesting.) But winevents are really legacy technology now, (and don’t work on all platforms,) and I should be using things like UIA TextSelectionChanged events to track caret movement. I'm not quite sure why I didn't go with UIA TextSelectionChanged events in the first place. Maybe because as I understand things, UIA doesn't make it trivial to get the caret location and size. Instead you need to get the TextRange for the current selection, and if there is no selection, you need to adjust the TextRange by a character to be able to get a useful bounding rect.

      In retrospect, I should have just gone with only using UIA when I added the feature. Instead I now have a mix of winevent handling and UIA, which I'll need to clean up at some point. I'd not really want to make my current code public, and encourage people to do what I did.

      But that said, I do want to share code from the app if it'd be useful. Is there a particular part of the app that you're interested in? I could post specific parts of it if that would help.

      Guy

      P.S. A demo video of the new caret highlighting feature is at http://herbi.org/HerbiHocusFocus/HerbiHocusFocus.htm. (It's the second video on the page.)

      1. P says:

        There isn't really a specific part. I was just trying to get my head around how to start using UIA. So examples of working apps are always a good way to understand thing for me. So I'm actually looking for a simple setup of how to get UIA events/callbacks so I can start acting on them. I have never programmed anything like this, so I'm currently just trying to get the basics.

        1. junkew says:

          Over the years I build some nice stuff with UIA from windows xp onwards.
          Its written with AutoIT native UIAutomatonCore.DLL
          https://www.autoitscript.com/forum/topic/153520-iuiautomation-ms-framework-automate-chrome-ff-ie/
          Not many people may know but its possible to use it from Visual Basic for Applications in Excel you just have to copy
          UIAutomationCore.dll to your documents folder (no clue why it works)

  3. Dennis R Zweigle says:

    Hi,
    Any updates on making the source avialable?
    Thanks!

  4. Dennis R Zweigle says:

    Hello again. If possible, please use C# (or VB.net) when sharing the source solution-- sorry. i am not a C programmer.

    1. Hi Dennis,

      Sorry, I've not made any progress on sharing the source yet. During this week, I'll consider options on where to share it and get back to you. I need to add more comments to the code to call out things like the use of the winevent for caret tracking. At some point I'd like to explore replacing all that so that the app only uses UIA for tracking both focus and caret, but I won't be able to get to that any time soon.

      The app is a C# app, like most of my Herbi.org apps app. It does use some Win32 interop in order to call functions like SetWinEventHook() and PostMessage().

      Thanks,

      Guy

      1. Dennis R Zweigle says:

        Look forward to see the solution. Can't wait.
        Thanks,

        /dz

        1. Hi Dennis,

          You can download the zipped Herbi HocusFocus solution at http://herbi.org/HerbiHocusFocus/HerbiHocusFocusSource.htm.

          I have to say that the code is far from sample quality. It's taken me a while to get my head round what it's doing, and it wasn't that long ago since I wrote it. It's got lots of poorly and ambiguously named variables and functions, and the functionality is distributed across classes in a pretty haphazard manner. I notice I put a lot of exception catching in too. Some of that might be justified, (eg to handle cases where we're trying to interact with UI and the UI has already gone,) but I'll bet a lot of it could be removed, (or at least tuned to handle specific exceptions).

          If I ever spend time on the app in the future, I would be interested in removing use of winevents, and sticking solely to UIA event handling. Having the current mix of winevents, IAccessible and UIA, makes things much more complicated than I'd like.

          All in all, this was written in a hurry and it shows. That said, I do think it's fantastic how such a little amount of code can provide some very useful functionality to people who want more control over the visuals highlighting where keyboard focus or the text cursor are. While the source isn't exactly easy to follow, hopefully it can help you consider what APIs you might consider using in your own assistive technology solutions.

          Thanks for helping everyone leverage technology.

          Guy

Skip to main content