A little UIA Q&A

Hi, over the last six months I’ve been in contact with a developer who builds Assistive Technology (AT) apps which use the Windows 8 UI Automation (UIA) client API. It’s been really exciting for me to see the results of his great work, as those results will be extremely helpful to his customers. It’s also been very interesting for me to see him focus so much on the performance of his app, as that’s an area where UIA’s caching functionality can be a big help.

Given that some of the discussions we’ve had during this time might be of interest to others building AT apps with UIA, with the dev’s permissions I’ve included some of the more general interaction below.

I hope this can help others build lots of exciting UIA client apps too! 



Building a cache

Q. I'm using the unmanaged COM API and building an element array using FindAllBuildCache. I trigger the build whenever the active window changes using SetWindowEventHook implemented in C#.  It works well but builds the element array immediately the new window takes focus, I've been using a delay before the build of the elements array to make sure everything is painted on the window but was trying to find a more reliable window loaded type of event.  Just wonder what technique you would recommend to do this?  So as an example if Windows Explorer becomes the new active window how do I ensure all the UI elements are loaded before building my element array?


A. I would say that it's good that you're using the COM UIA API, rather than the managed API that comes with the .NET Framework. The latter can be handy in some situations, (particularly for devs who aren't familiar with COM), but it can't access some of the data that is accessible through the COM API. It's also good that you're using UIA's caching, given that caching provides a better performance.

The thing that complicates your scenario is that different features will behave differently. They'll present all sorts of different UI, and will use different UI frameworks, eg Win32, WinForms, WPF, and Windows Store apps add more to the mix. There's no single event that you can react to which will indicate that all the UI you're interested in is now on the screen. Browsers will make this even more complicated, because when you navigate to a different page or place on a page in a browser, you might only get a bunch of UI restructuring for the hosted HTML, and no window-related events.

In this situation, I find it interesting to use the AccEvent tool that comes with the Windows SDK. It's not very intuitive to use, but it can be helpful. It can be set up to report UIA events coming from all UI, or just from a particular window. So as a test, I pointed it to an existing (but minimized) Windows Explorer window, and then restored the window such that it was the active window. (AccEvent has no Select All and Clear All when picking which events you want to listen for, which is frustrating.)

This was the output: 

UIA:PropertyEvent [Name = "Address: Computer"] Sender: ControlType:UIA_ToolBarControlTypeId (0xC365), Name:"Address: Computer"

UIA:StructureEvent [StructureChangeType_ChildrenInvalidated = TBI<rtid>] Sender: ControlType:UIA_ProgressBarControlTypeId (0xC35C), Name:""

UIA:StructureEvent [StructureChangeType_ChildrenInvalidated = TBI<rtid>] Sender: ControlType:UIA_ProgressBarControlTypeId (0xC35C), Name:""

(I've removed lots of structure changed events and bounding rect changed events here…)

UIA:PropertyEvent [BoundingRectangle = {l:141 t:682 r:1119 b:1091}] Sender: ControlType:UIA_WindowControlTypeId (0xC370), Name:"Computer"

UIA:FocusEvent Sender: ControlType:UIA_ListItemControlTypeId (0xC357), Name:"OS (C:)"


So in this case, the Window Explorer window raised lots of events relating to its contents being restructured and moving around, (which is what you'd expect when the window's unminimized,) and followed all that with a single focus changed event. There's no event after that saying the window UI has stabilized, and there's no guarantee that at the actual point you get the focus changed event, a feature has really completed the redisplay of its UI. When I tried a similar thing in Internet Explorer, I again got lots of structure changed and bounding rect changed events, with a focus changed event near the end of the list. (I also got events for the text changing in a search field, and a selection changing on the browsers tabs, but that doesn't help us here.)

There are a couple of documented UIA events that sound like they may be of interest, (ie UIA_LayoutInvalidatedEventId and UIA_AsyncContentLoadedEventId), but in practice these aren't often raised, so aren't useful to you.

So all in all, I don't think there's much more you can do than use the delay you already have. If it's very quick to gather the data you need, perhaps you could reduce the delay if it feels too long, and if you find you get more structured changed and bounding rect changed events after you've gathered the data, gather it all up again to account for the changes and re-present it.


Q. I used the COM API because I was having speed issues with the managed API although some of it may have been down to threading in the early days of playing with the API.  I am building the cache on change of active window again for speed issues so that when the user command is called I can display the UI pretty instantly.  If I don't build the cache until the user command event then it can, for example with Internet Explorer, take a couple of seconds to respond although with most UIs is pretty instant. 

You mentioned updating the Cache on bounding rectangle changes and the likes and I notice there is a BuildUpdatedCache method which you need to run on each element.  So if with my elements array I wanted to update my cacheRequest then I have to iterate through the array and update each element presuming it still exists on the window?  I can’t quite work that one out so I have basically just been running FindAllBuildCache on the various window changing events. 


A. Hi, as it happens, It don’t think I’ve ever called BuildUpdatedCache(). I wasn’t avoiding it, but I think whenever I wanted to refresh the data which I was presenting to the user, I called FindAllBuildCache() again, as you’re doing. In the case of an array of elements as you have, I’ll bet that’s definitely the best thing to do. When it comes to performance in things like this, it’s all about avoiding cross-process calls. So if you can make a call which involves (say) one cross-proc call, gathers a load of data once it’s in the other process, and then returns , that’s going to be a lot faster than making a load of cross-proc calls which gather some data in each call. BuildUpdatedCache() is going to involve at least one cross-proc call each time, so if you call it for each element in an array, that’ll be slow. Calling FindAllBuildCache() will make far fewer cross-proc calls, (sometimes only one.) So I would imagine BuildUpdatedCache()  is handy when you’re interested in a particular element. (By the way, if you’re making repeated calls to FindAllBuildCache(), you’ll want to release objects from earlier calls that you’re finished with. I seem to remember it not being obvious as to when some of these objects would be released by UIA and when you needed to explicitly release them, but my samples will demonstrate what I thought was the required action.)



Q. I am trying to implement a bounding rectangle property changed event for any of the descendant elements on a relevant window, e.g. Internet Explorer.  If I build my cache using TreeScope.Descendants I can display my rectangles as shown below.  I am then trying to handle a bounding rectangle property changed event for any of the descendants and seem to be getting sporadic results of when it is triggered when it is not.  Forgetting about threading for now, code is something like this, should this record a change to any of the bounding rectangles of the descendants are am I, as is most likely the case, not implementing correctly? 

The "activeWindow" is just the element for the current active window, in this case:

Internet Explorer (activeWindow = uia.ElementFromHandleBuildCache(Win32API.GetForegroundWindow(), myCache);


int [] propArray = {UIA_PropertyIds.UIA_BoundingRectanglePropertyId};

uia.AddPropertyChangedEventHandler(activeWindow, TreeScope.TreeScope_Descendants,

      myCache, new PropertyChangedEventHandler(activeWindow, win), propArray);

class PropertyChangedEventHandler : IUIAutomationPropertyChangedEventHandler
    numberedWindow myWindow;

    public PropertyChangedEventHandler(IUIAutomationElement sender, numberedWindow win)
        myWindow = win;

    public void HandlePropertyChangedEvent(IUIAutomationElement elem, int propertyID, object obj)
        if (propertyID == UIA_PropertyIds.UIA_BoundingRectanglePropertyId)
            MessageBox.Show("Property changed", "", MessageBoxButton.OK, MessageBoxImage.Error);


When I say behaving sporadically, as an example if I zoomed in on the window below the event would be triggered but with other webpages it isn't and if I scroll down the page it isn't.  I am presuming if I scroll down the page that the bounding rectangle property should change.  I basically think the problem is that the event is not being fired for all the bounding rectangle changes of the descendants.


A. I wouldn't assume there's anything not correct in your implementation. The Windows UIA API is a powerful tool, but every now and again some unexpected behavior is unearthed. It's always possible that UIA has some issue, and all we can do is try to find a workaround.

In the case you've described, it is interesting that it seems that sometimes you get the events and sometimes you don't. As you point out, one very relevant question is whether it's possible that the UI isn't firing the events you'd expect. The first thing I do in this situation is run the AccEvent SDK tool. That tool takes some getting used to, but it can be really helpful. If you have the Windows SDK on your machine, you'll have accevent.exe somewhere.

If you run that tool, go to the Mode menu and verify that the "UIA events" item is checked. (We're only ever interested in UIA events now.) Then click to the Settings item and see the AccEvent settings. In the "Event Type" list clear everything but the "Bounding Rectangle" property changed event. (There's no Clear All and Select All button here, so changing the options can be a real pain.) Then in the "Scope" list, click the top level browser window, (assuming the browser's already running.) Then in the "Report properties" list, have a small number of useful things selected, (such and Name, Control Type and BoundingRectangle.) Then click OK. Once back in the main AccEvent UI, go to the Events menu, and click "Start Listening".

AccEvent will now listen for UIA events, and report what it gets. It doesn't using caching, (ie it'll go back to the source process to get the properties of the element,) but it's pretty reliable. If it doesn't report some events, I always take that to mean the events aren't being raised. I just tried this with IE, and when a page was loaded, I saw some BoundingRectangle property changed events. However, when I scroll down, I see no events raised. On another page however, I did see some events generated when I scroll. (Actually, I think events those events might be useless in this case.) So you might want to give this a go, and see if the events you expect are really being generated. At the moment, I'm rather assuming you won't see those events.

If the source UI isn't generated the events you want, there's no way you can force the events to appear. Whether events are generated is entirely up to the source app or the framework that the UI's built in. Neither UIA nor your client app can affect that. So it's then a case of figuring out whether there's some workaround you can find, perhaps in the form of reacting to other events that get generated in the particular scenario. So what I then did in this situation is have AccEvent look for many more events, in case there's anything that I can use. I just gave this a go, and found that I still not got useful events.

This makes things more complicated, as it looks like there are no useful events being generated in the scroll scenarios. I know what I'm about to say sounds ugly, but one way to account for this would be to use a low level keyboard hook and mouse hook to detect certain key and mouse action. For example, if after certain key presses, (eg arrows, page up/down, home/end), or left mouse button down, assume that the UI shown on the screen might have changed and after a slight delay to allow the UI to settle down, refresh your own UI which sits on top of everything. Far from ideal, but sometimes if apps you care about don't generate the events you want, you need some fairly brute force workarounds.


A mix of perf and inaccessible UI

Q. I was trying to work out the best way of dealing with the situation:
1. The active window has no UIAutomation API

2. It simply takes too long to build the hierarchy of elements.

With the first option it's not too difficult because effectively I just get an empty array of elements pretty quick when running ElementFromHandleBuildCache.

However with the other scenario I'm not too sure how to abandon the cache build after a specific amount of time other than using some pretty crude devices. I also expect there will be some other applications where building the element tree is not possible in a realistic amount of time.


A. When you say that the active window has no UIA support, I assume that you can get a UIA element from the main hwnd for the app? UIA should always provide you with that. But I expect the problem is that various UI shown inside the hwnd doesn't seem to have UIA elements associated with it. If so, what happens if you point the Inspect SDK tool at the UI for which you're not getting UIA elements? If Inspect also can't see it, maybe this is some custom UI and the people who built the UI didn't add an accessibility support into it, (either in the form of UIA or MSAA.)

I don't have a great solution for the UIA calls simply taking too long. (On a side note, the Windows 8 UIA has some support for timing out long calls into UIA. That was really added because if the target app has hung for good, then a Windows 7 UIA call could block forever.)

I expect you've already constrained your cache to be as slimmed-down at it can be. Eg only requesting things that support the InvokePattern and are not offscreen. It's possible that some apps are very slow to respond regardless of what's in the cache requests.

If you point Inspect to some of the slow apps you encounter, I wonder what it typically reports. Eg is it because the UIA tree is really, really deep or wide? Or maybe the tree is corrupt.


Q. Yes I can always get the main window as a UIA element:

rootWindow = uia.ElementFromHandleBuildCache(Win32API.GetForegroundWindow(), myCache);


I am already restricting it to invoke pattern elements that are visible and when using Inspect it clearly shows the applications with no UIA support (i.e. no elements are picked up on the UI) and using it with the particularly slow app shows a fairly detailed element tree. I was more concerned with applications that I don't have and haven't come across that users of my application might come across and I was trying to find a generic way to timeout if any sort of issue was hanging the cache build or making it take an excessively long time.


A. For apps that support MSAA rather than UIA natively, UIA uses its own "MSAA Proxy" to talk to the app. The MSAA proxy takes the MSAA data provided by the app and converts it as best it can to UIA data, and returns that to the UIA client. Converting things like Name properties is straightforward, but converting the MSAA element hierarchy to UIA such that the navigation calls work is pretty complicated. Inspect will report what the provider is when you point it to an element, and if you point it to Internet Explorer, you'll see "MSAA Proxy". Internet Explorer isn't a native UIA provider.

So in general if the source app is either an MSAA or UIA provider, a client can in theory get to the UI. But if the app has not included some piece of the UI in the accessible element tree, there's really nothing UIA can do about it. UIA's a channel through which an app can expose its UI to Assistive Technology (AT) clients. In the past I've effectively been asked how a client app can use UIA to make an inaccessible app accessible, (and the answer is - it can't). In some cases if the client really wants to, it can make serious assumptions about specific inaccessible apps, like where UI is relative to the top left corner of the app window say. Eg I remember some popular app which could occasionally slap some completely inaccessible UI in the middle of the app window when it started. There's no way to programmatically invoke that UI unless you assume that you know where it is. While doing this sort of thing is undesirable, if a customer of yours relies on some inaccessible app, there may be no alternative. (Obviously you know all this already, having built client apps for so long.)


Q. After you mentioned the invoke pattern and visible elements in your last email I actually went back through my various property conditions that I had built the cache with and found that one of the edit property conditions I had set wasn't restricted to on-screen or enabled for that matter. That particular setting was causing much of the delay with the slow app. Anyway at that point I decided to go back to scratch and build the cache properties independently for each of the popular applications/groups of application I would be using. My application now checks the process name of the active window and uses an appropriate property structure for each of the different applications/group of applications. When I say appropriate I mean a set of properties which allow access as fast as I can get it without missing out any elements. I started with a simple AND property condition which was invoke pattern available, enabled and on-screen and took it from there.

As a simple example searching on those three conditions brings up pretty much all of the elements with Shell based UI screens (Windows Explorer, programs and features, control panel etc) but with the likes of Internet Explorer misses out quite a few elements. Combo boxes, edit-based elements like search boxes etc. Anyway I managed to get much more appropriate property searches for the different groups of apps and have got pretty much instantaneous response on most interfaces where Internet Explorer is probably the slowest but it's down to something like 1.5 to 2 seconds maximum for the most complicated HTML pages which is more than acceptable. With most webpages it is pretty instantaneous. I can also build the cache on change of active window and a few other tricks to speed up even more.

I find it best to change the cache request properties with different applications not only for quicker searches in some instances but to stop duplicate rectangles on various items, particularly Internet Explorer. Quite often as an example hyperlinks can have text control links as children leading to small rectangles inside large rectangles duplicating functionality making for a messy display. I could write methods to remove child rectangles inside parent rectangles and other ancestors but it's easier just to set a particular set of properties with different applications I find.

A couple of other questions if I may as it will probably save me some time when I start doing the clicking of rectangles. I can obviously invoke the elements with invoke patterns but I also see there is the method getClickablePoint () and I can also simply work out mouse coordinates from the bounding rectangles. After a quick search I can’t actually see any way of doing right clicks so I presume I can just use the Windows API mouse events for right clicking using whatever method to find the best coordinates.

A. You're right that UIA does not support telling an element to perform its right-click action, (which often would result in a context menu appearing.) I think the only way you can do this is to do exactly what you're considering. Get a point which you feel lies over the element, and use SendInput() to simulate a right click. Note that this could be more complicated if you're presenting your own UI over the element at the time. If you have some area of your own window that's not 100% transparent beneath the right click, your UI will get the click, not the target window. For some reason I seem to feel that not all UI does such a good job at implementing getClickablePoint as they do the BoundingRect property. (But I don't remember why I think that, so I might be wrong.) Some point inside the BoundingRect should be fine to click on, so long at the element is rectangular.


Response. It seems to work fine with either a midpoint on the bounding rectangle or using getClickablePoint (). Interestingly getClickablePoint() adds a little bit of latency to creating the window with the rectangles so I am going with the bounding rectangle midpoint.


Context Menus

Q. Guy have you dealt with UI automation in terms of context menus and other menus for that matter? So as an example if I was running Internet Explorer and opened the Favourites menu how would I know it had been opened and how can I then show UI over the menu items?

In fact most menus are accessible but for some reason with Internet Explorer the menus, e.g. Favourites, seem to be descendants of the desktop rather than the Internet explorer window.


A. I've just pointed the Inspect SDK tool at an IE context menu, and as you say, its parent is the desktop element. This isn't something you can influence, and so it's a case of working with that UI as it is. I assume the issue here is that you're specifically listening for events being raised by the foreground window or its descendant elements, and in the case of the content window, those events aren't being detected.

As a test I ran the Accevent SDK tool. I went to the settings UI, and said that I want to see events raised by the direct children, (not all descendants), of the desktop element. And I only want to detect the MenuOpened event, (and no other events). When I did that, and went to IE and right-clicked, I could see a MenuOpened event. I could also see that the accessible name of the element raising the event was "Context". I then went back to the Accevent tool, and changed the settings to report the process id of the element that raised the event, and I could see that the process was one of the iexplore.exe processes that I could see listed in Task Manager.

So if you want to be able to detect a context menu being opened while IE is the app that the user's in, perhaps when you register for the events you're listening to now, you also need to register for MenuOpened events from elements which are direct children of the desktop element. 

I notice that when invoke the Favourites menu in IE, I didn't get the MenuOpened event. But I did get a FocusChanged event, so maybe that would be one way of detecting that the Favourites menu has been invoked. 


Caching and properties

Q. Guy can you please explain the following. If I set up my cache as shown below you will see that one of the conditions included in the simplified overall condition (overallCondition) for FindAllBuildCache () is a ClassNamePropertyID (nameMessagePanelCondition) and I have not added those type of UIA_PropertyIds to the cache request yet it still gives me a CachedBoundingRectangle for the element that meets the nameMessagePanelCondition. I would have thought I would have had to include ClassNamePropertyIDs to the cache request as in:



However I have not had to do this. Also any comments on this structure generally as seen below. To simplify the question I am only using a tree item condition and the class name condition in the overall condition.

//create CacheRequest
myCache = uia.CreateCacheRequest();                                             

//all elements must be enabled and on screen
enabledCondition = uia.CreatePropertyCondition(UIA_PropertyIds.UIA_IsEnabledPropertyId, true);                                    
onScreenCondition = uia.CreatePropertyCondition(UIA_PropertyIds.UIA_IsOffscreenPropertyId, false);
enabledAndOnScreen = uia.CreateAndCondition(enabledCondition, onScreenCondition);

// Add a Property Condition which is a Control Type Property, included in cache
treeItemCondition = uia.CreatePropertyCondition(UIA_PropertyIds.UIA_ControlTypePropertyId, UIA_ControlTypeIds.UIA_TreeItemControlTypeId);

treeItemCondition = uia.CreateAndCondition(treeItemCondition, enabledAndOnScreen);

// Add a Property Condition which is a Class Name Property, NOT included in cache
//condition for Outlook 2007 message panel           

nameMessagePanelCondition = uia.CreatePropertyCondition(UIA_PropertyIds.UIA_ClassNamePropertyId, "SUPERGRID");      

// Create overall condition
overallCondition = uia.CreateAndCondition(treeItemCondition, nameMessageReadCondition);  

rootWindow = uia.ElementFromHandleBuildCache(Win32API.GetForegroundWindow(), myCache);

//set scope of cache                             
myCache.TreeScope = TreeScope.TreeScope_Element | TreeScope.TreeScope_Children;          

//populate the element array
elementArray = rootWindow.FindAllBuildCache(TreeScope.TreeScope_Descendants, overallCondition, myCache);  

//so this piece of pseudocode is where i happens to be the element that meets the nameMessageReadCondition

elementArray.GetElement (i).CachedBoundingRectangle   … This is still a valid cache condition, how come??


Although the element array is built using FindAllBuildCache any element from the array still gives me access to the "current" conditions as shown below. So does this mean that all the current properties are populated regardless of the element array being built with the CacheRequest?

elementArray.GetElement (i).CurrentName  


A. So as I read the code…

The myCache object is set up for the ControlType and the BoundingRect. This is used in two places later. The first is in the call to ElementFromHandleBuildCache() to get the element for the foreground window. And the second is in the call to FindAllBuildCache(), when certain descendants of the foreground window element are accessed.

Almost all the rest of the code relates to building up the condition to be supplied to the FindAllBuildCache() call. The condition is that the element’s enabled, not offscreen, a treeitem control, and has a “SUPERGRID” classname.

You question is, why were the properties involved with the condition not required to be set in the cache request? This is what I think the situation is…

The condition and the cache are independent of each other. The condition only relates to decisions around which elements are to be returned from the function call. You could pass in your overall condition with no cache at all, and still get the same elements back.

The cache is used to allow faster access to the data you want to access after you’ve made the call which got you the elements. Say some call to FindAllBuildCache() got you 100 elements, and you then need the bounding rects of all those elements. If you didn’t use a cache, you’d get the current bounding rects for all the elements in turn, and that would mean you’d be making 100 cross-process calls. Cross-process calls are slow, and so that hurts the perf of your app. So you create a cache request for the bounding rects. This means while UIA is gathering all the elements the FindAllBuildCache() call, it’s also collecting all the bounding rects. It’s doing that during the one cross-proc call where the elements are being collected, and so the work to get the bounding rects results in no more cross-proc calls. Later when you get the cache bounding rect element, it’s returning the cached value. Technically that cache value might be stale, but often a client app would prefer to live with that issue rather than incurring slow cross-proc calls.

Now, your other question is also an interesting one. That is, how come the CurrentName property works even though a cache request was used when getting the elements? You’d set up a cache request for the ControlType and BoundingRect, and that means you’re not making a cross-proc call when getting those cached properties. But by default, UIA still allows you to get any current property you want on the element. You can get the current name, or even the current bounding rect, (not that you’d do that, given that it’s already cached). But if you do get a current property, you’re incurring the perf hit of a cross proc call, and only you can decide whether that’s acceptable. If you think it’s pretty rare for you to want to get a current name, but once in a while it happens, you could get the current name at that time. But if you feel that it’s likely that you’ll need the current name for a few items, perhaps you’d add the name property to your cache request.

To get the current property, all you need is a full reference to the element. (That is, a reference that still connects to some element in the provider process.) The cache request contains something called AutomationElementMode. By default that’s set to AutomationElementMode_Full, which means you get a full reference. If you’d set the mode to AutomationElementMode_None in the cache request, then all you get is a way to get the cached properties that you’d requested. If you try to get current properties in that case, I’d expect the calls to fail.

By the way, the cache request itself also has a scope and condition. I think this can get pretty confusing, and so wouldn't necessarily change your code. If your code works, is understandable and maintainable, and perf is acceptable, then that’s good. But technically I wonder if the following would work…

1. Set the cache’s TreeFilter to the overallCondition.
2. Set the cache’s TreeScope to descendants.
3. Make only one call to ElementFromHandleBuildCache().
4. Call GetCachedChildren() to get the descendants that meet the condition.
5. Access the cached properties from those children.

I did something like this as an “alternate approach” at the end of LinkProcess.cs up at http://code.msdn.microsoft.com/Windows-7-UI-Automation-0625f55e/sourcecode?fileId=21468&pathId=534397530. But like I say, there may be no real advantage for doing this over what you already have, (and could have behaviours I don’t know about), so I’d stick with whatever works.


Timeouts interacting with unresponsive providers

Q. When I capture the elements on the HTML document within a particular browser I am getting a 4 seconds delay with some densely populated HTML pages but this only happens with Windows 7 and not Windows 8 where it is virtually instant. 


A. Regarding the difference in performance of the UIA API between Windows 7 and Windows 8, one possible reason for this relates to timeout functionality that was added to UIA in Windows 8. In Windows 7, when you made a call to a UIA client API, the call could block until the provider process finished responding to the call. So if the provider was slow, then your client app might be unresponsive for a while. Worse, if the provider hung permanently, then so might your client app. Obviously that's a big problem for your customer.

So in Windows 8, UIA was enhanced such that the client APIs automatically have timeouts associated with them. The intent is that if a provider process does not respond to some call within a certain time, then UIA will stop waiting and return control to the client. In that case, the client API should return an error code of UIA_E_TIMEOUT, but I don't know if that's always guaranteed.

There are two timeout settings. The first relates to establishing contact with a provider. A provider should be able to respond to this very quickly indeed, and so if there's even a small delay in this happening, that suggests that the provider is unresponsive. So the default timeout for that action is only two seconds. The other timeout relates to gathering information from the provider after contact with the provider has been established. Some operations will involve processing many hundreds of UI elements, (and this is likely the case in your situation when you're working with a browser), so it could take a while for the provider to gather all the data that's being requested. So by default the timeout for that action is 20 seconds.

The client app can configure these timeouts. If the defaults aren't optimized for the apps you're working with or the operations you do, the properties you'd set for configuring the timeouts are IUIAutomation2::ConnectionTimeout and IUIAutomation2::TransactionTimeout. (All the new functionality added for Windows 8 is listed up at http://msdn.microsoft.com/en-us/library/windows/desktop/hh437316(v=VS.85).aspx.)

Now having said all that, sometimes the actual timeouts your client app experiences may not be exactly 2 seconds or 20 seconds. Depending on the action being performed, sometimes the timeouts can get compounded internally because UIA might have to make multiple interactions with the provider beneath a single call to a client API. That can result in the client experiencing a longer timeout than the published default values.

But despite there sometimes being some variation in the actual timeout experienced, (and I believe it being possible for UIA_E_TIMEOUT to not be returned after a timeout has occurred,) the addition of the timeout functionality for the client APIs is a very important enhancement to UIA in Windows 8. If a provider process is hung, the client app must be able to continue to serve its customer.

I don't know if this can explain the differences you're seeing. I would have thought it might be relevant if you'd seen your Windows 8 client API call return after 2 seconds, but you've said that the call is virtually instant. Also, if the timeout had occurred, I wouldn't have expected you to get all the data you're after in that case. A timeout means that the provider didn’t supply you with the data you need in a responsive manner. I'm assuming that you did get the data you need. But I'm afraid I can’t really think of another reason for this difference in behavior between Windows 7 and Windows 8.




Following the discussion below around finding items in a context menu, I pointed Inspect to the context menu that appears when I right-click on the desktop. After I'd arrowed through the menu, I pressed Ctrl+Shift+5 to refresh the Inspect UI. Once I'd done that, I could see the menu items in Inspect, as shown here...



Comments (23)

  1. Jenny says:

    Hey Guy,

    I have read all your articles and have looked at all your sample apps, and have learnt a lot from them.

    I am building a small logging application for my college research project for which I am using UIAutomation to capture user activity.

    I have successfully implemented most of the things, but I am stuck with one thing, it is that I cannot get the value of a control after it has been modified.

    I have used FocusChangedEventHandler so I get all info when the control gets focus, but can't get the info when still in focus the control is being modified.

    I tried PropertyChangedEventHandler but to no success.

    Is there a way I can fetch the information of a control just before it looses focus?

    Please help me.

  2. (Part 1 of 2. I'm splitting this reply into two parts, because apparently a single reply can't be more than 3072 characters long.)

    Hi Jenny,

    Thanks for the feedback; I'm pleased to hear that my articles have been of some help.

    From the sounds of things, you can successfully get the UIA property you need from the element when it gets focus. What is that property? For example, the Name property, or the RangeValue.Value property that a slider might provide.

    Ideally you'd use a PropertyChangedEventHandler to react to changes in the property while it has focus. Given that this hasn't worked for you so far, the first step would be to check that the PropertyChanged event you're expecting, really is being raised. Whenever I'm interested in using a UIA event handler, I always take a look at what events are being raised. If the event I want isn't being raised, then I can't react to it. In that case I have to take a look at what events are being raised, and figure out how best to react to those.

    To check whether the PropertyChanged event you want is being raised, you'll want to run the AccEvent tool that's included in the Windows SDK. If, say, I was interested in PropertyChanged events for the RangeValue.Value on a slider, I'd set the AccEvent tool to watch for those events from all UI, and also to report the value after the change. Having done that, as I change the value on the slider, AccEvent reports the new values to me.

    If you find that the events you want are being raised as expected, the next question is why your handler isn't being called. For PropertyChanged events, in general you're only interested in events from the element with focus. When the element of interest got focus, did you call AddPropertyChangedEventHandler() to let UIA know that if should call your handler when the property has changed? (By the way, you'll want to be sure to call a matching RemovePropertyChangedEventHandler() later when you're no longer interested in PropertyChanged events from that element.)

    If you are calling AddPropertyChangedEventHandler() and your handler's not being called, I'm not sure why that is. It might be worth doing a test to simplify the call to AddPropertyChangedEventHandler() as much as possible. For example, if your currently using a scope of something other than TreeScope_Element, or passing in a cache request, or supplying multiple properties in the propertyArray, then instead try to only get the event for the one property on the element and use no cache request. I doubt that would make a difference, but sometimes reducing the complexity of the call can influence how it behaves.

    (to be continued…)

  3. (Part 2 of 2.)

    If after all that, you still find your handler's not getting called as expected, then I'm afraid I don't know what the problem is. As a workaround, perhaps you could take additional action when you get a FocusChanged event? For example, when your FocusChanged handler gets called, make a note of whether focus has moved to your control of interest. Also when focus has changed, check whether the previous element to have focus was that element of interest. Is the element did previously have focus, explicitly get the value of the property you're interested from that previous element. This would mean that you only know the final value of the element, rather than detecting the changes in value while the element has focus, but maybe that's ok for you.

    If you did go with this workaround, you'll need to have access to the element of interest after it's lost focus. (So you can't get the element through GetFocusedElement().) One option would be to explicitly get the element through some unique way of identifying it, (ag an AutomationId). Alternatively, perhaps you could cache a reference to the element when it gains focus, and get the property from that cached reference when you detect that it's lost focus later? I'm assuming the property you're interested is still has the appropriate value after the focus changes. I expect you'd want to AddRef() the reference to the element when you cache it, to avoid any possibility of the object being released by UIA while you still need it. (You'd need to call Release() on it later when you've finished with it to make sure UIA can destroy the object when everyone's finished with it.)



  4. Jenny says:

    Thanks for your response Guy.

    I tried with the accevent tool and realised that the events I was interested in were not at all being raised.

    So, I tried the work around solution you suggested.

    I took reference to the element of interest when it got focus and retreived the property value when the FocusChangedHandler was called again probably taking focus to other element.

    It is working as expected.

    Thanks again.


  5. Hi Jenny, thanks for letting me know the outcome of this. It's good to hear that things are working for you now.

    When working on UIA client apps, I always tend to experiment with the AccEvent and Inspect tools first. That way I know what events and properties are exposed from the UI I'm interested in, and I can code up my app to specifically look for what I know is available. Sometimes the events and properties that I'd like to interact with aren't exposed by the UI and I have to adapt my plans. For example, if an AutomationId property isn't available to identify an element, is there another suitable property? (Remembering that a Name property might be localized for worldwide release.)



  6. pwrhoades says:


    Sorry if this is a repeat question.  I am new to the forum.  I am using all of your unmanaged UIA techniques but am unable to find or control an Infragistics "PopupMenuTool." It is accessed via a right click on a list in our main application.  Inspect sees it just fine and I can navigate up and down all the menu items.  Below are the results from Inspect.  As you can see, there is no Parent.  I have even run a Raw Tree from the desktop, looking for the names in the menu but to no avail.  I have passed in the actual handle of the menu and tried all possible combinations of Find FindAll, but no luck.  Are there any tricks or tips you can suggest?  

    Peter Rhoades – Allscripts Senior Test Automation Engineer.

    How found: Navigating to Parent…

    Name: "PopupMenuTool1"

    ControlType: UIA_MenuItemControlTypeId (0xC35B)

    LocalizedControlType: "menu item"

    ProcessId: 7788

    ProviderDescription: "[pid:7788,hwnd:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]"

    ExpandCollapse.ExpandCollapseState: Expanded (1)

    LegacyIAccessible.ChildId: 0

    LegacyIAccessible.DefaultAction: "Close"

    LegacyIAccessible.Description: "PopupMenuTool1"

    LegacyIAccessible.Help: ""

    LegacyIAccessible.KeyboardShortcut: ""

    LegacyIAccessible.Name: "PopupMenuTool1"

    LegacyIAccessible.Role: menu item (0xC)

    LegacyIAccessible.State: focusable,has popup (0x40100000)

    LegacyIAccessible.Value: ""

    ObjectModel.UnderlyingObjectModel: [Error: calling getter for this property: hr=0xFFFFFFFF80070057 – The parameter is incorrect.]

    IsInvokePatternAvailable: true

    IsLegacyIAccessiblePatternAvailable: true

    FirstChild: "New" menu item

    LastChild: "Interaction Checking" menu item

    Next: [null]

    Previous: [null]

    Other Props: Object has no additional properties

    Children: "New" menu item

    "New…" menu item

    "Cancel" menu item

    "Cancel Entry" menu item

    "View" menu item

    "View in new window" menu item

    "Edit" menu item

    "Edit Selected" menu item

    "Enter Result" menu item

    "Send eReferral Response" menu item

    "Enter Goal" menu item

    "Annotate" menu item

    "View Annotations" menu item

    "Renew" menu item

    "Renew with changes" menu item

    "Void" menu item

    "Authorize" menu item

    "Continue" menu item

    "Add On Orders" menu item

    "Completed Today" menu item

    "Completed On" menu item

    "Order D/C" menu item

    "Record D/C" menu item

    "Stop Deferral" menu item

    "Record as admin" menu item

    "Record as sample" menu item

    "Record Education" menu item

    "Covered Problems" menu item

    "Enter in Error" menu item

    "Verify and Add" menu item

    "Remove" menu item

    "Review" menu item

    "Add to favorites" menu item

    "Cite selected" menu item

    "Cite all" menu item

    "Advanced Result Citation" menu item

    "Generate Next Order" menu item

    "Reconcile" menu item


    Ancestors: [ No Parent ]

  7. Hi Peter,

    When you say that Inspect shows the context menu in the UIA tree, is it shown as a direct child of the desktop element? It is fairly common for a context menu to appear as a direct child of the desktop, rather than as a child of some element in the app associated with the context menu. If Inspect does show the context menu, then I do think it'll be finding it through some regular call to the UIA client API, whether that's using a tree walker, or a call like FindAll(). If you get the UIA root element and then make a call like this when the context menu's visible…

       pElementRoot->FindAll(TreeScope_Children, null /* the condition */, &pElementArray)

    …do you not get an element representing the context menu?



  8. Peter says:

    In Inspect, on the left hand tree, I see the hierarchy from Desktop to IE (we have an old Winforms app that runs in IE).  The IE has the name of our app "Allscripts – Microsoft Internet Explorer."  Beneath that is a no-name node of type "pane."  below that is another no-name node of "menu."  That is as far as the tree goes.  when highlightling "menu" I can see the bounding yelllow highlight around the entire context menu. There is no "linkage" between this lowest "menu" item and the contents of the popup menu.  So, the context menu is not shown as a direct child of the desktop.  I have run find all from the desktop again, with no conditions.  I got 2455 elements, none of which were the named menu items in the context menu or the "PopupMenuTool1" shell of the menu.  Oddly enough, when I use MSAA in Inspect, I can even find a count of children for PopupMenuTool1!

    Is it possible that our app is showing the popup in another thread?

    Thanks for replying.   If it would help, how can I send screen shots?


  9. Thanks for the details Peter. If you got 2455 elements beneath the desktop element, I think you're probably finding descendants rather than direct children. Changing that won't help resolve your issue, but it might make it easier when determining if something is a direct child of the desktop.

    Could you set Inspect up to report details about the element with focus, and then invoke your content menu. As you press the up and down arrow keys, Inspect's yellow highlight should appear around the menu items. Could you then press Ctrl+Shift+5 to get Inspect to refresh its tree view. (You can see a number of handy shortcuts shown in the various Inspect menus. In the Action menu, Refresh shows Ctrl+Shift+5.)

    Maybe once inspect is refreshed, it'll show you exactly where that menu item is in the UIA tree. If so, it might be possible to trace the path between the menu item and the desktop.

    I don't think images can be attached to these replies, but in a minute I'll attach an image of what I just did, at the end of the original post above.



  10. Peter says:

    Guy, short update. I traversed down from the desktop to the "Allscripts – Microsoft Internet Explorer" node–>pane–>menu and then searched for children and descendents of "menu" but to no avial.

  11. Right you are, thanks for the update. Did you have a chance to try the steps I mentioned earlier about arrowing through the context menu and then refreshing Inspect? I originally asked about that because I was wondering if it was possible for the context menu that you're arrowing through to not actually be a descendant of the Allscripts menu at all. I've no idea if that's the case, but in theory some component could create the UIA element associated with the Allscripts menu and add it in the expected place in the UIA tree, but whatever component is creating the UI for the menuitems and some container for those items, could create the associated UIA elements and add them somewhere else in the tree.

    So rather than moving to the Allscripts element in the UIA tree, and trying to find the menuitems from there, it might be interesting to use Inspect to try to reveal where the menuitems are in the UIA tree.

    Of course, I could be completely wrong about that. It's possible that some component should be exposing the menuitems as descendants of the Allscripts menu, but isn't.



  12. Peter says:

    Guy, I did try the steps you mentioned and posted my results (or tried to post them!).  They did not go through.

    Here's what I did.  I had our application with popup menu on one part of my monitor and Inspect to the left. The inspect tree for the Allscripts menu stopped where I indicated earlier.  From Inspect, I could create the yellow highlight around the outermost border of the popup but no further.  I then hovered over one of the menu items in the popup and immediately got a yellow border and the right panel of Inspect showed all the results.  Using Inspect commands, I could navigate to next sibling, prev sibling and up to the so-called "Parent:"  PopupMenuTool1.  Between each navigation, I "refreshed" in Inspect as you suggested.  The tree did not "grow" any further.   I next closed up the tree and tried highlighting over other areas of the main application ("beneath" the popup).  As I did, Inspect's tree quickly expanded and highlighted the correct node.  When I closed up the tree again and hovered over a menu item…nothing happened in Inspect.  I repeated this test just now.  Same results.  From your other posts, I am convinced that UIAAutomation will work on this popup: as you said, if Inspect can "see" and navigate it, I can write unmanaged code that will do the same.   I will try searching other nodes of the left hand Inspect tree, to see if one of them is holding the context menu.    I hope you get this post.


  13. Peter says:

    I dutifully clicked through every node on the left hand window of Inspect with the main app and context popup menu in question showing.  Inspect did not find the popup or the individual menu items anywhere.  I did gain a newfound respect for Inspect: it found and yellow-highlighted every element in our application and on my desktop!

  14. Peter says:

    I think my recent post did not take:  I dutifully went through each node in the left hand tree of Inspect, looking for the popup menu & items.   While I did find everything else on my desktop and in our app, I could not find the popup menu anywhere!  Any other tips?

  15. Peter says:

    I tried two more things but with no success.  I turned on Spy++ and configured it to capture messages on the hwnd of the popup.  Then, using Inspect, I hovered over the popup and put it through its paces with the Inspect navigation keys.  Inspect worked like a charm: the yellow highlight went up and down all 45 menu items with the right hand pane displaying all the information I need.    Spy recorded the usual window messages–none of which shed any light on my problem.   Second: I got the same hwnd spy was using and displaying in the messages, set that as my IUIAutomationElement (from hwnd and also as "Native hwnd property condition).  Each way, starting from either the desktop or from that hwnd, all I was able to drill down to was the "menu"  thereafter, there were no children (FindAll, FindAllBuildCache, etc.)

  16. Peter says:

    Guy, some progress. I started up a second thread in my c# code and used a "from point" discovery method in the managed UI and I was able to "find" one of the menu items.  I then used "GetParent" and found "PopupMenuTool1."  (In Inspect, one can only navigate to the parent from an individual menu item.  Once at the Parent (PopupMenuTool1) in Inspect, the navigation options are only to Parent, First Child, Last Child.  Once at the first or last child, you can navigate up and down the siblings.  The problem I now have is that once at the parent PopupMenuTool1, I cannot get to either FIrst Child or Last Child.



  17. Peter says:

    Solved!  The second thread was not needed.  I used the unmanaged UI code and ElementFromPoint as described above (but with managed code).  Using unmanaged, my treewalker nicely went from menu item, up to parent, down to first child and then down through all 52 sibling menu items.  

    Thank you for your help.


  18. That's great news Peter! So where navigation of the tree alone didn't generate the expected results, you could use a mix of ElementFromPoint() and a treewalker to find all the data you need. While we don't know the root cause of the problem with the UI, given that some UI is constructed in non-typical ways, it is sometimes necessary to get creative as you've done here.

    Thanks for sharing the solution so that everyone can benefit from this.


  19. Peter says:

    Guy, Appealing to you again for help.  This time with troublesome buttons that expose combobox-like lists.  I have tried every technique I know and cannot make the list of items stay expanded (there is no expandcollapse pattern and default action does not expand the list.   I have used simple mouse click on the point, sendmessage, mouse_event, and SendInput.  In each case, the drop down list will quickly expand and then collapse.  I have put a thread sleep between SendInput's mouse down and mouse up and the list remains open.   But obviously I need the lists to remain open as they do with a human left mouse button click so I can pick one of the items.  Any suggestions?

    Thanks in advance,


  20. Peter says:

    Guy: A little progress. I discovered that by calling DoDefaultAction on the "combo," it's list of items does get populated.  IsSelectionItemPatternAvailable=true, so even though the combo drop down list has collapsed, I can access the item I want.  When I fire the ".Select()" method on the pattern, however, nothing happens:  the item I want is not "hit" and the underlying GUI does not change accordingly.

    Any Ideas?

    Thanks, again.


  21. (Part  1 of 3. MSDN limits comments to 3072 characters…)

    Hi Peter,

    Sorry for the delay in responding. I've been out in France exploring the Dordogne region. It's absolutely beautiful!

    It sounds to me like you've done a really thorough job at considering how you might be able to interact programmatically with the combo-like UI. I'm assuming that the UI is not built with a combo box common control from a Windows UI framework, (or at least a fairly modern one). If it were using a common control, it should be interactable fine through UIA. For example, I just pointed Inspect to the combo box in the Run dlg. Its combo box has a control type of UIA_ComboBoxControlTypeId, and that supports the ExpandCollapse pattern. When I programmatically expand the combo box, a bunch of elements with control type UIA_ListItemControlTypeId appear beneath a list beneath the combo box, and each of the list items supports the Invoke pattern.

    But the combo box UI you're dealing with doesn't support the ExpandCollapse list, and it really sounds like the UI is not really programmatically accessible at all. It can be tempting when people first start using UIA, to think that UIA can somehow make inaccessible UI accessible. But really UIA just provides a way for information to propagate from an app's UI over to the UIA client app, and to allow that UIA client to interact with the UI if the app allows it. And as powerful an API that UIA is, it can't make an inaccessible app accessible.

    So in cases like this, I do exactly what you've done – try to find ways to work around the UI's inaccessibility. It can sometimes be worth sending feeding to the creators of the UI too, (whether it's an app or a control,) to make sure they're aware of the issue. If they happen to be updating the UI in the future, they could consider making it programmatically accessible.

    Your use of DoDefaultAction here is an interesting one. I don't tend to use the LegacyIAccessible pattern, (even though most controls support it,) but that's really because most of the controls I work with support UIA natively. When the UIA API was first built, the LegacyIAccessible pattern was added to allow UIA clients to work with MSAA servers through an MSAA IAccessible-like interface if they want to. This would mean that if an existing MSAA client was being ported to become a UIA client, the initial work in the port could be reduced given that some existing client code which deals with the IAccessible interface could be re-used. Then over time, the client could be ported completely over to UIA, and the use of LegacyIAccessible dropped completely.

  22. (Part 2 of 3.)

    While I never use LegacyIAccessible and DoDefaultAction myself, there's no reason why you shouldn't use it if by chance it helps you interact with the inaccessible UI. Once you find that some UI isn't programmatically accessible, it’s a case of finding any steps to can to make things work for you. That's assuming the cost of getting all this to work is worth it, and it's also worth noting that if some unsupported steps do happen to work now, they might not work with future versions of the inaccessible UI.

    What I have tried in the past is to use the SendInput() API, (and again, you've already experimented with that). I think this is a great API, and can be very handy for dealing with inaccessible UI.

    You mentioned that simulating a regular click didn't leave the drop down list open, yet the list remains open if you add a delay between the MOUSEEVENTF_LEFTDOWN and MOUSEEVENTF_LEFTUP events. I think that's a fairly typical experience. There's a finite time between the UI receiving the MOUSEEVENTF_LEFTDOWN event, and all the resulting UI actually appearing. You often need to give a little time for the dust to settle after the UI's received the MOUSEEVENTF_LEFTDOWN, before issuing the MOUSEEVENTF_LEFTUP. This is because the UI code might have assumptions about how it should react to button up events based on the current state of the UI.

    I'm not really clear on what's happening next for you. Am I correct in thinking that through use of SendInput() and the delay, the list is left showing in the UI? If so, I'd next look through the entire list of supported patterns for the list items. Working with the Inspect SDK tool can be difficult with transient UI like combo box drop down lists, but we can still learn a lot. If you have Inspect running and reporting information about the element beneath the mouse cursor, what supported patterns does it show for the list items? From what you've said, it sound like they support the SelectionItem pattern, but there's no guarantee of what will happen when you call Select() on that. (It's basically up to the app.) What we'd really like here is to find that the list items support the Invoke pattern, but I'll bet we'll not be that lucky.

  23. (Part 3 of 3.)

    But hopefully if the list does remain visible, we'll be able to use SendInput() again to invoke some list item of interest. Once the original SendInput() call (with the delay) has opened the list, can you use UIA to find the list element that's a child of the combo box? And if so, find all the children of that list? If that is possible, you could work through the list of children, looking for the list item of interest. There won't be an AutomationId to help identify the list item of interest, as list items don't have very useful AutomationIds like elements outside of lists can have. So you'd either have to assume you know the index of the item of interest or use its Name property. (Using its Name might not be ok if you're working with localized UI.) But if one way or another, you can find the item of interest, you could then get its BoundingRectangle property, and call SendInput() to click in the center of that bounding rect. (Again you probably want a small delay between the button down and button up events.)

    If the above steps don't work in practice due to the nature of this particular UI, could you let me know at what point the steps fall down? While it's not exactly ideal to be having to simulate mouse input to work around inaccessible UI, I do think we should be able to get this working with SendInput(). And who knows, if for some reason mouse input simulation doesn't work, there's always keyboard simulation. A well-behaved combo box will provide a way to invoke its drop down list through the keyboard, eg tab to the combo box and then to Alt+DownArrow. So by using SendInput() with KEYBDINPUT, (and a small delay between the key down and key up events), the drop down list should be invokable.

    Please do let me know how it goes – this is an interesting problem. In the meantime, I'll get back to looking at photos of chateaux. 🙂


Skip to main content