This post describes how an element’s RuntimeId property is only guaranteed to not change for the lifetime of the element. So it is not appropriate for a UIA client app to try to find an element over multiple runs of a target app, via the element’s RuntimeId.
I recently had an interesting chat with a dev who wanted to use the Windows UI Automation (UIA) API to access specific UI elements in apps. He was planning on using FindFirst() or FindFirstBuildCache() to find an element relative to some other element. For example, maybe he’d first access a UIA element representing some app window by using ElementfromHandle(), and then he’d find some child or descendant of that element using FindFirst().
In order to find that element, he builds up a UIA condition which uniquely identifies the element. The goal is to generate a condition that he’ll be able to use in the future, to find the element whenever he needs it. So the question is, is it ok to include the RuntimeId property of the target element in that condition?
(For those of you in a hurry – the answer’s “No”.)
Using the AutomationId property to identify an element
When trying to find a UIA element, you want to build up a condition using properties that as far as possible are unique, and will never change on the element. This means a UIA client app, (such as an assistive technology tool or automation test,) can happily find the element of interest today, and in years to come.
A great property to check out first is the element’s AutomationId. At any given moment, an app should only have one element with a specific AutomationId, and it should not change regardless of whether the app is restarted. So if the element has an AutomationId, that’s a good property to include in your condition for finding the element.
Where are the AutomationIds in a XAML app coming from?
When I build a XAML app and I want to explicitly give my control an AutomationId, I can do this through the AutomationProperties class, as shown in this example markup:
<Button x:Uid=”HydrangeaButton” AutomationProperties.AutomationId=”HydrangeaButtonId” />
The following image shows the Inspect SDK tool reporting that the AutomationId is set on the element from the AutomationProperties.AutomationId that I set in the markup.
Figure 1: The Inspect SDK tool reporting the AutomationId of an element, where that was set using the AutomationProperties.AutomationId markup.
As it happens, if I don’t set an AutomationProperties.AutomationId, and I do set an x:Name, then the x:Name is repurposed by the XAML Framework as the AutomationId. For example:
<Button x:Name=”StillHydrangeaButtonId” x:Uid=”HydrangeaButton” />
Figure 2: The Inspect SDK tool reporting the AutomationId of an element, where that was set using the x:Name markup.
This repurposing of the x:Name is good news for UIA client apps, as it means a lot of controls expose a useful AutomationId, without the app dev doing any explicit work themselves to make that happen.
Is the AutomationId guaranteed to be good to use?
Seeing the Inspect SDK tool report an AutomationId on an element that you want to access, is a joy to behold. But some things to watch out for:
1. When a new version of an app is released, there’s no guarantee that the AutomationIds on its elements will not change. Sometimes they won’t, but sometimes they will, and that’s up to the people who built the app. So your UIA client app might have to account for the app version.
2. Sometimes when accessing an element through an AutomationId, you need to be aware of context. For example, maybe the app has a couple of pages that both contain an “OK” button, and both those buttons have an AutomationId of (say) “OkButton”. So if you find a button with an AutomationId of “OkButton” without being aware of the current page, maybe it’s not the button you’re looking for. Personally, if I were the app dev, I’d try to give the buttons different AutomationIds to avoid any potential for ambiguity.
3. In theory a target app could just break the rules, and ship UI which has elements showing at the same time with duplicate AutomationIds. I have to say that I don’t think I’ve ever seen that, and so it’s not a big concern in practice.
All in all, if you have an AutomationId to use, you’re off to a great start. The image below shows the Inspect SDK tool reporting the AutomationId exposed by the Refresh button in Edge. Visually the button can appear as a Refresh and a Stop Refresh button, and while its UIA Name property changes to match what’s shown visually, its AutomationId doesn’t change.
Figure 3: The Inspect SDK tool reporting that the AutomationId of the Refresh button shown in Edge, is “StopRefreshButton”.
Now, that’s all well and good, but what happens when the element doesn’t have an AutomationId?
Other ways of finding an element
The dev I was in discussions with pointed out that some of the elements of interest to him didn’t expose an AutomationId at all, (or it exposed one whose value is an empty string,) so how else could he reliably identify an element?
Basically, there are no hard and fast rules on what to do next. It’s a case of taking a look with the Inspect SDK tool to learn about all the properties that are exposed by the element, and deciding what will be sufficiently robust for you in your situation.
Perhaps some combination of the following might be of help…
At first glance the Name might seem perfect for what you need. Perhaps you’ve verified that the element’s Name doesn’t change depending on the state of the element, (and so doesn’t behave like the Refresh button in Edge does, where the Name property changes during a refresh operation).
However, the Name property is as localizable as any visual text. This means if you search for a Name of (say) “Transparency”, that might work in some locales, but it’s not going to work in the Netherlands where the element’s Name is “Doorzichtigheid”.
Whether that’s a concern for you depends on your situation. Maybe at the moment you’re only planning on having your UIA client app run on systems with a known UI language. In that case, the Name might work just fine for you.
UIA tree hierarchy
Once I’ve found that the AutomationId and Name aren’t helpful to me, I then use Inspect to take a close look at the UIA tree hierarchy and all the element’s properties, and try to find some way of identifying the element in a sufficiently robust manner.
For example, say I want to access the “Directions” button in the Maps app. The button doesn’t expose an AutomationId. In fact the only UIA properties that seem different between the Directions button and the other buttons on the AppBar are its Name, (which will be localized,) and its BoundingRectangle, (which is dependent on where the button appears on the screen).
But say for this version of the Maps app, the button is always the second child of the AppBar element. I don’t know if that’s true or not, but in some apps (at least for given versions) branches of the UIA tree hierarchy really don’t change. So let’s assume I’ve thoroughly investigated this and believe that for this version of the Maps app, the Directions button is always the second child of the AppBar.
Figure 4: The Inspect SDK tool reporting that the Directions button has no AutomationId, and is the second child of the AppBar element.
The AppBar has an AutomationId of “AppBar”, so I could use that to build up a UIA condition to find the AppBar element beneath the Maps app window. I could then use a TreeWalker to find the second child of the AppBar.
Actually, what I’d probably do instead is call FindFirstBuildCache() to find the AppBar, and supply a CacheRequest to get all the properties and patterns that I care about for the children of the AppBar. (For example, to get a reference to an Invoke pattern for the Directions button.) This allows me to get all the properties and patterns that I care about on the Directions button, with the same call that I use to find the AppBar.
You might feel that the approach of relying on the UIA tree hierarchy when accessing an element is pretty hacky. I prefer to think of it as doing what’s necessary given the information you have at your disposal. Sometimes you just don’t have all the information that you’d like, and all you can do is decide how to make the best of things. (Having said that, if you happen to have a way to contact the app manufacturers, you could always ask them to consider adding some AutomationId properties to their UI.)
Depending on the UI, you might be able to incorporate other properties into the condition used for the Find operation. For example, say I’m interested in a particular UIA element in a set of elements that are descendants of some other element which I have easy access to. Perhaps the target element’s Control Type could help differentiate it from the other elements.
The image below shows Inspect reporting that the link contained in the Tablet PC Settings dialog is the only element on the dialog that has a control type of Hyperlink. (At least it’s the only element at this moment.) So if I’ve called ElementFromHandle() to get the UIA element representing the dialog, I can then call FindFirst() with a condition such that I look for Hyperlinks amongst the dialog’s descendants, and I’ll know the result will be the one and only link in the dialog.
Note that in that particular example I do need to look for descendants rather than direct children of the dialog, because the link’s not a direct child. I’m always conscious of how big a descendant tree is before I ask UIA to search through it all, but this dialog’s descendant tree is pretty small. I’d never search through all an element’s descendants if I know a search through its direct children will be sufficient.
Figure 5: The Inspect SDK tool reporting that only one descendant of the dialog has a control type of Hyperlink.
So getting back to the RuntimeId…
As you examine an element’s properties to learn how it might be differentiated from other elements in the UI, you’ll notice it’s RuntimeId is different. So it can be tempting to consider incorporating that into a Find operation. But you’ll really not want to do that.
The RuntimeId is guaranteed to not change on a UIA element only for the lifetime of the element.
When a UI element is recreated by an app, there’s no guarantee that the UIA element representing it will have the same RuntimeId as it had in some previous incarnation of the element. So incorporating the RuntimeId into a condition used when trying to Find an element over multiple runs of an app will almost certainly fail.
In fact the value of the RuntimeId property is opaque to UIA client apps. The values are meaningless to the client, and the actual values used to make up the RuntimeId are determined by the UIA provider. That provider might have its own native UIA implementation, or it could be part of the UI framework used to build an app. And the logic used today by the provider to build up the RuntimeId might change over time.
So what’s the point of exposing a RuntimeId? Well, the RuntimeId can be handy to a UIA client app when the client app has accessed two UIA elements and then wants to know if they’re actually the same element. If the RuntimeIds of the elements are the same, then they’re considered to be the same UIA element. So you can pass in the elements’ RuntimeIds to IUIAutomation::CompareRuntimeIds() and learn whether the elements are the same element. This is largely what IUIAutomation::CompareElements() does too, if the elements passed in have RuntimeIds().
The discussion following the post at A real-world example of quickly building a simple assistive technology app with Windows UI Automation raises doubts about the validity of what I said above. That discussion relates to a case where the same element was accessed through different means, and yet CompareElements() said the elements were not in fact the same element. Unfortunately I can’t explain that unexpected behavior. Perhaps there was a bug in the version of UIA being used, and if so, I don’t know if it’s fixed in the Windows UIA API today. (I’ve not managed to reproduce the unexpected behavior with the latest version of UIA, but I’m not interacting with the same target UI.)
I’ve also seen once during app development a case where an app which had its own native UIA provider implementation unintentionally exposed multiple sibling elements with the same RuntimeId. That bug was fixed before the app shipped, and I’ve not seen that sort of problem in a shipping app.
So while it is technically possible for bugs somewhere to result in two different elements having the same RuntimeId, or an element accessed in different ways not being recognized as being a single element, I think that would be really rare in practice. The Narrator screen reader certainly expects both CompareElements() and CompareRuntimeIds() to work exactly as advertised.
When your UIA client app needs to find a particular element in an app’s UI, try to use the AutomationId property. If that property is either missing or empty, use a combination of other properties and the element’s relationship with the rest of the UI, to arrive at a mechanism that you feel is sufficiently robust.
Don’t incorporate the RuntimeId property in your Find conditions, because that property may change when the UI element’s next created.