UIA Custom Patterns: Part 1

Article
12/07/2010

With this post, I’m starting a new series on how to use custom patterns in UI Automation. Custom patterns (and custom properties and events) are a new feature of UI Automation added in Windows 7, and I thought it would be interesting to build on my previous samples to show how to use this new feature. The samples for this post are here.

The MSDN page for custom patterns is quite good and is a useful reference to go along with this introduction.

These posts are going to cover a few different topics, weaving in and out of each other:

What are custom patterns in UI Automation?
Why would you use them?
What work does UIA do and what do you have to do to use custom patterns?
What extra work do you need to do to use them in managed code?

To answer the second question, I’m going to pose an example. In the earlier series on building your own UIA provider, I built a state indicator, similar to a traffic light, that showed red, yellow and green states. My control has UI Automation properties like Name, Automation ID, and Value. The Value is “red”, “yellow”, or “green.” Suppose I wanted to expose some other information from my control that doesn’t fit nicely into UI Automation’s categories? For example, suppose I wanted a string called ReadyState that would be “ready” for green and not ready for the other states? Or suppose my control had some interesting internal state, like the number of times it had been clicked during the current run? There’s no UI Automation property to answer that question, so I’m stuck. Those two examples are focused on test automation, but there are Accessibility examples, too: If I had an object that represented an equation, how would I add a property to query the underlying math syntax (say, in MathML) for the equation? You wouldn’t. Try as we might, my team cannot invent every possible UI Automation property. We created the custom patterns and properties mechanism to allow developers to invent their own.

Now I can get back to the first question: what are custom patterns and properties in UI Automation? They are a set of patterns and properties that a client and a provider register with UI Automation so that they can communicate with each other. In a sense, they are like an IDL file or a network protocol: both sides need to agree on the interface or protocol before they can communicate with each other.

In the case of a custom property – and that’s going to be my focus for the rest of this post – the client and the provider need to agree on:

A unique ID (GUID) to identify the property
The property’s type: integer, string, bool, double, and UIA element are the main ones

Let’s start looking at our sample, which implements a ReadyState property, as I sketched above. We’ll start with the client. First, the client has to register the property with the UIA Registrar:

                 // Get our pointer to the registrar
                Interop.UIAutomationCore.IUIAutomationRegistrar registrar =
                    new Interop.UIAutomationCore.CUIAutomationRegistrarClass();

                // Set up the property struct
                UIAutomationPropertyInfo propertyInfo = ReadyStateProperty.Data;

                // Register it
                int propertyId;
                registrar.RegisterProperty(
                    ref propertyInfo,
                    out propertyId);

The UIAutomationPropertyInfo struct is populated in other code, but it just has the property GUID, type, and programmatic name. Once I register the property, I get back an integer property ID. Now I can add code to GetPropertyValue to respond to that property ID:

         public override object GetPropertyValue(int propertyId)
        {
            if (propertyId == ReadyStateSchema.GetInstance().ReadyStateProperty.PropertyId)
            {
                return (this.control.Value == TriColorValue.Green) ? "Ready" : "Not Ready";
            }

            return base.GetPropertyValue(propertyId);
        }

And that’s it – my provider work is done. That was easy. Now I can show a screenshot of the Inspect tool, like I usually do – right? Well, no, I can’t. Remember, both the provider and the client need to know about a custom property, and Inspect doesn’t know abut it. In order to consume my new property, I’ll need another client.

This gave me a reason to do something for these samples that I wanted to do for a long time: add unit tests. I’m a huge fan of unit tests. I’m accustomed to using the testing framework that we use in the Windows organization, but I can’t use that for samples, so I used the Visual Studio 2008 testing framework instead, in the Microsoft.VisualStudio.TestTools.UnitTesting namespace. My unit tests needed some functionality that was important, but not vital to this post: it needs to run the UiaControls.exe sample, find the window, and find the custom provider. You can look at the sample to see how this works; it’s not hard. Once I have that, my unit test for the custom property looks like this:

         UIAControls.ReadyStateSchema.GetInstance().Register();

...

         [TestMethod]
        public void TestReadyStateProperty()
        {
            // Query our custom property
            object readyStateValue = this.customElement.GetCurrentPropertyValue(
                UIAControls.ReadyStateSchema.GetInstance().ReadyStateProperty.PropertyId);
            Assert.IsInstanceOfType(readyStateValue, typeof(string));

            // By default, UIAControls.exe launches in not-ready state
            Assert.AreEqual("Not Ready", (string)readyStateValue);
        }

I can use my custom property just like any other UIA property: I pass its property ID to GetCurrentPropertyValue() and get the property back. I validate that it is a string, and more specifically the string I was expecting.

The ReadyStateSchema class is a singleton that encapsulates all of the data about the ReadyState property. UIA doesn’t require this – you can keep the data wherever you want. But since you only need to register a property once, it feels like a Singleton pattern, and it seems clean to keep the custom property’s GUID, type, and assigned property ID in one place. The schema is a bit heavyweight for one property, but when we have a whole pattern, it will be a good fit.

If you are working in native code, that’s pretty much it. The native code syntax is pretty similar to managed code, and you’ll need just about the number of lines of code I’ve shown here. I don’t have a published sample for this in native code, but the translation is easy, and there are good samples up already at the MSDN page on custom pattern registration.

For managed code, unfortunately, it takes a lot of work to make this all happen. I wrote this sample to make this easier, by creating a working body of code to borrow from. This part of UIA was just not written with managed-code interoperability in mind. I had fun taking on the challenge, but I imagine others might not – hence the sample. To summarize what I needed to do:

Customize the interop IL for UIAutomationCore.tlb to fix some parameters that were imported as “ref int” that should have been IntPtr. (CustomizeUiaInterop project)
Add custom project steps to disassemble the interop DLL, run the customizer, and re-assemble it. (UiaCoreInterop project)
Create a number of helper classes to assist with creating the native UIAutomation structures that are used to register custom properties and patterns. (CustomPatternHelpers.cs)

There were a few more things for custom patterns, but I’ll get to those in a later post. This was sufficient for custom properties, which actually cover a lot of scenarios, as long as you don’t need custom verbs/actions/manipulations for your UI. If you do, you’ll need custom patterns, which I’ll cover in a later post.

In the meantime, if you run the sample, you can see the TestReadyStateProperty test passing, demonstrating how to declare and use a custom property in UI Automation.

UIA Custom Patterns: Part 1

Additional resources