This post describes steps you can take to run one of my Windows 7 UI Automation (UIA) client sample apps with Edge on Windows 10.
I was recently asked whether my Windows UI Automation (UIA) client sample app at Windows 7 UI Automation Client API C# sample (hyperlink processor) Version 1.0 is compatible with Windows 10. I built that sample app on a Windows 7 device, and it demonstrates use of some of the Windows UIA client API. The app finds links on a web page in Internet Explorer, and programmatically invokes those links. It also uses the Windows Magnification API to magnify the area on the screen where the link is shown.
I’d not tried running the sample app on Windows 10, and so set about seeing whether it runs ok or not.
Building the sample in Visual Studio 2015
I thought I’d try building the sample app on my Surface Pro 2 running Windows 10 build 14379.rs1_release.160627. This device is on the Windows Insider Program, and is running a pre-release version of the Windows 10 Anniversary Update. I expect my experience working with my UIA client sample app would be the same on any version of Windows 10.
The version of Visual Studio 2015 that’s installed on my Surface Pro 2 is the free Community edition that I’d downloaded from MSDN.com.
As part of getting the sample app to build on this device, I did the three things described below.
1. Update the UIA wrapper
Most of my sample apps are written in C#. And that’s because I like C#. But given that I’m calling the native Windows UIA API, I need a wrapper that makes it easy for me to call a native COM API from C#. I generate this wrapper using the tlbimp.exe tool that comes with the Windows SDK.
My UIA sample app had a custom Build Event action, as shown in the project properties. It had a “Pre-build event command line” of:
"%PROGRAMFILES%\Microsoft SDKs\Windows\v7.0A\bin\tlbimp.exe" %windir%\system32\UIAutomationCore.dll /out:..\interop.UIAutomationCore.dll"
This meant that every time the project was built, the tlbimp.exe tool would be called to generate the wrapper.
I decided that I would remove this build step from my project, because really, there’s no need to generate the wrapper every time I build the project. Instead I would run the tlbimp.exe tool once to generate the wrapper, reference the output wrapper in my project, and then forget about it.
So the next thing I did was look for where tlbimp.exe is on my device. It turned out it was in a few places, so I picked the one that looked most relevant to an x64 Windows 10 device. I then ran the following in a cmd window from the project’s folder.
"C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6 Tools\x64\tlbimp.exe" %windir%\system32\UIAutomationCore.dll /out:.\interop.UIAutomationCore.dll"
And hey presto, I have my new UIA wrapper for calling the native Windows UIA API from C#. (The tlbimp.exe tool reported some warnings when I ran it, but I’ve found that I can ignore those.) And now that I have the shiny new wrapper I need, I can go to the project’s References, remove the stale reference to the old wrapper and include a reference to the new wrapper.
2. Update the target .NET Framework
When I then tried to build the sample app, I got a big bunch of errors and warnings, with one relating to the target version of the .NET Framework. So I took a look at the Application tab of the project’s properties. That showed a “Target framework” of “.NET Framework 3.5 Client Profile”. So I decided to update that to the latest available to me, “.NET Framework 4.6”. This fixed most of the errors on my next attempt to build.
Figure 1: Project properties showing the updated target .NET Framework.
3. Reference Microsoft.CSharp
I did however still get the following error reported:
Missing compiler required member 'Microsoft.CSharp.RuntimeBinder.Binder.Convert'
Looking around the web, I found a suggestion that adding a reference to Microsoft.CSharp can fix that error. And sure enough, with that reference, I could build the project with no errors. My final list of references is shown below.
Figure 2: Project references shown in the VS 2015 Solution Explorer.
While I can build the app, I still get a few warnings reported about the UIA wrapper generated by the tlbimp.exe tool. I can ignore those, as they don’t impact my use of the wrapper.
Ok, so the sample app builds now. But does it run?
The sample app was specifically written to work with Internet Explorer. It finds a window with a class name of “IEFrame”, and then finds hyperlinks beneath that window. So I started up IE 11, and ran the sample app.
To my delight, the sample app ran fine. It presented a list of hyperlinks shown on the web page in IE, and could programmatically invoke the hyperlinks. It could also magnify the area on the screen where the hyperlink is shown. (The sample app intentionally inverts the colors shown in the magnified area.)
Now I must confess I didn’t run through all the action taken by the sample app to verify that everything works. But at least the basic stuff seemed to work pretty well.
The image below shows the sample app listing links shown on an MSDN.com page, and one of those links being highlighted.
Figure 3: The sample app listing hyperlinks on a web page hosted in IE.
Interestingly, the sample app only lists the hyperlinks that are visible on the page. If a hyperlink is scrolled out of view, then it won’t get included in the list shown in the sample app.
While I was rather pleased that the sample app still works with IE, I was really curious about what it would make of the Edge browser. After all, I’d built the sample app years before Edge existed.
One change I’d certainly have to make is to point the sample app at Edge, instead of hard-coding it to look for IE. There are a few handy ways to have UIA find an element of interest, including finding the element beneath the mouse cursor or the element with keyboard focus, but for this test I stuck to finding the element from a window. So I ran the Spy++ tool from the VS Tools menu, and took a look at the properties of an Edge window. I found that the top-level Edge window had a class name of "ApplicationFrameWindow", so I changed the sample app to look for that, instead of an “IEFrame” window.
Note: Using a class name of “ApplicationFrameWindow” doesn’t sound at all robust to me. I really don’t know what other windows might have that same class name. But for this test, the sample app found the one Edge window I had running, so it was fine to use that window class for this experiment.
Having done that, I found the sample app worked fine with Edge. As with IE, the sample app could list the hyperlinks on a page and programmatically invoke them.
Figure 4: The sample app listing hyperlinks on a web page hosted in Edge.
How come the sample app shows more links from Edge than from IE?
One thing that surprised me was that the list of hyperlinks shown in the sample app was longer when working with Edge than it was when working with IE. So I had to look into why that was.
It turned out when working with Edge, hyperlinks that are scrolled out of view have different UIA properties relative to the equivalent links in IE.
I ran the amazingly-helpful Inspect SDK tool to examine the properties of a hyperlink that was scrolled out of view. Inspect showed me that with IE, the hyperlink does not expose a BoundingRectangle property at all. Yet with Edge, a BoundingRectangle is exposed.
Figure 5: Inspect reporting that a hyperlink element that’s scrolled out of view in Internet Explorer is exposed through UIA with no BoundingRectangle property.
Figure 6: Inspect reporting that a hyperlink element that’s scrolled out of view in Edge is exposed through UIA with a BoundingRectangle property for a rectangle that’s not zero-size.
And while I’ve not taken a close look at what the sample app does with the BoundingRectangle, I suspect that it ignores any hyperlink that either doesn’t expose a BoundingRectangle, or exposes a BoundingRectangle for a rectangle that’s zero-size. Hence it ignores the IE out-of-view hyperlinks and doesn’t ignore the Edge out-of-view hyperlinks. Whether you would choose to ignore those hyperlinks yourself would depend on the needs of your customers.
Also, it’s worth pointing out that the UIA IsOffscreen property of the out-of-view links has a value of True for both IE and Edge. The above images show Inspect reporting the IsOffscreen property for elements in both IE and Edge.
Should anything else be changed in the sample app?
I built the UIA client sample app for Windows 7, when UIA had more threading constraints than it does today. I really went out of my way to try to prevent any threading issues from leading to delays (or even hangs) when using the sample app. This involved adding my own message queue. It could well be that some of that work is no longer required in Windows 10, but I left it all in, because as far as I know it still works fine.
While the threading constraints have been relaxed over time, there are still some requirements around threading which your UIA client code must adhere to, and it’s really important to keep these in mind. These are listed up at Understanding Threading Issues, and include the following:
1. If you interact with your own UI, you must make the UIA calls on a background MTA thread.
2. Don’t change UIA event handler registration from inside a UIA event handler. For example, don’t add a PropertyChanged event handler to an element while inside a FocusChanged event handler. (That sort of action can be tempting, because you might want to know when properties change on the element with keyboard focus.)
3. Remove an event handler on the same thread that you added that event handler.
Not forgetting CUIAutomation8!
While the sample app seemed to work fine now, there is one very important change that I would still make. The sample app uses a CUIAutomation object to perform all its UIA-related action, as that was what was available in Windows 7. When Windows 8 came along, a new CUIAutomation8 class was introduced. For me, the most important benefit of using a CUIAutomation8 class is that it’s more resilient against non-responsive providers than the CUIAutomation class is. That is, if you make a UIA client call and try to interact with an app that’s hung, if you use CUIAutomation, then your client app might hang too. But if you use CUIAutomation8 instead, there’s a good chance that your call will timeout, and your client app can continue along its merry way.
So my final change to the sample app is to update the UIA initialization in two places to be the following:
// The first step in calling UIA, is getting a CUIAutomation8 object.
_automation = new CUIAutomation8();
I’m afraid I’ll probably not have an opportunity to upload an updated version of my sample app soon, but if you make the changes described in this post, it should work fine with Windows 10. The changes are:
1. Update the project’s properties to target the latest version of the .NET framework.
2. Remove the custom tlbimp.exe build step, and manually run tlbimp.exe to generate the UIA wrapper.
3. Add the output wrapper dll, and Microsoft.CSharp, to your list of references.
4. Initialize UIA with CUIAutomation8, (in both places where this is done in the sample app).
I have to say that I’m really pleased to see the sample app’s UIA client calls work with Internet Explorer and now also with Edge. I hope this helps you build assistive technology tools for people you know, using the Windows UIA client API.