UI Automation in Silverlight - Simulating User Interactions
I was recently tasked with automating Silverlight Rich Internet Applications (RIAs) in our immediate group. Some tools out there provide limited assistance in this regard; for example, you can write unit tests against your Silverlight controls for in-proc testing. You can read more about that approach here: https://www.jeff.wilcox.name/2008/03/31/silverlight2-unit-testing/
Unfortunately, my requirement is to enable scenario automation. We must simulate a user that goes through several user flows both within and outside of our RIA; move the mouse somewhere, click on a thing, go to a 3rd party authentication provider, start typing some keys, and so on. The Silverlight unit testing framework doesn't quite address this requirement.
UI automation in Silverlight has been a hotly followed topic around here. With the release of Silverlight Beta 2, we started seeing some accessibility stubs come into play. More correctly, we started seeing the WPF way of doing UI automation start to trickle in. If you want to follow along, you're going to want to grab UISpy (https://blogs.msdn.com/windowssdk/archive/2008/02/18/where-is-uispy-exe.aspx).
The Microsoft UI Automation (UIA) assemblies were released with the .NET Framework 3.0. Traditionally, we've had various COM wrappers to work with Microsoft Active Accessibility (MSAA), which isn't going to get you very far with your Silverlight application. If you've done any sort of UI automation in WPF, then you're going to feel right at home. So, without dwelling on that further, let's start working with UIA!
First, let's start a new Test project in Visual Studio 2008. The UIA namespaces that you'll need to start automating applications in Silverlight are System.Windows.Automation and System.Windows.Automation.Providers. You'll need to add references to these assemblies that ship with .NET 3.0+ to get the relevant parts:
- UIAutomationProvider.dll
- UIAutomationClient.dll
- UIAutomationClientsideProviders.dll
- UIAutomationTypes.dll
Let's say that we need to automate a Silverlight control that we own. We'll need to override the OnCreateAutomationPeer method (from the Control class) to return our own Peer type that handles the accessibility functions of the control. This is important, because the accessibility functions will be key in letting us automate our application.
Assume a hypothetical Search control that consists of a text box and a search button:
public partial class SearchBar : Control
{
...
public SearchBar()
{
this.GotFocus += (sender, args)
=>
{
this.SearchText.Focus();
};
InitializeComponent();
}
protected override AutomationPeer OnCreateAutomationPeer()
{
return new SearchBarAutomationPeer(this);
}
}
We've overridden the OnCreateAutomationPeer, which will get called by anything that's inspecting your control tree for accessibility functions (and consequently, your automation functions). The Peer object will be responsible for returning your combination of controls in a manner that is coherent to anything that needs accessibility.
In the process of doing so, we've also wired up the GotFocus handler to set up our controls' default .Focus() behavior.
Let's take a look at what we need to implement the SearchBarAutomationPeer class:
public class SearchBarAutomationPeer : FrameworkElementAutomationPeer, IValueProvider
{
public SearchBarAutomationPeer(SearchBar searchBar) : base(searchBar)
{
}
Our Peer class needs to derive from FrameworkElementautomationPeer to provide all the methods that we're going to need to work with. IValueProvider maps out to interacting with the TextBox component of our custom control. You can learn more about the Provider interface mapping to individual components here: https://msdn.microsoft.com/en-us/library/system.windows.automation.provider.aspx
We need to give our control a class name and an accessibility identifier to find it in the control tree. To do this, we must override GetAutomationIdCore() and GetClassNameCore() from FrameworkElementAutomationPeer.
protected override string GetAutomationIdCore()
{
return "SearchBar"; // You're going to want to make this unique. ;)
}
protected override string GetClassNameCore()
{
return "SearchBar";
}
protected override bool IsKeyboardFocusableCore()
{
return true;
}
IsKeyboardFocusableCore is an important override to add in, as well; without it, our calls to SetFocus() on the control will fail. We should also think about implementing our Provider interface. The SearchBar that we passed into our constructor maps out to the base.Owner property. Casting base.Owner to SearchBar is going to get tedius, so we'll add a property to make working with that easier as well.
public SearchBar SearchBar
{
get
{
return (SearchBar)base.Owner;
}
}
#region IValueProvider Members
public bool IsReadOnly
{
get
{
return this.SearchBar.SearchText.IsReadOnly;
}
}
public void SetValue(string value)
{
this.SearchBar.SearchText.Text = value;
}
public string Value
{
get
{
return this.SearchBar.SearchText.Text;
}
}
#endregion
If we take a look at our control in UISpy, it should now look something like this:
Identification
ClassName: "SearchBar"
ControlType: "ControlType.Custom"
Culture: "(null)"
AutomationId: "SearchBar"
LocalizedControlType: "custom"
Name: "SearchBar"
ProcessId: "2276 (iexplore)"
RuntimeId: "42 197110 6"
IsPassword: "False"
IsControlElement: "True"
IsContentElement: "True"
Visibility
BoundingRectangle: "(356, 286, 949, 36)"
ClickablePoint: "830,304"
IsOffscreen: "False"
ControlPatterns
Value
Value: ""
IsReadOnly: "False"
The "Value" property under ControlPatterns automagically comes from the IValueProvider interface, mapping out to the value of our underlying TextBox. Slick, huh?
So we've done some plumbing to enable our Silverlight control. Let's take a look at our test method looks like:
[TestMethod]
public void TestMethod1()
{
Process process = System.Diagnostics.Process.GetProcessesByName("iexplore").First();
AutomationElement browserInstance = System.Windows.Automation.AutomationElement.FromHandle(process.MainWindowHandle);
TreeWalker tw = new TreeWalker(new PropertyCondition(AutomationElement.ClassNameProperty, "SearchBar"));
AutomationElement searchBar = tw.GetFirstChild(browserInstance);
myElement.SetFocus();
Thread.Sleep(1000);
searchBar.SetFocus();
Thread.Sleep(1000);
SendKeys.SendWait("Hello, world!");
}
You might be asking yourself a couple of questions at this point: "Why did I implement IValueProvider?" for example. Well, the snippet above simulates user input. If that isn't your thing, in comes the ValuePattern. As an aside, I found interacting with the ValuePattern/TryGetCurrentPattern/etc and found the whole experience to be a bit clunky. You can see what I mean below:
[TestMethod]
public void TestMethod1()
{
Process process = System.Diagnostics.Process.GetProcessesByName("iexplore").First();
AutomationElement myElement = System.Windows.Automation.AutomationElement.FromHandle(process.MainWindowHandle);
TreeWalker tw = new TreeWalker(new PropertyCondition(AutomationElement.ClassNameProperty, "SearchBar"));
AutomationElement searchBar = tw.GetFirstChild(myElement);
object valuePattern;
searchBar.TryGetCurrentPattern(ValuePattern.Pattern, out valuePattern);
((ValuePattern)valuePattern).SetValue("Hello, world!");
}
This is by no means a comprehensive guideline, but it should be enough to get those of you out there interested in UI automation going.
One caveat: applications with Windowless enabled show up as one huge control if you're looking in UISpy. Hopefully, support for accessibility (and subsequently, automation) will be in RTW builds of Silverlight. If you want to perform UI automation on your Silverlight application today, you'll have to do it without Windowless.