Visual Studio UI Test Extensibility–Scenarios & Guiding Principles

1 Overview

In this article, I will delve into the design of extensibility model for Visual Studio UI Test Framework and layout a roadmap which 3^rd party UI Control vendors can use to build UI Testing support for their controls. I will also describe how end-users who have built custom controls on top of supported Microsoft technologies can use the extensibility model to support UI Testing. To set the context, I will also describe the scenarios targeted by Visual Studio UI Test Framework and its high level architecture.

This is the first in a series of articles that I plan to publish on UI Test Framework.

2 Background

In Visual Studio 2010, we introduced new Automated UI Testing capabilities. These are available in Microsoft Visual 2010 Premium and Ultimate SKUs. Manual Testing and Fast forward of manual tests are available in Microsoft Test Professional 2010 & Microsoft Visual Studio 2010 Ultimate. The underlying UI Testing framework implements an extensible architecture where support for different UI Technologies is enabled using a plugin model.

3 User Scenarios

1. Ellen the Manual Tester enables action recording in Microsoft Test Runner. She then executes a manual test case. At the end of the test run, she saves the results. In the next iteration, Ellen can fast forward the test case. The action recording created in the initial test run can be played back in subsequent runs to fast-forward through the test steps. This results in significant productivity gains for Ellen.

2. When Ellen is performing the manual test steps, she can view the actions being recorded. These actions are intent-aware and meaningful. She has the option to delete unwanted actions.

3. Elvis, the Automation Engineer can create Coded UI Test by recording the test actions. Elvis is able to quickly create functional test automation. He is able to select a UI Control on the desktop and view its properties. He can also navigate to the parent, children and siblings of selected control. This navigation capability helps him to easily select controls which would be otherwise difficult to locate. He can add validations on the properties of the selected controls. Elvis is thus able to create powerful functional automated tests.

4. Elvis, the automation engineer can create functional automation using UI Testing API. He can perform test actions and add validations on the properties of controls. He has a rich API with classes defined for each control. Each such class has properties and methods appropriate for it.

Each component in the UI Test Architecture (described below) provides certain capabilities to enable the above scenarios.

4 UI Test Architecture

A high-level layer diagram of the UI Test Framework (with the extensibility points called out) is shown below.

· The lowest layer is of technology (specific) adapters. A technology adapter is a module that understands the corresponding UI technology and provides the UI technology specific services to rest of the modules. E.g.:- For Internet Explorer testing support, we have built an IE plugin which uses MSHTML/DOM to retrieve properties and identify controls hosted with Internet Explorer. Similarly for testing Windows Forms application, we have built a MSAA plugin which uses Microsoft Active Accessibility(MSAA) to retrieve properties and identify controls in Windows forms applications.

· Next up is the technology abstraction layer which helps abstract rest of the code from various technologies.

· After this are the recorder and playback modules.

· The recorder first records the raw steps and based on the filter\aggregation rules, these raw steps are converted into filtered steps or user-intention. E.g.:- To launch an application, you may have opened the Start Menu, navigated to a folder under it and selected the application. These set of actions will be aggregated as a Launch ‘application’ action.

· The playback has the public API for the users to use. Apart from this, it also has property provider to give information on properties supported by each control and browser service for browser specific operations like navigate to URL. E.g.:- A Windows Forms Button has properties such as DisplayText & Shortcut and an Html Edit box has properties such as Text & IsPassword.

· Microsoft Test Runner “interprets” the recording on the fly and calls appropriate API on the playback side.

5 Supported UI Technologies

Fully supported platform Partial solution. Further work required in future to complete Best efforts with known issues. Users can workaround these cases in the code. No major ongoing investment. Currently no support but on the roadmap for future releases Currently no support and none planned for now. Opportunity for partners to add this using the Extensibility support. Docs and samples around extensibility are here. For latest info refer to this article.

The above chart shows the list of supported UI Technologies and the level of support for each. Each of these UI Technologies is implemented as a plugin using the UI Test Extensibility framework.

6 Guiding Principles

The UI Test Framework is heavily dependent on the UI technology being used in the application. There are a large number of UI Technologies present in the market today and new ones are coming up all the time. On top of that, external vendors create additional set of UI Controls which provide richer features for each one of these technologies. The following are the guiding principles Microsoft will use to build UI Testing support for these multitude of UI Technologies:

Microsoft will build and support UI Testing for

· Microsoft platforms [Windows, Internet Explorer, Windows Phone, SharePoint, Office]

· Core controls on Application development technologies [Windows Forms, Windows Presentation Foundation, Silverlight]

Microsoft encourages third party custom UI framework vendors, such as, ComponentOne, DevExpress, Infragistics and Telerik to build and support UI Testing for their custom UI frameworks.

Third party vendors/partners have the opportunity to add UI Testing support for non-Microsoft platforms, such as, Java, Flex, SAP, Chrome, Opera and Safari. NOTE that Microsoft already supports FireFox 3.6 through Visual Studio 2010 Feature Pack 2.

7 UI Test Framework Extensibility

UI Testing on any new technology or custom control can be enabled by implementing the extension points available in UI Test Framework.

If you have built a custom control on top of an existing supported technology, you have the option of reusing the Visual Studio UI Test plugin for that technology. In this case, you will be required to implement appropriate accessibility for your custom control. See the next section for more details.

If you are building support for a new UI technology, you have the choice to determine the level of UI Testing support to enable for your technology. Based on the level of required support, you will have to implement a set of extension points in the UI Test framework. See the following section for more details.

7.1 Custom Control support in an existing UI Technology

If you have a custom control in any of the supported technologies listed above, we recommend that you leverage the existing plugin. E.g.:- If you have a custom Windows Forms control, we recommend you to implement Microsoft Active Accessibility (MSAA) for that control. Similarly, implement UI Automation (UIA) for Windows Presentation Foundation custom controls.

This will ensure that you can leverage the well-tested infrastructure and lessen the overall effort to build UI Testing support for the custom control.

See the following two blogs which describe how to build custom control support.

https://blogs.msdn.com/b/gautamg/archive/2010/03/23/writing-extension-for-wpf-custom-control.aspx

https://blogs.msdn.com/b/tapas_sahoos_blog/archive/2010/12/10/adding-new-control-support-in-a-coded-ui-test-plugin.aspx

Watch out for an upcoming blog on how to implement accessibility for a Winforms Custom control.

Since we are using MSHTML/DOM for Internet Explorer testing, you will have access to all controls from the IE plugin. The only extension point that may be required, if you are building a JavaScript based custom control, is the Action Filter. You may want to aggregate the raw actions on your custom control to an intent aware one. E.g.:- In the jquery datecalendar control, UI Test framework records actions such as click on the date, move to next month etc. Ideally you want to get a single intent aware action that a date has been selected in the date calendar. The multiple raw actions can be aggregated into a single intent aware action by implementing the action Filter extension point in UI Test Framework.

Gautam’s blog describes how to implement an Action Filter extension in the UI Test Framework.

Watch out for an upcoming blog on how to write an action filter for the jquery datecalendar.

7.2 HowTo: Build Support for a new UI Technology

7.2.1 Identify the level of Support

Visual Studio 2010 UI Test Framework provides support for multiple user scenarios as described above. You may choose to support only a few of these scenarios. We have classified the level of support as listed below.

Level 1:- Basic Recording & Playback: Create an action log for a rich bug & fast-forward action recording with Microsoft Test Manager.

Level 1a:- Intent Aware actions: Improve the resilience of your action recording playback & improve the readability of the action log.

Level 2:- Rich Property validation: Write powerful validations in Coded UI Test.

Level 3:- Automation API: Author expressive automation scripts in C# or VB.

For all the UI Technologies supported in the box with Visual Studio, we have implemented all these levels. We recommend that you implement support in the order described above. (Level 1a is optional).

7.2.2 Level 1:- Basic Recording & Playback

For implementing UI Testing support for any new UI framework, you need to decide on the following key questions.

1. How will you determine which controls have been acted on?

2. What are the properties used to uniquely identify a control?

3. How will you bring a control to focus (and thereby make it ready to accept input)?

4. How will you retrieve the information from the application and send it to the UI Test framework.

The UI Testing framework does not mandate any one approach for the questions above. You have the flexibility to choose the most appropriate model for your custom UI Framework. In fact, Microsoft itself has used different approaches to solve these problems for currently supported technologies. (See the list below).

There is a sample Excel plugin for UI Test Framework on MSDN, which uses VSTO and .NET Remoting services as its implementation model.

There are pros and cons for each implementation approach.

· If accessibility technologies are used, we are limited by the accessibility implementation. Most vendors treat accessibility as a primary tenet. There will be greater chance of an accessibility solution being applicable across the board. This again has to be offset by the fact that UI Testing requirements may be more than those required for Accessibility tools. E.g.:- A vendor may have a custom Winforms SplitButton for which he has implemented accessibility. For reliable playback of actions on the SplitButton, it is necessary for UI Testing framework to access the various child controls of the SplitButton. In this case, the vendor has to implement GetChildCount() & GetChild() methods in the Accessibility implementation and return appropriate Accessible objects for each invocation of GetChild().

Watch out for an upcoming blog on implementing accessibility for custom Winforms SplitButton.

· If code injection is used, you need to set up the infrastructure to inject code into the application under test, build the communication channel and clean it up appropriately. In this case, you will get a richer set of features since the plugin is within the application and can access all its properties. E.g.:- In the sample Excel plugin for UI Test Framework on MSDN, VSTO plugin is injected into the Excel application, WCF is used to establish communication between the application under test and the Visual Studio Excel plugin manages the interactions and shuts down the communication channels as required. Since the VSTO component is present inside Excel, it has access to all Excel properties (such as WidthInChars, Formula etc).

NOTE: We have observed that most of the existing custom UI Technology frameworks do not implement sufficient accessibility for Visual Studio UI Test framework to support them seamlessly. Thus it becomes imperative to use the code injection model for most such frameworks. In order to make this easier, we are working on a project to build an infrastructure which will dynamically inject a UI Test Framework plugin component into the application, establish the connection with UI Test framework, and maintain the lifecycle of the injected component and finally cleanup when testing is done.

7.2.3 Level 1a:- Intent Aware Actions

UI Test Framework provides a queue of actions to each plugin. The plugin may filter or modify the actions as appropriate and put updated actions back into the queue. After the action queue is fully processed, it is presented to the user. E.g.:- When launching the calculator application from Start Menu, the following 4 actions will be present in the action queue.

1. Click Start Button

2. Click All Programs

3. Click Accessories

4. Click Calculator.

The built-in action filter removes these 4 actions from the queue and inserts one action back.

1. Launch application ‘calc.exe’

Each plugin writer can thus apply his own filtering rules for his technology to make meaningful actions.

Gautam’s blog describes how to implement an Action Filter extension in the UI Test Framework.

7.2.4 Level 2:- Rich Property Validation

While creating automated tests, it is important to add validations for various controls on the user interface. E.g.:- After purchasing some article in an e-commerce site, you may want to verify that the final price is correct. Therefore it is important that you expose a rich set of properties for each supported control. This can be done by implementing the property provider extension point in the UI Test framework.

Gautam’s blog explains how to achieve this.

7.2.5 Level 3:- Automation API

Coded UI Tests enable users to record actions and generate code for them. This code is then executed when tests are run. The generated code by default uses the base class of the UI Test framework - UITestControl. In order to improve the readability and maintainability of the code, you can implement an extension which will generate code using classes specialized for your technology. E.g.:- We have provided specialized classes for HtmlButton, WinEdit etc for technologies supported in the box by Visual Studio 2010 UI Test Framework.

8 Summary

This article provides an overview of the extensible UI Test Framework in Visual Studio. It describes the targeted user scenarios and design choices to build support for custom UI frameworks. The article also lays out the guiding principles that Microsoft follows in building plugins for the UI Test framework.