Guidelines on improving performance of Coded UI Test playback
In this blog post I’ll go through some approaches you can make use of in your Coded UI Test project to achieve better playback performance. Meanwhile, do check an earlier post here that covers some common scenarios. As I see, there are lot of APIs and settings available that the Coded UI Test user can leverage to hand-code a more optimized test script based on their specific scenarios.
First of all, be aware that Coded UI Test allow users to do pure functional UI testing unlike some other automation tools which does programmatic playback. By pure functional UI testing I mean, it replicates the actual user action by sending the mouse/keyboard inputs to the controls within the screen area. (And because of this, the playback requires an active desktop session). The playback essentially happens in 2 broad phases: Search and
Action. The fraction of time spend on each of them would vary across technologies, controls and type of action, however in general you will observe the action taking a major proportion of the overall playback time.
The search performance would depend on how optimized are the control’s search properties (a.k.a. Query ID). Generating a good Query ID depends on various factors such as unique accessible properties, control depth,
virtualization, etc. The Coded UI Test recorder makes a best effort to optimize it, but keeps a balance between recording performance and the optimized Query that it generates.
Once the control is found, the internal playback engine ensures that the control is brought into the screen area [“EnsureVisible”] and is “ready” to accept the input [“WaitForReady”]. Once the input is sent to the control, it also verifies that the input is actually received by the target control’s window [“UISynchronization”]. And finally, it does (optional, in certain scenarios only) validation of the final expected state of the control.
General guidelines
Extending the auto-generated search hierarchy
In one of the earlier posts here, I’ve explained how to modify the ancestor Query ID hierarchy of a control. This is pretty much important to understand when you are doing a lot of hand-coding of your test script.
Make use of PlaybackSettings
The post here mentions the set of playback settings that can be leveraged to optimized your playback scenario. It is important to understand the fallbacks of each setting to keep a proper balance between performance and false positives.
Use MaxDepth to restrict your search sub-tree.
By default, the playback searches within the entire sub-tree of the container specified. Restricting the depth of the search (BreadthFirstSearch) improves the search performance to a good extend especially in scenarios such as Datagrid.
Give precedence to window search properties compared to accessibility properties when you are framing your control’s search properties.
Window search is faster compared to navigation and search using accessibility tree. The post here explains the concept of window search and windowed properties.
NextSibling can produce faster search
If the control does not have good accessible properties (and let’s assume it’s parent is uniquely identifiable), you can use Instance or NextSibling search configurations to disambiguate the control. NextSibling can be faster in certain plugins such as WPF where search for a virtualized “previous sibling” control (i.e. the NextSibling ancestor) will be quick and then to get the target control is just one navigation step away. Details on NextSibling can be found here and an useful discussion here.
When to use AlwaysSearch configuration option?
This can be used typically for dynamic control scenarios where you would want to ensure that every time you access the control, it does not return any stale reference. This could happen because of pre-designed application behavior as well bug in the application itself.
For example, say your test scenario is to launch a context menu and select a menu item. If the context menu UI object (along with its underlying accessibility object) is not disposed of correctly on menu collapse, for the next iteration when the test launches the context menu and searches for another menu item under it, it would incorrectly hold on to the stale (invisible) Context Menu reference. Using AlwaysSearch (along with VisibleOnly) configuration for the Context menu control will ensure in this case the target menu item is always searched within the correct menu.
Use unique properties for items within containers such as List / Combo Box.
For example, use “Name”/”AutomationId” property instead of “ControlType” (along with “Instance”) to search for an item control. You can see a visible difference in performance (for specifically for virtualized controls).
Scenario specific guidelines
CopyPastedText property
This is available across all plugins except Silverlight. Use this to speed up your form filling scenarios. Unlike Text property, this does not ensure that the contents of the edit control are cleared before the value is entered. So you need to hand-code something like –
control.Text = String.Empty;
control.SetProperty(WinControls.WinEdit.PropertyNames.CopyPastedText, <newValue>);
SelectedItems versus SelectedIndices property
For large number of items, it is preferable to use SelectedItems. SelectedIndices essentially does an Instance search internally, which is slower compared to unique automation property based search. Use SelectedIndices only when the items don’t have unique Automation properties. And of course exercise caution over changes to the list content.
Expanding all the nodes in a Tree View control
There are various ways to achieve this. You can recursively search for the Tree Item nodes at each level and invoke .Expanded = true on the expandable tree item. Instead you can make use of your tree view control default response to keys. Something like –
- Select the root node of the Tree View control.
- Do SendKeys(“*”) on the Root node.
Post here has some relevant performance related discussion.
Be cautious about Items property usage.
I have come across lot of instances where user tried to do a
for (int i=0; i < listControl.Items.Count; i++) { UITestControl item = listControl.Items[i]; /* do something with the item */ }
There is no caching involved here. Effectively the above code fetches all the items N number of times which is a very expensive operation. Cache the UItestControlCollection before you use it.
WinList, WinTable, WinTree comparison
Seen some queries from the users regarding this. Essentially what they are looking at is take snapshot of 2 containers and compare them or do some validation. What I would suggest is, try as much as possible to bind the snapshot information to some accessibility information of the container and retrieve that property value. Related discussion here.
Using Exists property
If the performance is not up to the expectation, check if the playback settings such as SearchTimeout or SmartMatch can be tuned to get it fixed. A related discussion is available here.
Debugging performance issues
If the playback is taking a lot of time, warning or error messages in the logs would show up some useful information
“QTAgent32.exe, Playback – [WARNING] Internal warning: Element “[MSAA, VisibleOnly]ControlType=’Button’ && Name=’ABC” was not found, so all intermediate elements were ignored. To improve performance replace this element Id with another Id or exclude it from QueryId. If this element is the target element, replace or remove intermediate elements”
- This indicates something is wrong about the Query ID. One of the ancestors is not reachable (or does not exist anymore). Try to fix the search properties of this ancestor or fix the hierarchy itself to include some other ancestor. Some more explanation provided here.
“QTAgent32.exe, Playback – [WARNING] Internal warning: Timeout for thread state based WFR expired. Consider calling SetRobustnessLevel(0) or SetRobustnessLevelEx with proper parameters”
- Check if any of the application threads (primarily the UI thread) are busy. Use modified Playback.PlaybackSettings.WaitForReadyLevel and Playback.PlaybackSettings.WaitForReadyTimeout settings to validate this claim, but the correct way would be to fix any performance bottlenecks in the application itself.
“QTAgent32.exe, Playback – [WARNING] Internal warning: Window with Class “ClassX” and Title “TitleX” was not found, so similar window with Class “ClassY” and Title “TitleY” was accepted.”
- This is a typical case of Smart Match kicking in. Smart Match happens in the 3rd phase of search (after exact match search and other search attempts such as SearchInMinimizedWindow). Unfortunately Coded UI Test does not have a direct regex match capability yet.
“QTAgent32.exe, PERF WARNING: CacheQueryId: took 150 ms. Expected it to take maximum 100 ms.”
- We might hit upon this warning messages once in a while during recording. In all probability this would hint that the control is deeply nested and generating the hierarchy is taking time. Another option (which cannot be completely ruled out) is that the fetching the properties of the control is taking time; probably the accessibility calls (MSAA, UIA, whatever the case may be) are taking more time, so some experimentation is required to validate this claim.
“QTAgent32.exe, Playback – [FAILED] SetValueAsEditBox “ABC” – “[Web, AlwaysSearch]ControlType=’Edit’; TagName=’INPUT’; Name=’username'”; Primitive “SetValueAsEditBox” did not set the value correctly”
- Could be case of UISynchronization failure (i.e. the key inputs are not getting received at the control properly) or some SetFocus issues with the Edit control. One more symptom will be, you might observe duplicate characters being typed into the Edit control. Try using CopyPastedText option, or plain SendKeys instead. If the value is indeed being set, but the verification still fails, try using the SkipSetPropertyVerification settings.
Keep track of the page loading time for Web / Silverlight scenarios
Check for the page loading time and/or for presence of AJAX timers running in the background. The WaitForReady logic waits for the document to be in the ready state. In case of Silverlight, the WaitForReady logic waits till the controls have been rendered. One related link here talks about how search can be affected due to slow page loading in Silverlight and how you can use the WaitForReady setting to indirectly configure the search timeout.
Beyond this, you can try out various other neat tricks to automate your scenario with better performance. Some random ones I can think of –
Navigate through Cells in a Datagrid column.
Various ways to achieve this, for example iterating through each Row and using ColumnIndex to get the Cell (or using RowIndex/columnIndex directly). How about you search for the first Cell and then using vicinity logic to get to the next Cell underneath it ? i.e. Using the bounding rectangle information of first Cell, then using some Y-offset to get to the next Cell and do a UITestControlFactory.FromPoint(). Of course you need to hand-code this more carefully i.e. start off with bringing the first Cell in focus; and once you reach the last visible Row, do some EnsureClickable to bring the next set of Rows in view port; and likewise. By this approach, you are effectively eliminating the entire search
for the Cells !
Use inner accessible object for programmatic playback.
You can get hold of the accessible object from UITechnologyElement.NativeElement and use accessible methods/properties directly on them. For example, in case of WPF you can get hold of the inner AutomationElement reference of the UITestControl. [Note, not all plugins in coded UI test expose this in the manner that they can be used as I just mentioned; Silverlight plugin is one instance]
Light
Dark
0 comments