UITest framework in Visual Studio 2010 – Part 2

In Part 1 of this blog we saw UITest architecture and how it is used in Microsoft Test runner. In this blog we will see how it is used in CodedUITest and how to troubleshoot and get resilient playback.

Automated Functional Tests using CodedUITest

CodedUITest can be used to automate the manual test actions in the form of C#/VB code to play them back later. One can generate the code for manual testing actions on the application under test (AuT) using one of the two ways:

1. From the recorded strip stored in TFS (mentioned above)

(For more details refer: https://msdn.microsoft.com/en-us/library/dd286652(VS.100).aspx)

2. By doing recording using CodedUITestBuilder (For more details refer https://msdn.microsoft.com/en-us/library/dd286681(VS.100).aspx)

The generated code can either in the form of C# or VB.net syntax. Once code is generated the CodedUITestBuilder also allows users to add validation of the GUI based on UI Object properties. The generated code can be modified in order to enhance the testing.

Please take a look at the below blogs to know about creating CUIT tests

How to: Generate a Coded UI Test from an Action Recording

How to: Generate a Coded UI Test by Recording the Application Under Test

How to: Add UI Controls And Validation Code Using the Coded UI Test Builder

Customizing CodedUITest

This section elaborates on the playback settings and when it is advised to tweak these settings in code.

Continue On Error :

There might be instances where some actions which we record during playback may or may not appear during playback. For ex: Say we record actions on ‘Save password dialog’ during recording. This dialog might not appear during playback. To prevent playback from failing on this action & to proceed with the other actions we use the ContinueOnError flag. This flag is set to false by default implying that we would fail whenever playback fails to perform any action. We can make playback to continue executing by turning on this flag. This is done by the following set property statement:

Playback.PlaybackSettings.ContinueOnError = true;

Note : Make sure to turn this flag back to false for actions where potential failures are not expected.

SetProperty Verification:

Every time playback does a set-property action (SetValue in a textbox, setstate of a checkbox etc) the playback verifies the state of the control after performing the action. If this verification fails then the action playback fails. There might be certain applications where doing a setvalue might trigger some logic by which the state of the control changes. In these cases, the playback verification fails. To avoid playback failing in these cases, we can temporarily disable SetProperty Verification by the following set property statement

Playback.PlaybackSettings.SkipSetPropertyVerification = true;

Note: Make sure to turn this flag back to false when you don’t expect the application to behave in the manner stated above.

Search Time Out & Search Fail Fast:

Search timeout indicates how long the playback should wait to locate the control. By default this is set to 2 minutes. In the case of search using IE plugin, we don’t hit this timeout as the IE plugin implements a retry mechanism. The IE plugin does a 3 time retry for searching a control. Search inturn calls WFR(Wait For Ready) to make sure the document is in a ready state. Once the document is in a ready state Search in IE Plugin is very fast as accessing the DOM using mshtml is very quick. So the time taken for performing 3 retries is much lesser than the default search timeout of 2 minutes. By this retry mechanism the IE Plugin is reasonably confident that the search will not succeed and it fails fast after 3 retries.

The user however has an option to disable fail fast. If you are sure that the control will become available after a specified duration you can increase the search time out & disable the fail fast mechanism.

Playback.PlaybackSettings.SearchTimeOut = 180000; // Value in milliseconds

Playback.PlaybackSettings.ShouldSearchFailFast = false; // This would disable fail fast fast and playback would wait for the full duration of search timeout for the control to be available

WaitForReadyTimeOut:

WaitForReady is a mechanism to ensure that the controls are being searched only when the document is in a state where user actions are possible. This is accomplished by tracking a number of parameters like timers in a page, pending ajax requests, percentage of navigation etc into consideration. By default the waitforready timeout is 60 seconds. There is a fail fast mechanism in WFR similar to search where if the state of the page isn’t in a ready state and the state doesn’t change for 50% of WFR timeout (30 seconds) we exit WFR and continue to search. There might be applications which might take a long time to receive the ajax responses, or might take a long time for navigation etc. In these cases we can increase the WFRTimeout to make sure the page is in a ready state thereby ensuring that the actions we perform on the page doesn’t fail. We can change the WFR timeout by the following setproperty statement:

Playback.PlaybackSettings.WaitForTimeOut = 120000; // Setting the timeout to 2 mins.

WaitForControlEnabled

This is actually not a playback setting. This is a setting specific to a UITestControl. By using this setting we can block the playback thread until the control becomes enabled. If you are sure that the control will get enabled after a delay we can use this setting.

UITestControl.WaitForControlEnabled(int millisecondsTimeout)

Blocks the current thread until the control is ready to accept mouse/keyboard input, or the specified timeout expires.

MatchExactHeirarchy

This setting is specified to indicate whether to match exact hierarchy specified for searching the control. Each control is specified as a hierarchy of containing controls for the search. If this flag is set to true, all ancestors of the control will be searched in order to locate this control. If this flag is set to false, some of the ancestors may be skipped during the search. For example, if the hierarchy of a control is A;B;C;D if A is found but playback fails to find full hierarchy B->C->D under it, then directly search for D under A if this property is set to false. This is set to false by default. To turn on this feature you can set it by the following setproperty statement.

Playback.PlaybackSettings.MatchExactHeirarchy = true;

ImeLanguageList

IMEs are components that allow the user to enter the thousands of different characters used in East Asian languages using a standard keyboard. The IME engine attempts to guess which ideographic character or characters the keystrokes should be converted into and lists them in a dropdown box. The aggregator in the recorder module aggregates the keyboard actions performed with IME enabled into an appropriate setvalue action. Playback uses copy paste to play back keyboard related actions for these language controls. The list of IME languages for which IME ends composition on pressing the {Enter} key that are included by default are : Chinese (People's Republic of China), Chinese (Taiwan), Japanese and Yi.

For UITest framework to support an IME language, we need to add the IME enabled language to the config settings by adding the Locale Id of the language to the comma separated list of IMELanguages node in the MTM.exe.config file.

SendKeysAsScanCode

Playback by default sends keys to any application as Unicode. It might be necessary to send keys as scancode to certain applications. We can do this by using an over load of Keyboard.Sendkeys function.

Workarounds using Playback Settings for special cases

The configuring of playback settings listed in the above section show how they can be manipulated in a CodedUITest project. At the same time there are certain settings that can be set globally in each machine. In case of MTM, the user can configure the playback settings by editing the mtm.exe.config file or CodedUITestBuilder.exe.config for CUIT present at %VSINSTALLDIR%\Common7\IDE.

The table lists the different playback settings, a high level description of the setting and whether the setting can be configured via Coded UI Test(CUIT) and Microsoft Test and Lab Manager(MTM).

How to get resilient playback?

Recording

1) Determine whether you will launching the application along with the test case or the testcase assumes the AuT is already launched

2) If you are launching the application in the beginning of the testcase, then create the shortcut to the AuT in desktop, quick launch bar or in the start menu. This will ensure that Launch Application is recorded.

3) Ensure that AuT is ready to take user actions before doing the action

4) Remember that recording is one time operation so that do mindful about what actions are you recordingDuring recording watch the recorded steps in the MTM/CUIT Builder window and ensure that right set of actions got recorded. You can even delete actions if they are unwanted.

5) If you are recording actions on OS components please be aware that it may fail while playingback on other OS. The recorder aggregates certain actions automatically like start menu, task bar, quick launch etc to enable playback on other OS but not all actions are aggregated.

6) Create marker segments by clicking pass button in MTM or GenerateMethod button in CUITBuilder if you have accomplished a set of actions that are related and can be grouped as one entity. This way you can decide to playback only those portions during playback.

7) Use shared steps so that you can reuse the recording in many testcases. Maintainance will be easier.

8) Once the testcase is done determine you want to close the AuT as part of the testcase. If not, try to bring the AuT to a known state so that the next test case can be played back or recorded.

Playback

1) Based on the action log, determine whether the AuT is launched as part of playback or you have to keep the AuT opened before playback. In many cases this can be confusion and you may get a Seacrh failure.

2) If the AuT is launched as part of the testcase, then ensure that the AuT is installed in the same location as in the time of recording. Otherwise playback will fail to launch the application

3) Ensure that the state of the application is same as it was recorded.

4) Do not lock the machine or screen saver is not on, playback need an active session

5) Play back of the strip will fail, if an object on which a user performs action during recording is not available during playback. For example, if you are getting a dialog box in your scenario which has an option to not to show this dialog in future and you wish to choose that option, please pause the recording by clicking on pause button in MTM and check the “Do not show this dialog” option and choose Ok or Cancel. Basically if you record an action on a dialog box which only appears during recording, playback will fail because that dialog box will not show up second time which playback will expect. Remember password, Enable AutoComplete are some examples for this.

6) Please do not move your mouse or perform any actions during playback, as this might end up in playback failure.

Supported & Non-supported configurations

Exceptions & what they mean

Playback on IE can fail due to various reasons and the playback engine throws the best exception out of a variety of playback exceptions for the user to rootcause the reason for the failure.

Recording Time Failures

Playback time failures

Playback does two major things in order to perform action on the control search and action on the control. Following are the possible reasons and their exceptions.

SearchFailure

Following are some of the suggested tips to debug the search failure during playback.

• See screenshot (stored in %temp%\uitestlogs\lastrun if logs are enabled) to see whether the control was actually visible or not.
• Verify the search properties and UI hierarchy (QueryID) using IE Developer Toolbar.

Action Failure

Sometime though the search passes but the playback may not be able to do the action on the control. Following may be the possible reason/tips to debug the issue

• Verify if there is any other control with the same properties under the parent mentioned in the QueryID. Some time there may be more than one control with the same properties under a parent hence during playback the first control will be selected and the action may fail on it. User can work around it after adding the instance property in the code for the ambiguous control.
• User can use .DrawHighLight() method to actually see which control is chosen for the UITestAction. This method is available for all the UITestControl and will draw a rectangular highlighter around the control.
• Verify whether the control state is same as it was during recording e.g., SetValue was recorded on a text box but during playback the text box is read only.

Enable Trace logs for troubleshooting

Sometimes the exceptions thrown by the UITest framework may not be enough to debug a failure. In those instances you can enable RnP trace logs which contains debugging information to troubleshoot failures. Below blog contains more information on how to enable trace logs.

https://blogs.msdn.com/gautamg/archive/2009/11/29/how-to-enable-tracing-for-ui-test-components.aspx

Summary

In this blog we saw the UITest framework engine of VS 2010 which is used in Fast Forward and CodedUITest features. There will be more blogs that will be followed to explain each technology that this record & play engine supports.