A custom data diagnostic adapter for Visual Studio Coded UI Test using Windows PSR
This is a guest blog from the Windows Azure AppFabric team. They are using Visual Studio Coded UI Test and want to share an exciting idea for generating rich troubleshooting logs.
Hello, my name is Jerrin Elanjikal and I’m a test developer with the Windows Azure AppFabric product group at Microsoft IDC. My team develops automated UI test frameworks and tools to ensure the quality of our user interface components. We deal a unique set of technical challenges around UI test automation and debugging, which is very different from both API test automation and manual UI testing.
Our product has multiple user interfaces including a Silverlight web portal, Visual Studio designer surfaces, WPF dialogs and a few legacy WinForms dialogs. We target 85% code coverage through automated UI test cases. And most of our UI automation is done using Coded UI, along with a mix of internal frameworks for specialized functions.
When one of these test fail and you have to debug or reproduce the failure, we run into a few common pain points that are unique to UI test automation:
1. The average test takes a lot more time to run as compared to your typical API unit test. Our tests on average take about 10 minutes each.
2. And while a UI test is running, you cannot use your computer for anything else that involves user interaction. This means that you either keep a dedicated second machine for debugging UI test failures, or just sit back and watch.
3. UI tests can fail randomly due to a number of external reasons – a pop up thrown by another application, someone accidently logging into the machine, flaky internal test frameworks and the list goes on. It’s almost impossible to reproduce or debug such failures on your development machine.
The standard solution to these problems involves diagnostic logs and video recording of the test automation, so that you can debug without having to re-run the test. Often logs alone are not sufficient. Video recordings take up a lot of disk space, typically a few MB per test case. And you spend a lot of time watching the video and looking out for something interesting to happen.
Fortunately, there is a better alternative. The ‘Problem Steps Recorder’ is an awesome little tool that comes built in with Windows 7 and later. PSR was originally intended to help customers explain their problem scenario to tech support. What it does is take screenshots of each UI action and generate a zipped html report with images and descriptions. Looking at a PSR report is like watching the highlights of a match – short & interesting. It’s every UI developer’s dream come true!
So, what we did is plug PSR into our CodedUI tests by writing a custom diagnostics data adapter that essentially just invokes psr.exe with the right parameters. This MSDN article < http://msdn.microsoft.com/en-us/library/dd286727.aspx > describes how to write and install a custom diagnostics data adapter for Visual Studio. And the command line parameters for PSR are as follows:
psr.exe [/start |/stop][/output <fullfilepath>] [/sc (0|1)] [/maxsc <value>]
[/sketch (0|1)] [/slides (0|1)] [/gui (o|1)]
[/arcetl (0|1)] [/arcxml (0|1)] [/arcmht (0|1)]
[/stopevent <eventname>] [/maxlogsize <value>] [/recordpid <pid>]
/start :Start Recording. (Outputpath flag SHOULD be specified)
/stop :Stop Recording.
/sc :Capture screenshots for recorded steps.
/maxsc :Maximum number of recent screen captures.
/maxlogsize :Maximum log file size (in MB) before wrapping occurs.
/gui :Display control GUI.
/arcetl :Include raw ETW file in archive output.
/arcxml :Include MHT file in archive output.
/recordpid :Record all actions associated with given PID.
/sketch :Sketch UI if no screenshot was saved.
/slides :Create slide show HTML pages.
/output :Store output of record session in given path.
/stopevent :Event to signal after output files are generated.
To start PSR recording at the beginning of each test case, we invoke psr.exe with the parameters /start /output <PSRFileName> /gui 0 /sc 1 /sketch 1 /maxsc 100 in the OnTestCaseStart event handler of our custom diagnostics data adapter.
To complete the PSR recording and generate the zipped report we invoke psr.exe /stop in the OnTestCaseEnd event handler, with an appropriate delay to complete the report file creation.
But PSR records a screenshot only when a UI action is performed. If the test fails because a UI element was not found on screen, as is often the case, PSR will not record a screenshot at that point. So we add the following code to the OnTestCaseEnd event handler to take an extra screenshot at the point of failure.
Playback.Initialize();
Image img = UITestControl.Desktop.CaptureImage();
Playback.Cleanup();
img.Save(ScreenshotFileName, ImageFormat.Jpeg);
We then attach the PSR report and the final screenshot to the test case.
dataSink.SendFileAsync(e.Context, ScreenshotFileName, false);
dataSink.SendFileAsync(e.Context, PSRLogFile, false);
The complete code for the PSR diagnostics data collector is given below:
using System;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Imaging;
using System.IO;
using System.Threading;
using System.Xml;
using Microsoft.VisualStudio.TestTools.Common;
using Microsoft.VisualStudio.TestTools.Execution;
using Microsoft.VisualStudio.TestTools.UITesting;
namespace PSRDataCollector
{
// Problem Screen Recorder diagnostic data adapter
[DataCollectorTypeUri("datacollector://Microsoft/PSRDataCollector/1.0")]
[DataCollectorFriendlyName("Problem Screen Recorder", false)]
public class PSRDataCollector : DataCollector
{
#region Constants
private const string PSRExe = "psr.exe";
private const int SaveDelay = 5000;
#endregion Constants
#region Fields
private DataCollectionEvents dataEvents;
private DataCollectionLogger dataLogger;
private DataCollectionSink dataSink;
private XmlElement configurationSettings;
private string PSRLogFile;
#endregion Fields
#region DataCollector
// Required method called by the testing framework
public override void Initialize(XmlElement configurationElement,
DataCollectionEvents events,
DataCollectionSink sink,
DataCollectionLogger logger,
DataCollectionEnvironmentContext environmentContext)
{
dataEvents = events; // The test events
dataLogger = logger; // The error and warning log
dataSink = sink; // Saves collected data
// Configuration from the test settings
configurationSettings = configurationElement;
// Register common events for the data collector
// Not all of the events are used in this class
dataEvents.SessionStart +=
new EventHandler<SessionStartEventArgs>(OnSessionStart);
dataEvents.SessionEnd +=
new EventHandler<SessionEndEventArgs>(OnSessionEnd);
dataEvents.TestCaseStart +=
new EventHandler<TestCaseStartEventArgs>(OnTestCaseStart);
dataEvents.TestCaseEnd +=
new EventHandler<TestCaseEndEventArgs>(OnTestCaseEnd);
dataEvents.DataRequest +=
new EventHandler<DataRequestEventArgs>(OnDataRequest);
}
protected override void Dispose(bool disposing)
{
if (disposing)
{
dataEvents.SessionStart -=
new EventHandler<SessionStartEventArgs>(OnSessionStart);
dataEvents.SessionEnd -=
new EventHandler<SessionEndEventArgs>(OnSessionEnd);
dataEvents.TestCaseStart -=
new EventHandler<TestCaseStartEventArgs>(OnTestCaseStart);
dataEvents.TestCaseEnd -=
new EventHandler<TestCaseEndEventArgs>(OnTestCaseEnd);
dataEvents.DataRequest -=
new EventHandler<DataRequestEventArgs>(OnDataRequest);
}
}
#endregion DataCollector
#region Event Handlers
public void OnSessionStart(object sender, SessionStartEventArgs e)
{
// TODO: Provide implementation
}
public void OnSessionEnd(object sender, SessionEndEventArgs e)
{
// TODO: Provide implementation
}
public void OnTestCaseStart(object sender, TestCaseEventArgs e)
{
this.PSRLogFile = Path.Combine(Environment.CurrentDirectory, e.TestCaseName + ".psr.zip");
// Kill all running PSR processes
TryKillProcess(PSRExe);
// Start PSR
try
{
InvokeProcess(PSRExe, String.Format("/start /output "{0}" /gui 0 /sc 1 /sketch 1 /maxsc 100", this.PSRLogFile));
}
catch (Exception exp)
{
dataLogger.LogError(e.Context, string.Format("Unexpected exception while trying to start PSR process : {0}", exp));
}
}
public void OnTestCaseEnd(object sender, TestCaseEndEventArgs e)
{
try
{
// Save the PSR logs
InvokeProcess(PSRExe, @"/stop").WaitForExit(60000);
// Sleep to ensure PSR completes file creation operation
Thread.Sleep(SaveDelay);
if (!File.Exists(this.PSRLogFile))
{
dataLogger.LogError(e.Context, "No user actions were recorded by PSR!");
}
else if (e.TestOutcome != TestOutcome.Passed)
{
string ScreenshotFileName = Path.Combine(Environment.CurrentDirectory, e.TestCaseName + ".screenshot.jpg");
CaptureScreenshot(ScreenshotFileName);
dataSink.SendFileAsync(e.Context, ScreenshotFileName, false);
dataSink.SendFileAsync(e.Context, this.PSRLogFile, false);
}
else
{
File.Delete(this.PSRLogFile);
}
}
catch (Exception exp)
{
dataLogger.LogError(e.Context, string.Format("Unexpected exception while trying to stop PSR process : {0}", exp));
// Kill all PSR processes
TryKillProcess(PSRExe);
}
}
public void OnDataRequest(object sender, DataRequestEventArgs e)
{
// TODO: Provide implementation
// Most likely this occurs because a bug is being filed
}
#endregion Event Handlers
#region Helpers
private Process InvokeProcess(string processName, string parameters)
{
ProcessStartInfo startInfo = new ProcessStartInfo(processName, parameters);
startInfo.WindowStyle = ProcessWindowStyle.Hidden;
startInfo.UseShellExecute = true;
startInfo.ErrorDialog = false;
Process proc = new Process();
proc.StartInfo = startInfo;
proc.Start();
return proc;
}
private void TryKillProcess(string processName)
{
Process[] processes = Process.GetProcessesByName(processName);
foreach (Process proc in processes)
{
try { proc.Kill(); }
catch (Exception exp) { }
}
}
private void CaptureScreenshot(string ScreenshotFileName)
{
Playback.Initialize();
Image img = UITestControl.Desktop.CaptureImage();
Playback.Cleanup();
img.Save(ScreenshotFileName, ImageFormat.Jpeg);
}
#endregion Helpers
}
}
You can download the full project from http://psr4vs.codeplex.com.
To install the PSR data collector into Visual Studio, download and build this project. Copy PSRDataCollector.dll to %VSINSTALLDIR%Common7IDEPrivateAssembliesDataCollectors.
To enable PSR data collection, open test settings from Solution Explorer in your CodedUI test and enable ‘Problem Screen Recorder’ under the ‘Data and Diagnostics’ tab.
Run your CodedUI test again and let it fail. Wait for the test run session to complete. Now, you will find the PSR report and final screenshot attached to the test report.
Debugging automated UI tests just got a whole lot easier!
Light
Dark
0 comments