- Overview of TestApi
- Part 1: Input Injection APIs
- Part 2: Command-Line Parsing APIs
- Part 3: Visual Verification APIs
- Part 4: Combinatorial Variation Generation APIs
- Part 5: Managed Code Fault Injection APIs
- Part 6: Text String Generation APIs
- Part 7: Memory Leak Detection APIs
- Part 8: Object Comparison APIs
Visual Verification (VV) is the act of verifying that your application or component is displayed correctly on screen. The TestApi library provides a set of VV APIs. This post discusses these APIs.
Avoid It If You Can
First and foremost, I want to emphasize that visual verification is a test technique that should be used with caution. It is difficult to do correctly and any extensive use typically results in hard-to-maintain test codebases. Here are a few things to consider before you embark on VV test development:
- The UI of an application tends to change a lot during development. If you use visual verification extensively, you may end up in a situation where you have to update a large number of visual verification tests on a daily basis, which is a waste of effort.
- There are small differences in rendering between different video cards on different versions of the OS and of .NET. These differences are particularly pronounced in font rendering.
- Before employing any form of UI verification, one should always review the underlying application architecture. Extensive need for UI testing is typically indicative of poorly architected systems, lacking proper view-model separation, so it’s almost always better to invest in proper system architecture than in extensive UI testing.
- Whenever possible, attempt to do analytical visual verification i.e. one that does not employ the use of master images.
The WPF test team has gone through several iterations of cleaning up and retiring unnecessary visual verification tests in an attempt to speed up and stabilize our test suite.
The core VV terminology is:
- Snapshot: A pixel buffer used for representing and evaluating screen image data.
- Verifier: An oracle object which determines whether a snapshot passes against specified inputs.
- Actual: The snapshot being evaluated.
- Master: The reference data (image) which is used to evaluate the actual snapshot.
- Tolerance: The accepted bounding range based on which the actual snapshot will be accepted as valid.
The general VV workflow is:
- Capture some screen content.
- Generate an expected snapshot (e.g. load a master image from disk, etc.)
- Compare the actual snapshot to the expected snapshot and generate the difference (diff) snapshot.
- Verify the diff using a verifier.
- Report test result.
TestApi Visual Verification Technology
TestApi provides the following VV technology:
- Snapshot: this class represents image pixels in a two-dimensional array for use in VV. Every element in the array represents a pixel in a given [row, column] of the image. A Snapshot object can be instantiated from a file (Snapshot.FromFile), or captured from screen (Snapshot.FromWindow and Snapshot.FromRectangle). Snapshot also exposes image cropping (Snapshot.Crop), resizing, diff-ing (Snapshot.CompareTo) masking (Snapshot.And) and merging operations (Snapshot.Or) operations.
- Verifiers: the library provides a set of verifiers that can be used to verify a (diff) snapshot. SnapshotColorVerifier reports passing if
- Various utilities: example of these are the Histogram class, providing basic functionality for handling image histograms
With these prolegomena out of the way, let’s look at some code. The first example below demonstrates master visual verification using a basic color verifier, which ensures that the difference between the master snapshot and the actual snapshot is within a defined tolerance:
This approach works fine if you are evaluating the correctness of an application logo or some other application art. You may find that you will need to increase the tolerance a bit to accommodate differences in GPU rendering, but in general the SnapshotColorVerifier provides all the functionality you need.
A somewhat more sophisticated approach involves using of image histograms (see this link for a good introduction to the subject). An image histogram is a histogram that represents the frequency of pixels with a certain brightness. One can define a histogram that represents his/her expectation of the “proximity of the match” between the actual and expected snapshots.
For example, one can define a histogram semantically equivalent to the following statement:
“When I compare the actual snapshot to the expected snapshot (both of 320 pixels), I expect no more than 30 pixels with color channel difference of 1, no more than 10 pixels with color channel difference of 2, and zero pixels with higher differences.”
This histogram would look as follows:
Figure 1 Image Histogram
Such form of verification is done by using the SnapshotHistogramVerifier and the Histogram classes, as demonstrated in the sample below.
The histogram file is just a XML file with the following schema:
This visual verification approach was pioneered in the WPF test organization about 6 years ago by Marc Cauchy and Pierre-Jean Reissman.
However, none of the two approaches above work particularly well for evaluation of a typical application window, containing controls, text, etc. Such windows tend to have regions that need different tolerance settings. For example, consider the application window below:
Figure 2 Sample Application Window
If you try to perform master based visual verification, you will hit 2 issues:
- The non-client area of the window (the window frame) will tend to be slightly different between different runs of the application. It will also depend on environment factors such as the desktop wall-paper.
- Some regions of the client area of the window will also tend to exhibit significant variance between runs of the application.
Issue (1) is easy to resolve, using Snapshot.FromWindow(…) and excluding the non-client area from the capture. Issue (2), however, is a bit more involved. Here are the expected, actual and diff snapshots of the client-area of the application.
Figure 3b Actual Client-Area Snapshot
The differences between the expected snapshot and the actual snapshot are difficult to see on Figure 3c, so on Figure 3d I have replaced purely black pixels with pink.
It is not surprising that most of the variation occurs around the text regions in the application window (ClearType renders differently on different machines). So it may make sense to increase the tolerance (or completely mask away) those regions, providing of course we are not specifically interested in their rendering.
In order to achieve that, we use the SnapshotToleranceMapVerifier class. Here’s an example:
The tolerance map that we use looks as follows:
What appears pure black (0x00FFFFFF) is actually an off-black color (0x000A0A0A) to handle the small variations that appear as black dots on Figure 3d. Then we also have 4 regions with significantly higher tolerance to handle the variability of the text rendering.
The visual verification API in TestApi provides a solid foundation for visual verification tests. As a general best practice, however, avoid visual verification as much as possible.