Introduction to TestApi – Part 3: Visual Verification APIs

Series Index

+++

Visual Verification (VV) is the act of verifying that your application or component is displayed correctly on screen. The TestApi library provides a set of VV APIs. This post discusses these APIs.

 

Avoid It If You Can

First and foremost, I want to emphasize that visual verification is a test technique that should be used with caution. It is difficult to do correctly and any extensive use typically results in hard-to-maintain test codebases. Here are a few things to consider before you embark on VV test development:

  1. The UI of an application tends to change a lot during development. If you use visual verification extensively, you may end up in a situation where you have to update a large number of visual verification tests on a daily basis, which is a waste of effort.
  2. There are small differences in rendering between different video cards on different versions of the OS and of .NET. These differences are particularly pronounced in font rendering.
  3. Before employing any form of UI verification, one should always review the underlying application architecture. Extensive need for UI testing is typically indicative of poorly architected systems, lacking proper view-model separation, so it’s almost always better to invest in proper system architecture than in extensive UI testing.
  4. Whenever possible, attempt to do analytical visual verification i.e. one that does not employ the use of master images.

The WPF test team has gone through several iterations of cleaning up and retiring unnecessary visual verification tests in an attempt to speed up and stabilize our test suite.

 

General Concepts

The core VV terminology is:

  • Snapshot: A pixel buffer used for representing and evaluating screen image data.
  • Verifier: An oracle object which determines whether a snapshot passes against specified inputs.
  • Actual: The snapshot being evaluated.
  • Master: The reference data (image) which is used to evaluate the actual snapshot.
  • Tolerance:The accepted bounding range based on which the actual snapshot will be accepted as valid.

The general VV workflow is:

  1. Capture some screen content.
  2. Generate an expected snapshot (e.g. load a master image from disk, etc.)
  3. Compare the actual snapshot to the expected snapshot and generate the difference (diff) snapshot.
  4. Verify the diff using a verifier.
  5. Report test result.

 

TestApi Visual Verification Technology

TestApi provides the following VV technology:

  • Snapshot: this class represents image pixels in a two-dimensional array for use in VV. Every element in the array represents a pixel in a given [row, column] of the image. A Snapshot object can be instantiated from a file (Snapshot.FromFile), or captured from screen (Snapshot.FromWindow and Snapshot.FromRectangle). Snapshot also exposes image cropping (Snapshot.Crop), resizing, diff-ing (Snapshot.CompareTo) masking (Snapshot.And) and merging operations (Snapshot.Or) operations.
  • Verifiers: the library provides a set of verifiers that can be used to verify a (diff) snapshot. SnapshotColorVerifier reports passing if
  • Various utilities: example of these are the Histogram class, providing basic functionality for handling image histograms

 

Examples

SnapshotColorVerifier

With these prolegomena out of the way, let’s look at some code. The first example below demonstrates master visual verification using a basic color verifier, which ensures that the difference between the master snapshot and the actual snapshot is within a defined tolerance:

 // 1. Capture the actual pixels from a given window
Snapshot actual = Snapshot.FromRectangle(new Rectangle(0, 0, 100, 100));

// 2. Load the reference/master data from a previously saved file
Snapshot expected = Snapshot.FromFile("Expected.png"));

// 3. Compare the actual image with the master image
//    This operation creates a difference image. Any regions which are identical in 
//    the actual and master images appear as black. Areas with significant 
//    differences are shown in other colors.
Snapshot difference = actual.CompareTo(expected);

// 4. Configure the snapshot verifier - It expects a black image with zero tolerances
SnapshotVerifier v = new SnapshotColorVerifier(Color.Black, new ColorDifference());

// 5. Evaluate the difference image
if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the diff file for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

 

This approach works fine if you are evaluating  the correctness of an application logo or some other application art. You may find that you will need to increase the tolerance a bit to accommodate differences in GPU rendering, but in general the SnapshotColorVerifier provides all the functionality you need.

SnapshotHistogramVerifier

A somewhat more sophisticated approach involves using of image histograms (see this link for a good introduction to the subject). An image histogram is a histogram that represents the frequency of pixels with a certain brightness. One can define a histogram that represents his/her expectation of the “proximity of the match” between the actual and expected snapshots.

For example, one can define a histogram semantically equivalent to the following statement:

“When I compare the actual snapshot to the expected snapshot (both of 320 pixels), I expect no more than 30 pixels with color channel difference of 1, no more than 10 pixels with color channel difference of 2, and zero pixels with higher differences.”

This histogram would look as follows:

image

Figure 1  Image Histogram

 

Such form of verification is done by using the SnapshotHistogramVerifier and the Histogram classes, as demonstrated in the sample below.

 // Take a snapshot, compare to the master image and generate a diff
Snapshot actual = Snapshot.FromRectangle(new Rectangle(0, 0, 100, 100));
Snapshot expected = Snapshot.FromFile("Expected.png"));
Snapshot difference = actual.CompareTo(expected);

// Load the quality histogram from disk and use it to verify the diff
SnapshotVerifier v = new SnapshotHistogramVerifier(Histogram.FromFile("ToleranceHistogram.xml"));

if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the actual and diff images for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

The histogram file is just a XML file with the following schema:

 <histogram>
  <tolerance>
    <point x="0" y="0.87500" />
    <point x="1" y="0.09375" />
    <point x="2" y="0.03125" />
    <point x="3" y="0" />
    <point x="4" y="0" />
    ...
    <point x="254" y="0" />
    <point x="255" y="0" />
  </tolerance>
</histogram>

This visual verification approach was pioneered in the WPF test organization about 6 years ago by Marc Cauchy and Pierre-Jean Reissman.

SnapshotToleranceMapVerifier

However, none of the two approaches above work particularly well for evaluation of a typical application window, containing controls, text, etc. Such windows tend to have regions that need different tolerance settings. For example, consider the application window below:

 

 

image

Figure 2  Sample Application Window

If you try to perform master based visual verification, you will hit 2 issues:

  1. The non-client area of the window (the window frame) will tend to be slightly different between different runs of the application. It will also depend on environment factors such as the desktop wall-paper.

  2. Some regions of the client area of the window will also tend to exhibit significant variance between runs of the application.

Issue (1) is easy to resolve, using Snapshot.FromWindow(...) and excluding the non-client area from the capture. Issue (2), however, is a bit more involved. Here are the expected, actual and diff snapshots of the client-area of the application.

Master0-expected Figure 3a  Expected Client-Area Snapshot

 

Master0-actual

Figure 3b  Actual Client-Area Snapshot

 Master0-difference Figure 3c  Difference Snapshot

 

image Figure 3d  Difference Snapshot – Completely Black Pixels Are Replaced With Pink

 

The differences between the expected snapshot and the actual snapshot are difficult to see on Figure 3c, so on Figure 3d I have replaced purely black pixels with pink.

It is not surprising that most of the variation occurs around the text regions in the application window (ClearType renders differently on different machines).  So it may make sense to increase the tolerance (or completely mask away) those regions, providing of course we are not specifically interested in their rendering.

In order to achieve that, we use the SnapshotToleranceMapVerifier class. Here’s an example:

 // Take a snapshot, compare to the master image and generate a diff
Snapshot actual = Snapshot.FromWindow(hwndOfYourWindow, );
Snapshot expected = Snapshot.FromFile("Expected.png"));
Snapshot difference = actual.CompareTo(expected);

// Load the tolerance map. Then use it to verify the difference snapshot
Snapshot toleranceMap = Snapshot.FromFile("ExpectedImageToleranceMap.png");
SnapshotVerifier v = new SnapshotToleranceMapVerifier(toleranceMap);

if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the actual and diff images for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

The tolerance map that we use looks as follows:

image Figure 4  Tolerance Map in “ExpectedImageToleranceMap.png”

What appears pure black (0x00FFFFFF) is actually an off-black color (0x000A0A0A) to handle the small variations that appear as black dots on Figure 3d. Then we also have 4 regions with significantly higher tolerance to handle the variability of the text rendering.

 

In Conclusion

The visual verification API in TestApi provides a solid foundation for visual verification tests. As a general best practice, however, avoid visual verification as much as possible.