Smart Unit Tests – Test to Code Binding, Test Case Management

Pratap Lakshman

April 18th, 20150 0

[Editor’s note: “Smart Unit Tests” has been renamed to “IntelliTest” with effect from Visual Studio 2015 Release Candidate (RC).]

In an earlier post we had mentioned how Smart Unit Tests can emit a suite of tests for a given code-under-test, and how it can manage this test suite as the code-under-test itself evolves. For any given method serving as the code-under-test, the emitted test suite comprises of a “parameterized unit test” and one or more “generated unit tests”, and the following figure illustrates the *Pex** custom attributes used to identify the ***test-to-code binding***, that in turn enables such management. These attributes are defined within the Microsoft.Pex.Framework namespace.

At compile time, these attributes are baked into the assembly as metadata for the types and methods to which they were applied. On subsequent invocations, Smart Unit Tests can reflect on the assembly and, with the help of this metadata, re-discover the test-to-code binding.

As you would have noticed, the generated unit tests are just traditional unit tests; indeed, they will show up in the Visual Studio Test Explorer just like any other hand written test, although it is not expected that such generated unit tests will be edited by hand. Each generated unit test calls into the parameterized unit test which then calls into the code-under-test.

The programmatic separation between the generated unit tests and the parameterized unit test allows the parameterized unit test to serve as a single location where you can specify correctness properties about the code-under-test, that all of the generated unit tests can validate. In a future post we will discuss ways to express such correctness properties, but it is not a matter of concern for the test-to- code binding, the topic of this post. What does matter is that the generated unit tests and the parameterized unit test are placed into the same assembly.

The management of these generated unit tests involves the following:

Preventing the emission of duplicate tests: the code-under-test is always explored from scratch by the testing engine. Therefore it might generate tests that had already been generated before, and such tests should not be emitted.
Deleting tests that have become irrelevant: when the code-under-test changes, previously relevant tests might become irrelevant, and might need to be replaced with new tests.

Here is how it is done:

Given that the parameterized unit test and the generated unit tests are placed into the same assembly, the testing engine pre-processes any existing “generated unit tests” (emitted previously) by scanning the assembly for the test fixture (identified by the type having the PexClass annotation). Generated unit tests are recognized by the Test attribute (the TestMethod, and PexGeneratedBy annotations in this case). For each such unit test method, it fetches the source code, trims away all whitespaces and uses the resulting string as a “hash” of the generated test. Once this is done, the testing engine has a dictionary of such hashes that it can consult to determine whether a newly generated test already exists. Duplicate tests are not emitted, and if a previously existing test case is not generated, it is deleted.

Keeping a suite of unit tests in sync with a fast-evolving code base is a challenge. Manually adapting the code of a potentially large number of unit tests incurs significant costs. We hope such automatic management of the suite of generated unit tests helps in addressing this challenge. As ever, report any issues or overall feedback below, or through the Send a Smile feature in Visual Studio.