- Overview of TestApi
- Part 1: Input Injection APIs
- Part 2: Command-Line Parsing APIs
- Part 3: Visual Verification APIs
- Part 4: Combinatorial Variation Generation APIs
- Part 5: Managed Code Fault Injection APIs
- Part 6: Text String Generation APIs
- Part 7: Memory Leak Detection APIs
- Part 8: Object Comparison APIs
The state and behavior of every software system depends on a number of parameters. These parameters can be both inputs as well as environmental factors.
One can think of software testing as a controlled experiment, where the system under test is observed upon varying of the parameters that affect it, in an attempt to discover unexpected system behavior.
Depending on who you talk to, the act of generation of these variations is called variation generation, matrix expansion, design of experiments (DOE), etc.
Modern-day software systems are complex and depend on many parameters. Expanding all possible combinations of all parameter values often results in a phenomenon called matrix explosion – having an overwhelmingly high number of test variations.
Matrix explosion is undesirable because:
- It increases the runtime of your tests (which, among other things, makes TDD impractical and TDD is the single most effective way to ensure high product quality);
- It increases the support costs of your tests – for a test pool of 100,000 tests, even with 99% pass rate (which in reality is hard to achieve), you still have to investigate 1,000 test failures. Investigating 1,000 test failures is always more expensive than investigating say 10 failures;
- It is often impractical or impossible to cover all combinations, so you end up with partial test coverage (very often “vanilla” test coverage).
There are many ways to deal with matrix explosion, which depend on various system and/or experimental constraints. The theory and practice of DOE deals with that.
TestApi provides a generic API for combinatorial variation generation, using the algorithm presented in Jacek Czerwonka’s “Pairwise Testing in Real World” article. The API uses the following nomenclature:
- Parameter – represents a single factor / variable and its values;
- Constraint – represents a relationship between parameters, their values, constants, and other constraints;
- Model – contains all parameters and constraints for the system, for which we are generating variations;
- Variation – represents a tuple with a single value for every Parameter in the Model.
All of these are represented as correspondingly named types. Following are several examples demonstrating the use of the API.
Example 1 : Simple Matrix
For the purposes of a simple artificial example, consider having a system with the following parameters and values:
|Color||White, Green, Red|
|Size||Small, Medium, Large|
Assuming, there are no constraints, here is the code you would use to create a model:
Executing the code below produces the following 3*2*3=18 variations, representing all possible combinations of the values of the three parameters:
The variations are generated by the call to GenerateVariations(3, 1234). The number "1234" is the seed for the random generator, utilized by the algorithm. The number "3" is the order of the generated combinations. The order of generated combinations must be a number between 1 and the number of parameters in the model. The output for orders "1" and "2" are presented below:
Example 2 : Vacation Planner
This example (created by Nathan Anderson – our engineer who designed and implemented the combinatorial variation generation API) demonstrates the use of parameter constraints.
The execution of this code would result in output similar to the following:
Example 3 : The WPF Platform Matrix
This last example demonstrates how we deal with a real-world problem we face in the WPF team...
WPF must work reliably on all OS configurations, defined by the following matrix:
|Parameter||Number of Values||Values|
|OS||6||XP SP2, Vista SP1, 7, Server 2003 SP2, Server 2008 SP1, Server 2008 R2|
|Language||25||(using 3-letter language abbreviations)
ARA, CHS, CHT, CSY, DAN, DEU, ELL, ENG, ESN, FIN, FRA, HEB, HUN, ITA, JPN, KOR, NLD, NOR, PLK, PSE, PTB, PTG, RUS, SVE, TRK
|System Locale||2||Same as OS language, TRK|
|Platform||3||x86, x64, x64 wow|
|IE version||3||OS default, IE7, IE8|
|High DPI||2||120 DPI, 96 DPI|
|Theme||6||Native, Classic, Luna, Royale, Classic High Contrast, Aero|
|Side-by-side||9||3.5 SP1 + 4 (3.5 tests), 3.5 SP1 + 4 (4 tests), 3.5 SP1 + 4 - 4 (3.5 tests), 4 + Mock 4.5 (4 tests), 4 + Mock 5 (4 tests), 4 + 3.5 SP1 (4 tests), 4 + 3.5 SP1 (3.5 tests), 4 + 3.5 SP1 - 4 (3.5 tests), 4 (4 tests)|
Most of the parameters are self-descriptive. “Side-by-side” captures the .NET installation state. For example “3.5 SP1 + 4 - 4 (3.5 tests)” means “install .NET 3.5 SP1, install .NET 4, uninstall .NET 4, run tests built against 3.5 SP1”. This is done to confirm that there are no unexpected side effects as a result of the installation and un-installation of .NET 4.
The trivial full expansion of the matrix results in 583,200 combinations (=6*25*2*2...). Of course, some of these combinations (e.g. XP SP2 OS with a Aero theme) are not valid, but even after removing the invalid combinations, we still end up with a prohibitively large number of platform configurations to test on.
There are several ways to deal with this problem. One is identifying the so called equivalence classes. For example, from the point of view of Side-by-Side, Vista SP1 and Server 2008 SP1 can be regarded as equivalent OS-es and so on. Another popular approach is reducing regular testing to “vanilla configurations” (e.g. mostly ENG (English), 96-DPI configurations), venturing outside of the “vanilla domain” in accordance with a predefined schedule (e.g. during test passes at the end of major milestones). A third approach is using a pair-wise combinatorial variation generator, reducing the number of platform variations to about 230 – still a fairly high number for any real-world test pool, but clearly much better than the original number above.
In the WPF team, we use the third approach, combined with an adaptive random algorithm, which prioritizes testing on platform configurations that have not been tested on recently. The simplified code below demonstrates how to construct a model for platform config variation generation.
Pairwise variation generation is an important tool in your toolbox as a test author. TestApi provides a simple facility for combinatorial variation generation. We will of course be evolving this facility, but do let us know if you have specific scenarios or requirements you'd like to see supported.