The execution section of a test case is itself composed of three subsections: setup, execution, and teardown. Execution is often thought to be the most important part of a test case (after all, a test case that doesn’t verify anything is rather pointless). However, setup and teardown are in fact at least as important.
The most obvious reason this is true is that execution would be impossible if the required starting state was not set up. The more important reason, albeit one rarely recognized, is that setup and teardown often form the majority of the test case and thus effectively test a larger percentage of the application than the execution steps do.
Much emphasis is placed on ensuring every possible execution path is tested in some test case somewhere, but seldom is any emphasis placed on ensuring that setup and teardown execute multiple paths as well. As a result, setup and teardown is generally executed via whatever means is easiest and thus every setup and teardown tends to be implemented using the exact same execution paths as every other setup and teardown. This trains the product to work correctly in those specific circumstances but does nothing to ensure it works correctly in any other circumstance.
This holds true for data values just as much as it does execution paths. While explicit tests are usually written to test program behavior when it is presented with boundary values and edge cases, setup and teardown code does not explicitly take part in this. One tester may accidentally use different values than another tester when implementing a test case’s setup or teardown code, or a test may make an effort to use a range of values, but this is unlikely to provide true variability.
Data-driven tests might seem to ameliorate this somewhat, but in fact they do not. Even more than other types of test cases, data-driven test cases assume details regarding the context in which they are executing. Foremost among these assumptions is the source of their data. These assumptions make reusing test cases and their constituent parts within other test cases problematic or even impossible. Thus while a tester might like to reuse parts of or even entire test cases in the setup or teardown of a test case, doing so is generally not possible.
We have developed a method that ensures execution during setup, teardown, or any other part of the test case uses all possible execution paths and data values without any extra effort (e.g., writing additional test cases) on the part of the test case author.