Many experts have written articles and devoted chapters of books on the attributes of what constitutes a ‘good’ test case. Unfortunately, most books repeat a common (yet limited) perspective that a good test case has a high probability of finding bugs, and Kaner goes to the extreme by stating “A test that did not reveal a problem was a waste of time.” I pondered this for a while and thought, if I run a test to prove that a feature functions as expected is that really a waste of time? Isn’t it a good thing that testers actually spend some time executing tests that demonstrates the product does what the customer expects it to do? Or do we simply want to restrict the value of testing and reduce our testers to keyboard monkeys who bang away at the keyboard in search of bugs? (I believe limiting the perspective of the purpose of a test in testing literature has only perpetuated the faulty assumption that the purpose of testing is to simply find bugs; a point I previously dismissed). So, what is the purpose of a test?
Jorgensen states in his book, Software Testing: A Craftsman’s Approach that “a test case has two purposes: either to expose an error, or to demonstrate correct execution.” This explanation broadens the purpose of a test case to include both verifying the product meets the requirements, and revealing potential errors or defects. I tend to agree more with Jorgensen in this matter, but also think the purpose of a test case involves even more.
There are several objectives of any effective test whether manually executed or automated. The rationale for test case usually falls into one of seven categories. Each category provides value to the organization in different ways, but they all essentially function to reduce risk and qualify the testing effort. So, a good test provides some measurable value to the organization with the explicit purpose of:
- Verifying conformance to applicable standards and guidelines and customer requirements
- Validating expectations and customer needs
- Increasing control flow coverage
- Increasing logic flow coverage
- Increasing data flow coverage
- Simulating 'real' end user scenarios
- Exposing errors or defects