Did You? Did You Really? Loosely Coupled Comprehensive Verification

Verifying that a test case’s actions had the expected result is perhaps the most important part of testing. Every test case does something at least a little differently than every other test case, so the expected results are often a little different. These minute differences make it difficult to factor verification out to shared code and so verification code tends to be embedded in and duplicated across each test case.

Intermixing test case execution code with test case verification code further complicates matters. Initial state data necessarily must be gathered before individual operations are executed. Expected state can be calculated anytime after initial state is recorded to just before actual state is verified. Verification that actual state matches expected state must of course be done sometime after each operation is executed; often immediately after, if subsequent steps in the test case will destroy the current actual state. All of this makes it difficult to differentiate between execution code and verification code.

Separately, the set of properties that are typically verified is nowhere near the complete set that would be necessary for truly comprehensive verification (that is, verifying every property after every operation). The copious amount of work required to do so is generally deemed not worth the trouble. This is especially true since for any particular operation most properties will be unchanged. Experienced testers, though, will recognize that this is exactly how the most insidious bugs are manifest by changes in something that should be completely unaffected by the operation.

We have bypassed these problems by decoupling verification from execution. Loosely Coupled Comprehensive Verification is easy to explain and almost as easy to implement. Just before a test case or LFM method executes an operation, it notifies the Verification Manager that it is about to do so and also provides any relevant details. The test case or LFM method next executes the operation, and then finally it notifies the Verification Manager that it has completed the operation. That’s it as far as the test case or LFM method is concerned!

When Verification Manager is notified that something is about to happen it baselines current state and then works with a set of Expected State Generators to determine the expected state. Upon notification that the operation has completed Verification Manager compares actual state against expected state and logs any differences as failures.

This very loose coupling between verification and the rest of the system makes it very flexible. If the details regarding how a particular expected state is calculated change, the corresponding Expected State Generator is the only entity that has to change. Similarly, if the set of properties being verified changes nothing outside the verification subsystem needs to be modified.

Another benefit we get from this scheme is a dramatic reduction in follow-on failures – failures that occur solely because some previous action failed. Because we baseline expected state before every action it is always relative to the current state of the application, so a previous failure that has no effect on an action won’t fail that action just because the verification code expected that previous action to succeed. This eliminates “noise” failures and allows us to concentrate on the real problem.

Because verification details are decoupled from execution, the set of properties being verified can start small and expand over time. Helping this to happen is the ability to say “I don’t care” what happens to a particular property as a result of a particular operation. Any property with such a value is ignored when actual state is compared to expected state after the operation has completed. Once the expected result is known the tester simply updates the Expected State Generator appropriately and suddenly every test case automatically expects the new behavior.