Oracle tradeoffs: musings of a test designer

There are many levels of sophistication when designing tests. A simple and straightforward test might have hard-coded logic to setup the test environment, make some change to the system, and verify that the software reacts correctly.

Data-driven tests will read all this information from somewhere and act on it. Suddenly, coverage can be increased simply by coming up with new cases that the test can handle, and lo and behold! we are actually reusing software.

We can introduce an oracle into the picture, that will tell us what the expected results should be. In this case, we're no longer reading this in from somewhere, but rather we're computing it. This enables testers to run thousands of tests without actually having to maintain thousands of data sets; the input conditions can be generated maybe through some combinatorial technique, then the output calculated, and voilà - the tester now only needs to concern himself with finding the right discrete values to feed into the system.

Well, almost. Writing oracles can be tricky. Fortunately, testers have a different set of requirements than product developers, and so we get to make different tradeoffs. A classic tradeoff is to be less performant but use simplified data structures or less elaborate algorithms. Clarity and debugability are super-important in testing, because we don't get thorough testing of our own code from a separate group of people. Another tradeoff is to have different pieces of code deal with only a subset of possible cases.

For an example of the latter case, I can have code to test what should happen when I click a button, but I'll make the assumption that the button is visible, enabled, and has non-zero width and height. Maybe I'll extend my oracle to cover these other cases next week. I can build such things in iterations, and I don't need to keep this encapsulated in a single place - it may be OK to have enabled-ness handled separately for all controls. In any case, if I can take some assumptions that make sense to me now, I can streamline my code.

One last thing for today: ieSpell rocks. Go get it for free, fall in love with it, and send a donation.