When should (automated) testing be random?

When should (automated) testing be random?

I recently posted a blog post about unit tests. In it I used some random number generation to do some testing. Some people pointed out that randomness might be bad in some circumstances. So when should you use randomness in tests and when shouldn’t you?

Those in favor

The benefits of randomness are that it allows us to cover ground we wouldn’t have otherwise. A good example is a Unicode string. There are about 100,000 characters in the Unicode standard. That breaks down into many equivalence cases. Now at some point we want to figure those cases out and cover them all. Here is the problem, you don’t want to spend weeks on a Unicode generator while early in the cycle. You could write hundreds of simpler test cases in that time. Randomness comes to the rescue. You can create a Unicode randomizer cheaply and start testing with it. Now you have a chance to uncover bugs in Unicode handling, but you don’t need to spend a lot of time to do it.

Another big advantage of randomness is speed. You can come up with one random value and run a test quickly. If test space were something like a ten character Unicode string (10.0e+49 possible strings) you wouldn’t want to hit every combination. But you can cover a lot of interesting cases just by getting a different random one every day.

Those against

The drawbacks of randomness are many. You don’t want your BVT’s failing randomly for no discernable reason. So for a BVT a random Unicode string might not be a good answer. For something simple enough randomness makes sense (like a byte) you might as well just cover all the cases.

Reproducing problems with random data can be a nightmare. Especially if the product has a state machine and drove it into a corner in a non-reproducible way. Randomness is really ugly if you don’t have a good mechanism for reproducing bugs it uncovers.

Logging is key

If you do add some randomness to your tests, be sure to at least log all the values. You might go so far as to record them in a replayable format.

When do I act random?

Use randomness in your less formal tests. It’s great for uncovering wacky bugs early in the process. Don’t use it as a replacement for more thoughtful tests down the road. I would be happy to pass a test suite that uses random numbers to the developer for pre-check in testing. As long as they realize the results might be random.

Steer clear of randomness in any circumstance where random failures would make the product or the tests look less healthy than they are.

Finally, if you can make it through every combination of equivalence cases in a reasonable amount of time, do that instead.