One of the areas I’m focusing on these days is cross-feature testing – purposely taking a wide-and-shallow view of Sparkle and using its features in combination. Cross-feature interaction is easy to forget as we beat the life out of each feature on its own, but any experienced tester knows that that interaction is where the gnarly and deadly bugs hang out.
But it can be hard to know where to start. I’m using a giant pairwise test to guide my beatings. Pairwise testing is a great technique for covering a large test matrix with a smallish number of tests. The bigger the matrix is, though, the harder it is to automate. Each additional axis, and each additional value on an axis, adds a disproportionate number of boundary conditions. If you aren’t careful you’ll find yourself going down an infinite number of rat holes trying to automate your beast of a pairwise.
My cross-feature matrix definitely is one of these possibly-too-big-to-be-worth-automating tests. And in fact I’m not trying to automate it but rather am executing it manually. This is turning out to be very worthwhile as every row in the matrix has found multiple bugs. Several of these bugs would make automating difficult if not impossible, so I am actually finding more bugs doing this testing manually than I would were I attempting to automate it.
I am noting as I work through these tests that I never do the same action the same way. Each time is a little bit different, and so each time I find different bugs. This is a lot harder to achieve with automated tests (although our automation stack makes doing so relatively simple). In fact, it strikes me that my cross-feature pairwise matrix is a “Nouns and Verbs“-like catalyst for finding bugs!
So the next time you’re stumped as what to test next, try generating and running a mondo cross-feature pairwise test. I bet your block will disappear like ice on a hot summer day!
*** Want a fun job on a great team? I need a tester! Interested? Let’s talk: Michael dot J dot Hunter at microsoft dot com. Great coding skills required.