The test cases go marching two by two, hurrah, hurrah

One of the banes of my existence is test matrices.  Everywhere I look, it seems, another matrix is lurking, impatiently waiting to lob another set of mind-numbinginly boring test cases at me that are guaranteed to suck up two more weeks' worth of the extra time I don't have.  Matrices, matrices, matrices galore, all focused on me and mine.  No sooner do I vanquish one but two more pop their squirmy heads out of a trans-dimensional rift and splat themselves all over my office.

Since my current product requires Longhorn, I escape (this release, anyway) the dreaded Platform Matrix.  This is perhaps the most well-known of the species, dreaded and hated by testers everywhere.  Let me give you a typical example.  I'll include only Microsoft products just to keep things simple (ha ha): 

  • OS
    • Windows 2000 Gold (the original, un-SPd version)
    • Windows 2000 SP1
    • Windows 2000 SP2
    • Windows 2000 SP3
    • Windows 2000 SP4
    • Windows Server 2003
    • Windows XP Gold
    • Windows XP SP1
    • Windows XP SP1a
  • Browser
    • IE 5.01 Gold
    • IE 5.01 SP1
    • IE 5.01 SP2
    • IE 5.01 SP3
    • IE 5.01 SP4
    • IE 5.5 Gold
    • IE 5.5 SP1
    • IE 5.5 SP2
    • IE 6.0 Gold
    • IE 6.0 SP1
  • .Net Framework
    • 1.0 Gold
    • 1.0 SP1
    • 1.0 SP2
    • 1.1 Gold
  • Office Suite
    • Office 2000 Gold
    • Office 2000 SP1
    • Office 2000 SP2
    • Office 2000 SP3
    • Office 2000 Developer Gold
    • Office 2000 Developer SP1
    • Office 2000 Developer SP2
    • Office XP Gold
    • Office XP SP1
    • Office XP SP2
    • Office XP SP3
    • Office 2003 Gold
  • Your app here
    • Current version
    • Previous version
    • Version before that

To well and truly test this matrix, you have to test every single combination.  Let's see...9 OS x 10 Browser x 4 .Net Framework x 12 Office x 3 versions of your app...that's 12,960 different configurations you need to run all of your tests on!  If you can get through 36 configurations every day, that's still a year's worth of working weekends and holidays (and don't even think about vacation).

Of course, some of these combinations aren't valid -- you can't get IE5 on Windows XP, for example.  Even with these removed, however, there's still a huge number of configurations to cover.  It's so huge, in fact, that most teams (at Microsoft and elsewhere) don't even bother trying.  Instead they'll take a (vertical, horizontal, and/or diagonal) slice or two through the matrix, add in a few additional configurations that seem likely to be popular, and leave it at that.  This is pretty much voodoo testing, however, and your chances of missing a particularly gnarly bug because you chose to skip some particular configuration is pretty high.

Even if you happen to have the luxury of testing on just a single OS with a single web browser and a single version of Office and your app, you still aren't safe.  Test matrices are a resilient breed that can live off just about anything.  If your app supports richly formatted text, then you have a matrix of font, font size, the various font styles (e.g., bold, small caps), and so on.  If you have a method in your API with more than one parameter, you have a matrix of the various test values for each parameter.  Matrices are everywhere!

All hope is not lost, however.  Starting with Siddharta Dalal and company in 1997, people started to realize that many bugs are caused by either a) individual incorrect values, or b) particular pairs of values.  A Google search on "pairwise combinatorial test -gene" (the "-gene" weeds out a bunch of bio-results) will tell you more than you probably want to know about the gory details of how pairwise testing works.  The short version is that pairwise reins in test matrices like a champion calf roper.

Here's how it works:  since most bugs are a result of a particular pair of values, you don't need to cover the full test matrix but rather need only hit every individual pair.  If you have more than two axes in your matrix, you can cover multiple pairs in each test case.  As you can imagine, this makes for a rather large reduction in the number of test cases (the more axes you have the bigger the reduction you'll see).  Taking the above Platform Matrix as an example, pairwise whittles the 12,960 configurations a straight multiplicative combination gives you down to just 126 combinations.  Here's the first five my tool spat out:

  • Windows XP SP1a, IE 5.01 SP3, .Net Framework 1.0 Gold, Office 2000 SP3, Previous version of your app
  • Windows XP Gold, IE 5.5 SP1, .Net Framework 1.1 Gold, Office XP SP3, Current version of your app
  • Windows 2000 Gold, IE 5.5 Gold, .Net Framework 1.0 SP1, Office 2000 Gold, Version of your app before the previous one
  • Windows 2000 SP2, IE 6.0 Gold, .Net Framework 1.0 SP2, Office 2000 Gold, Previous version of your app
  • Windows 2000 SP1, IE 5.01 SP3, .Net Framework 1.0 SP2, Office 2000 Developer SP1, Current version of your app

Combining disallowing invalid combinations with a few judicious decisions not to support particular configurations reduces this number even further. 

It's very important to note that you *must* test every combination the pairwise tool spits out.  If you decide to skip a particular configuration for some reason (it contains an invalid combination, for example), you're skipping not only that invalid combination but several other pairs of values as well.  Live dangerously if you want, but you're much better off adding that combination to your matrix definition's "Don't do this" restrictions and generating a new set of combinations.

It's not too hard to write your own basic pairwise combinatorial tool, but to generate really efficient results requires some fancy math.  Luckily for those of us who aren't math geniuses, pairwise tools abound on the Web.  I found the following without trying very hard; I've only glanced at them and so can't vouch for how well they work, but they should get you started:

*** Comments, questions, feedback?   Do test matrices excite you?  Or do you just want a fun job on a great team? </g>  Send two coding samples and an explanation of why you chose them, and of course your resume, to me at michhu at microsoft dot com. I need testers, and my team needs a data binding developer, program managers, and a product manager. Great coding skills required for all positions.


Comments (11)

  1. sstearns says:

    You forgot to add Windows XP SP2 and Windows Server 2003 SP1 to the matrix. 🙂

  2. Bob Jenkins says:

    jenny and aetg support "Don’t do this" restrictions. Allpairs does not. I don’t know one way or the other for the java application or precalculated matrices you point at.

  3. When we testers find yet another &quot;Did you even run this?&quot; bug, it’s easy to believe developers purposely…

  4. One of the areas I’m focusing on these days is cross-feature testing – purposely taking a wide-and-shallow…

Skip to main content