Testing Problems (with pairs)

I'm teaching in Vancouver, BC this week (incidentally, I think this is the first time I've taught a testing course since February or so). As usual, all of the students are smarter than me, so I try to make up for it by telling stories about testing and otherwise attempting to keep the group entertained.

When I talk about testing techniques, I don't live the safe life of well thought out examples - instead, I just grab examples out of thin-air, cross my fingers, and hope they work. I failed, for example, to do a very good job describing state modeling with a white board marker (it worked well for a while, then...), but, it turns out that coffee works well for demonstrating pair-wise testing. Pair-wise testing (or all pairs testing) ensures that every pair of combinations is tested at least once together. Studies show that most errors are caused by the interaction of two variables, so this is a technique to reduce test cases while keeping a reasonable level of confidence.

I'm an Americano drinker, but I'm always amazed at the variety of drinks available at my local coffee shop. Lattes, cappuccinos, flavors, and varieties of milks can make a potentially huge number of different drinks. Just so I can fit my example into a single blog post, let's imagine I'm at a "limited" coffee shop run by my friend Frank. Frank only make lattes, and he only has 3 flavors. Here is what Frank has available:

Sizes: Small, Medium, Large, Oh Boy!
Milk: Non Fat, 2%, Whole, Soy, None
Flavors: Chocolate, Vanilla, Caramel, None

That's nothing compared to what I see the guy on the corner selling out of the back of his minivan, but it's still (4x5x4)==80 different combinations of latte I can order. There are 20 options for Small drinks alone!

I want to have a pretty good idea if Frank is the real deal or not, but I'm not ready to drink all 80 sizes and flavors currently available to see how he measures up. Since I know that errors usually occur in pairs of combinations, I can test every pair - e.g. make sure that I test small at least once with non-fat, whole at least once with vanilla, soy at least once with large, and so on. I can test every pair of combination with only 21 different varieties of coffee - some testing that I could definitely take care of within a week!

The better news is, that next week when Frank adds Cherry, Mint and Orange flavors to his store, I can test all pairs with only 14 more tests (35 total) - much better than the 140 that would be needed to test every possible combination.

Besides, how much Orange coffee could someone drink?

I used the PICT tool to determine my the set of pairs. You can get it from the downloads page on testercenter. More information on pairwise testing, including case studies and research can be found at https://www.pairwise.org.