explaining exploratory testing

I just got finished talking (actually the conversation was more like a debate) to a colleague, exploratory testing critic and a charter member of the plan-first-or-don’t-bother-testing-at-all society.

I am happy to say, he conceded the usefulness (he would not grant superiority, if he had I would be on my way to the pub right now with his credit card) of exploratory testing. Perhaps I have finally found a way to explain the utility of exploration. Here’s what I said:

“Software testing is complicated by an overload of variation possibilities from inputs and code paths to state, stored data and the operational environment. Indeed, whether one chooses to address this variation in advance of any testing by writing test plans or by an exploratory approach that allows planning and testing to be interleaved, it is an impossible task. No matter how you ultimately do testing, it’s simply too complex to do it completely.

However, exploratory techniques have the key advantage that they encourage a tester to plan as they test and to use information gathered during testing to affect the actual way testing is performed. This is a key advantage over plan-first methods. Imagine trying to predict the winner of the Super Bowl or Premier League before the season begins … this is difficult to do before you see how the teams are playing, how they are handling the competition and whether key players can avoid injury. The information that comes in as the season unfolds holds the key to predicting the outcome with any amount of accuracy. The same is true of software testing and exploratory testing embraces this by attempting to plan, test and re-plan in small ongoing increments guided by full knowledge of all past and current information about how the software is performing and the clues it yields in the testing results.

Testing is complex, but effective use of exploratory techniques can help tame that complexity and contribute to the production of high quality software.”

Comments (11)

  1. mark.spiteri says:

    That is a great explanation. I have been having this debate continuously for the past couple of weeks and have not yet managed to get anywhere.

    I will try using your same line of thought, and see if it works 😉


  2. anutthara says:

    I don’t think I understand why there is a contest between structural and exploratory testing methods at all. You need a balance of both to test the product to a meaningful level.

    I know that traditionally at Microsoft, we begin with a structured test plan, but rely a whole lot on exploratory testing to find bugs as the product cycle progresses. I don’t think I would choose one over the other, but quite simply a mix of the 2. Wouldn’t you agree that’s common sense?

  3. manndras says:

    I guess the biggest question for me is what would lead a tester to taking an exploratory approach vs. say, an planned API level approach? Obviously, they’re not mutually exclusive, nor can the tester not use them both to feed each other.

    It really comes down to a practical problem of time and investment. The more I read your blog the more I realize that there are so many "schools" of testing that you will find many testers are drawn to one (or at best two). The best testers can employ any technique necessary to get the job done. Approproateness of methodology is what (in my my view) makes the best testers – and that requires an agile and broad set of perspectives and varied skills.

    For exploratory testing, what I struggle with is that it could be applied to any kind of testing task. Should it be? Or are there times when it’s too costly and a more statistical approach (if possible) is the better bet.

  4. Toddsa says:

    A good STE can put even a senior SDET to shame on bug counts most of the time. You lose easy reproducibly with a manual approach but you find bugs sooner in the product and you find more interesting bugs often.

    It is hard to find good testers and it is even harder to find good SDET’s. Companies don’t hire STE’s and train them, they mostly hire SDET’s out of college.

    In most cases I would rather higher a high school kid with some programming experience instead of college graduate for a test job.

  5. MSDNArchive says:

    It’s not a contest. It’s a debate. A good and healthy debate. And if this debate ends with a better understand of when planning is superior and when doing is superior and how to find the optimal mix, then everyone wins.

    But I can’t help my competitive nature … I am an explorer and I will use every chance I have to promote it.

  6. phuuo says:

    I agree strongly with anutthara. Proponents of exploratory testing will always cite something along the lines of the argument you have given, and then use this as evidence of the superiority of the technique. But the reality is that ET is just one of many approaches that should be employed during testing, depending on the sdlc and restraints in the scenario.

    Too many times have I seen sloppy testing being defended as being "exploratory".

    I think it always makes sense to produce planning before hand when the necessary information is available – e.g. functional specs, prototypes etc. Exploratory approaches can then be applied when the product goes into test in order to build on this initial planning.

  7. Guy kolbis says:

    James Whittaker say that testing is Hard and I quote: “Software testing is complicated by an overload

  8. calkelpdiver says:

    I have to agree with the people who say you need a mix of both.  Both have their place and pros/cons in the project.  For me I like to plan ahead and have some direction to go on.  BUT, I will take the time to go and explore when the opportunity arises.  To me this is a process of continous refinement of what I’m trying to test.  

    Also, because of doing the structured approach I get ideas of where and when to go "off the beaten path".  And then again this will be folded back into my structured methodology to re-use later on.

    It comes down to proving something is working the way intended (and if not then why), and then trying to see if does funny things when not used the way intended.  Having a hard sciences background (BS in Zoology) my thought process is to use Scientific Methodology first and then experimental methods next.  We need to do both the defined structural tests and the "What If" tests to determine if the SUT is really going to be sound.

    Finally, I think this all speaks to the context of the situation at hand.  Which method will work best for you at that time.

  9. [Nacsa Sándor, 2009. január 13. – február 3.]  A minőségbiztosítás kérdésköre szinte alig ismert

  10. [ Nacsa Sándor , 2009. február 6.] Ez a Team System változat a webalkalmazások és –szolgáltatások teszteléséhez