Most Of Each Test Case Exercises A Small Fraction Of The Code


The execution section of a test case is itself composed of three subsections: setup, execution, and teardown. Execution is often thought to be the most important part of a test case (after all, a test case that doesn’t verify anything is rather pointless). However, setup and teardown are in fact at least as important.

The most obvious reason this is true is that execution would be impossible if the required starting state was not set up. The more important reason, albeit one rarely recognized, is that setup and teardown often form the majority of the test case and thus effectively test a larger percentage of the application than the execution steps do.

Much emphasis is placed on ensuring every possible execution path is tested in some test case somewhere, but seldom is any emphasis placed on ensuring that setup and teardown execute multiple paths as well. As a result, setup and teardown is generally executed via whatever means is easiest and thus every setup and teardown tends to be implemented using the exact same execution paths as every other setup and teardown. This trains the product to work correctly in those specific circumstances but does nothing to ensure it works correctly in any other circumstance.

This holds true for data values just as much as it does execution paths. While explicit tests are usually written to test program behavior when it is presented with boundary values and edge cases, setup and teardown code does not explicitly take part in this. One tester may accidentally use different values than another tester when implementing a test case’s setup or teardown code, or a test may make an effort to use a range of values, but this is unlikely to provide true variability.

Data-driven tests might seem to ameliorate this somewhat, but in fact they do not. Even more than other types of test cases, data-driven test cases assume details regarding the context in which they are executing. Foremost among these assumptions is the source of their data. These assumptions make reusing test cases and their constituent parts within other test cases problematic or even impossible. Thus while a tester might like to reuse parts of or even entire test cases in the setup or teardown of a test case, doing so is generally not possible.

We have developed  a method that ensures execution during setup, teardown, or any other part of the test case uses all possible execution paths and data values without any extra effort (e.g., writing additional test cases) on the part of the test case author.

Comments (5)

  1. Pardon my skepticism, but I’d be surprised if you have a method that uses /all/ possible execution paths and data values–it ain’t possible.

    In a previous posting, you did (IMO) a very good job of identifying something that most people ignore, which is that there are plenty of ways to open a file–via the File / Open menu with keyboard, File / Open with mouse, clicking on the file in Explorer, using the "start foo.doc" syntax from the command line, and so on and so forth. Far more than most testers will identify unless they’re pushed, and unless they brainstorm for a while.

    Your claim would be entirely credible if you said "a good variety" or "a large number" or "reasonably good coverage", but "all" isn’t possible. For example, using Word:

    Type Alt-F. Click on Open. Tab into the file list. Select the filename such that it appears in the filename combo box. Backspace over the extension. Change your mind, and type the extension again.

    Click on File. Click on Open. Click on the filename. Backspace over the extension. Change your mind, and type the extension again.

    Click on File. Type O for open. Tab down to the filename combo box. Hit the down-arrow key. Select a file that looks interesting. Hit the right-arrow key. Hit backspace. Hit the down arrow key.

    Take all those cases, and vary ad infinitum. Instead of tabbing to the filename combo box, tab past it and keep tabbing until you get there again. Now shift-tab until you’re back to square one. Now click. Et cetera.

    Now: it may seem unreasonable that I would quibble over the use of the word "all". However, I contend that bugs are unreasonable, and will appear in lots of places that reasonable approaches won’t find. Naturally, risk should drive the decision as to how recherche we want to get, but there will sometimes be bugs beyond that frontier.

    The funny thing is that, in writing this up and reproducing the steps above, I’ve managed to find a bug in Word. It may be related to the fact that earlier, when I opened Word, I double-clicked on a filename twice (that’s four clicks), really fast, such that I opened the file for a second time before the first instance was open. This led to complaints about Normal.dot being corrupted or some such. In any case, I’ve just now managed to get Word (2000) into a state where it’s open, sort-of, but flashing on the task bar and won’t stop until (I presume) I kill the process. That’s a hang–a high-severity, pain in the ass bug.

    My point is that a claim of "lots" is defensible; that "what we consider to be excellent coverage" is defensible; that "we’ve addressed what we consider to be the most important risks" is highly defensible; and even "in our test suite, we cover a lot of varying ways in a user could reasonably accomplish the same task, unlike just about every other test automation implementation on the planet." Those are great things, but they’re not "all".

    —Michael "So you think YOU’RE a pedant?!" B.

    DevelopSense: Software Testing in Plain English

    http://www.developsense.com

  2. PeteVanC says:

    I don’t believe you 🙂 In general using all possible execution paths and data values is not possible (in a finite amount of time).

    For example you describe the following test case in a previous blog entry:

    Typing Alt to invoke the main menu, repeatedly pressing the left arrow key until the File menu is selected, repeatedly pressing the down arrow key until the New submenu item is selected, pressing the left arrow key a single time to expand the New submenu, repeatedly pressing the down arrow key until the New Document menu item is selected, then pressing Enter to invoke the New Document menu item.

    What if I press the left arrow key followed by the right arrow key, followed by the left arrow key, nearly forever before finishing the setup?

    In that same blog entry you describe 8 different ways to perform the test’s setup. What about mixing and matching snippets from the various different ways and undoing and redoing those snippets in different combinations nearly forever before finishing the setup?

    How do you ensure that hardware failures or other non-software failures are properly executed?

    That said, let’s say that you can define a finite number of execution paths and data values and execute them without writing additional test cases — that would be cool!

    I’m assuming here that someone needs to at least define the set of execution paths and data values — is that correct?

    And then someone also needs to verify that the execution paths and data values correctly perform the setup, teardown, or other part of the test case — but I guess these might not be defined as additional test cases.

    When the test is run, are all the possible exection paths and data values excersized at once, or only across a set of similar test cases?

    In any case, how do you ensure that they all get performed within a reasonable amount of time? I imagine that any sufficiently robust combination would quickly become quite large.

    Anyway, it was cool to see your picture in the profiles — although I had imagined you differently.

    Lastly, a caveat — I haven’t read all of your previous posts on this subject so it’s possible that I’m not fully grasping the context of your statements.

  3. You’re both correct – "all" is a (rather big? <g/>) bit of an overstatement, especially when it comes to data values. The great phrases Michael suggests are a much better description of what I was trying to say.

    Pete: "…let’s say that you can define a finite number of execution paths and data values and execute them without writing additional test cases — that would be cool!" I’m saying you can do exactly that; more details coming soon to a blog near you! This and your other questions should be answered by the end of this series. If they aren’t, though, let me know and I’ll expand further.

  4. In many of my posts I have alluded to the automation stack my team is building, but I have not provided…

  5. In many of my posts I have alluded to the automation stack my team is building, but I have not provided…

Skip to main content