What’s wrong with this test case code? – Identifying False Positives


This is my first attempt at a series of “what’s wrong with this test case code” entries. Let me know what you think of this idea and if you’re interested in seeing more examples.

A False Positive is defined as a test case that has been logging a passing result; however, the test case logic is incorrect, masking a product bug. In other words, if the test case writer were to run the test case manually, he or she would see the product bug.

Can you identify why the test case below could contain a false positive? Remember, you’re looking for the logic error.

You can make the following assumptions (for the sake of this example)

  1. The current line in the text file contains at least one whitespace
  2. Each individual method is bug free and does what the comment above says

‘ put the contents of the current line into a string var

Dim currentLine As String = vsEditor.Cursor.GetCurrentLineText()

‘ find out where in the line the whitespace is

Dim whitespaceLocation As Integer = currentLine.IndexOf(" ")

‘ move to the left until we’re at the whitespace location

vsEditor.Cursor.Move(CursorUnits.Character, whitespaceLocation)

‘ press delete

vsEditor.SendKeys("{DELETE}")

‘ get the new contents of the line

currentLine = vsEditor.Cursor.GetCurrentLineText()

‘ if the whitespace is gone, the test case passes

‘ otherwise, the test case fails

If currentLine.Substring(whitespaceLocation, 1) <> " " Then

LogPass()

Else

LogFailure("Failed to delete whitespace…")

End If

Got it figured out? If not, here’s a hint…

One common mistake when writing test cases is to verify against the side effects of what we’re expecting to happen. In the above code, the side effect is that the whitespace is gone from that location. In order to correctly verify that some desired action has occurred properly, the test case needs to simulate what the desired action is really doing.

Ready for the answer?

We’re testing that pressing the delete key will remove the whitespace, a single character, in front of the cursor. But let’s say delete is broken, and it deletes the whitespace and the next character. If our line of text was, “A lazy brown dog” and pressing delete deleted “ l”, the string would become “Aazy brown dog”. The second character isn’t a whitespace, yet it would pass our test case logic.

A better test case would simulate what the delete key is doing, for example

‘———————————————————————

InitScenario("Verify user can delete whitespace in middle of line")

‘———————————————————————

‘ put the contents of the current line into a string var

Dim currentLine As String = vsEditor.Cursor.GetCurrentLineText()

‘ find out where in the line the whitespace is

Dim whitespaceLocation As Integer = currentLine.IndexOf(" ")

‘ move to the left until we’re at the whitespace location

vsEditor.Cursor.Move(CursorUnits.Character, whitespaceLocation)

‘ press delete

vsEditor.SendKeys("{DELETE}")

‘ get the new contents of the line

Dim currentLineAfterDelete As String = vsEditor.Cursor.GetCurrentLineText()

‘ Remove the whitespace from the string variable

‘ and verify this is what the current line really is

If currentLineAfterDelete = currentLine.Remove(whitespaceLocation, 1) Then

LogPass()

Else

LogFailure("Failed to delete whitespace…")

End If

Comments (11)

  1. Uwe Keim says:

    I know what’s wrong: The language is Visual Basic… 😉

  2. Frederik says:

    The first test would return a false negative as well, if the line would countain a double space…

  3. I really like the idea. Thank you for share your experience!

  4. G. Man says:

    I think you shouldn’t give us the answer right away! Do that in a follow-up post

  5. There are actually multiple whitespace characters in the Unicode character set (http://www.digitalmars.com/d/archives/digitalmars/D/4742.html).

    Now, for source code, it’s safe to assume that a space (ASCII 32) will be the only whitespace that matters, but I still would have documented my assumption in this case.

  6. josh ledgard says:

    Yeah, don’t give the answer upfront in the future.

  7. sara ford says:

    And I thought people would be able to exhibit a little self-control. Apparently not =)

    Okay, i broke up tonight’s example into two separate blog entries. Let’s see how that works out.

  8. Andrew says:

    Your point about relying on side effects is a very good one, but we don’t always have a reference solution to compare the code under test against. In this example an independent implementation of the delete key functionality is readily available in currentLine.Remove(). This "reference" implementation often doesn’t exist for many of the complex test scenarios I work on, at which point it becomes a trade off between creating it or looking for the side effects and accepting that some bugs will go undetected. Do you ever run into this issue, and if so how do you deal with it?

    Andrew.

  9. sara ford says:

    Andrew, great question! I’ll respond to it in my next blog posting called "What are Test Hooks and why are they needed?" Hopefully, test hooks will be able to help you out.