Software Testing 6: Good Tests for Bad Parameters


Getting a bit more specific I figured I’d post something
that my team created as a helpful resource for people building test plans.  
This document started as a one page guide, and turned into a five page list of interesting
tests as it passed from tester to tester in the developer division.  Please don’t
consider this a comprehensive list and assume we covered everything.  It was
and is only meant as a starting point to help get the creative juices flowing around
building your test plan.  The more people that see it the more good examples
we’ve added so feel free to contribute to any gaping holes in this guide!

Thanks,

      Josh

The guide is broken out by the different resource types
that a program might depend on.

File Tests: Does your code rely on files?

  • Remove (rename) a file the application
    is dependant on (For example in Visual Studio the ToolBox depends on a .tbd file that
    is stored in the users/appdata dir. What would happen if we deleted that file?)
  • Corrupt files – what happens if the
    file doesn’t have the expected structure or contents? For example, VS Start Page expects
    that a custom tab xml file will comply with the schema and will be correctly formatted
    xml, what happens if it doesn’t or instead of xml we have a simple text file or a
    binary file?
  • Bad Data – what if the file is in
    the expected format, but the values of the data that we get from the file are invalid.
    Wrong data type, out of value bounds, containing invalid chars, etc… see string, numeric,
    and file path tests below.
  • Does your feature expect
    that a file would be ASCII? What happens if instead it had a different encoding say
    Unicode
    or UTF8?
  • Does the application expect that the
    file it reads from will not exceed a certain size? What happens when we try working
    with big files? While you are doing this, what if the file is 0 length?
  • Try using a file that is used by or
    in use by another process / user
  • What happens if the application depends
    on a file but the permissions set on the file don’t allow it to access the file? Try
    security vs. read/write/execute permissions. What if the file is just hidden?
  • Fill the disc (use floppy for example)
    and do reads/writes open/close app. You can use tools to emulate disc full conditions. Canned
    Heat
    is a good tool for that. Use FileMon to
    identify file system access.
  • Errors when accessing media – these
    can happen if the hard drive / floppy / cd-rom, etc… are not available or slow. Use Canned
    Heat
    to emulate the errors / delays. What happens if you can access the media
    at first but it becomes unavailable while the application is doing work? Just pop
    out the disc you are writing to or disconnect from the share you were using and see
    what happens.
  •  Windows
    2000 and above allow junction points. It’s basically a link from one point on disc
    to another. If your features do any type of recursive dir walking you should try creating
    a loop and see how your feature handles it. For example, say you map c:\myproject\linkedfolder
    to c:\myproject and you have a file foo.txt in c:\myproject that contains the word
    foo in it. Now, if you do “Find in Files” from Visual Studio starting from c:\myproject
    and searching for “foo” you’ll find 14 matches instead of 1… You can use linkd.exe to
    create junction points.
File Paths: Are file paths ever given as an input to your program?
  • Invalid file name (invalid characters etc). Make sure your feature
    uses the OS to handle invalid file names, rather then writing its own interpretation
    of that code.
  • File name/path longer than the allowed max path. Be aware of anytime
    your feature adds something to the file name specified by the user. What if the user
    file path was already Max length and we still added .aspx extension to the end of
    the path? BO and crash most probably…
  • Valid Longest File Name
  • File name with spaces, dots, semicolons
  • Using reserved names such as COM1, AUX to test any functions that create
    or open files.
  • The list of reserved names: CON, PRN, AUX, CLOCK$, NUL, COM1-COM9,
    LPT1-LPT9
  • Varied filenames, using all OS allowed characters ( `output^%$#@#.,!@#$%^)(&
    is a valid filename)
  • Does your feature depend on the name of the file it uses? Try same
    file name canonical representations – trailing white spaces, .\foo = foo, short file
    format aka 8.3 format -<6chars>~<number>.<extension> (see Canonical
    Representation issues from “Writing Secure Code” for many more examples)
  • Paths of type – \\?\, file://?
  • Try saving/opening etc… by specifying a dir path rather then a file
    path.
  • What is the default path that your features use when the user tries
    to open/save/find etc… Does that default make sense? Are we changing the default according
    to the path the user navigated to last time?

Input from registry: Use the registry to store/retrieve
information?

  • Keys deleted (removed from system)
  • Registry Data Type changed (delete key, create same
    named key with different data type.)
  • Change ACL on folder that contains the key. (read only,
    can’t set value, etc ) For example: Remove key and then remove the right to create
    a key, see how application responds.
  • Delete folder that contains the key.
  • Data Contents changed (see API tests below)
  •  Make
    sure no user specific data is written in the HKLM tree (and vice versa.)
  • You can use RegMon to
    find the registry keys that your features are using.
Strings: Most apps have at least one location where strings are used as an input.
  • Null String
  • String without ‘/0’ (in C)
  • Extended ASCII chars (e.g. Alt-0233)
  • Reserved strings or sequences that require
    special treatment.
  • Strings that mean something to the underlying
    code – for example, “++”, “&”, “\n” (c++)
  • Special ASCII and UNICODE chars – for
    example CTRL characters
  • Reserved Device Names – AUX, COM1
  • A good table enumerating special/troublesome
    ASCII chars can be found on page 29 of “How to Break Software” by James Whittaker.
  • Using long strings that reach the maximum
    limit and over the limit of any functions that have string parameters.
  • White space, carriage return, tab chars
  • International Strings – lots of things
    you can (need to) do here that I won’t get into details on. Enable INTL character
    sets on your machine and have at it. Most INTL strings tend to take up 30% more space
    than the ENU versions. Look for overlaps and poor UI in your LOC versions.
  • You can sometimes get around various
    UI verifications by pasting the text rather then typing it.


     
Numeric Values
  • Negative values
  • Zero
  • Nothing (in VB, where applicable)
  • Least numeric value for type
  • Greatest numeric value for type
  • Does your numeric value input have boundaries (from 1 to 50)? If yes
    test those. – test at least 1, 50, 0, 51, and, of course valid inputs.

    Different Data Types (for example if the input expects int enter real number or a
    character)

Inputs to Web Applications

  • All previous tests apply here as well
  • Escape sequences not allowed or checked to ensure they do not allow
    something malicious to render (There are 4 ways to represent ‘\’. ‘\’ = %5C = %255C
    = %%35%63 = %25%35%63 )
  • HTML Encoding check where applicable: ‘<’ = < = &#x3C = &60
  • See http://www.w3.org/TR/REC-html40/sgml/entities.html
  • See Chapter 12 of Writing Secure Code for more checks

Network Connectivity Tests

For
all features that need network access. You can control the network speed/access by
using tools like Canned Heat.

  • What is the behavior of your features
    when the network is slow?
  • What happens when the network is
    down / goes down at runtime?

Things
to check for: hangs, crashes, bad error messages.

General Advice:

1. Try
to get a list of all the errors that the code is supposed to handle and try causing
all of them at least once.

a.     Are the invalid inputs handled
correctly?

b.     Are there any cases that
you think the code should be handling, but you don’t see an associated error message?
Test those…

c.     Are all error messages easy
to understand and descriptive?

2. Try
using the default values or blank inputs (where applicable)

d.     Ask yourself whether the
defaults are reasonable. Is that what the user would want to have as the default value?

3. Try
strings that have special meaning to the OS, Underlying Programming Language or are
special for the corresponding char set

4. Test
for buffer overruns.

5. Make
sure that your feature doesn’t allow code/script to be injected through input. Online
features and features using SQL are especially susceptible to this kind of problems.

6. Test
repeating the same input or series of inputs many times.

7. Don’t
test input only as individual fields, find related inputs and test interesting combinations.
For example if you had month and date fields, test February and -1, 0, 28, 29.

8. Think
about all the possible outputs that some input can result in. Try to cause all of
those to happen, by changing the context/environment in which the code executes. For
example, when trying to read a file from disc some possible outputs are:

e.     File is not found

f. We could not read the file – File is corrupted so
we can’t read it, No permissions to read the file, File is locked so can’t read

g.     Successfully read the file

h.     The program hung because
the file was too big

 

Common Data Type – Boundary Values

Type

Length

Range

 

2 bytes

0 to 65,535

4 bytes

0 to 4,294,967,295

 

2 bytes

-32,768 to 32,767

4 bytes

-2,147,483,648 to 2,147,483,647

 

1 byte

256 different values

2 bytes

65,535 different values

Boolean

1 byte

True or False

 

4 bytes

1.2e-38 to 3.4e38

8 bytes

2.2e-308 to 1.8e308

16 bytes

3.4e-4932 to 1.1e4932

 

 

REFERENCES / RESOURCES:

1.     Writing
Secure Code

2.     How
To Break Software

3.     Developing
International Software

Tools:

1. Canned
Heat – http://se.fit.edu/cannedheat

2. FileMon
http://www.sysinternals.com/ntw2k/source/filemon.shtml

3. RegMon
http://www.sysinternals.com/ntw2k/source/regmon.shtml

 

Comments (6)

  1. Good content here, thanks for the work! Wanted to pass along a link to the best book I’ve seen on building these sort of test parameter lists (along with other useful-in-the-real-world software testing stuff): http://www.amazon.com/exec/obidos/tg/detail/-/0131774115/qid=1070300900/sr=1-5/ref=sr_1_5/103-6870832-4476643?v=glance&s=books. (I’m not involved with the book in any way, other than being an extremely pleased owner.)

  2. bloo says:

    Are there any other tools similar to Canned Heat (besides Holodeck, its predecessor)?

  3. josh ledgard says:

    We write a lot of custom software for testing to be honest. It’s part of our dogfooding efforts. I’ve mentioned other tools we use. I’ll also soon start having testing tools show up on the powertoys blog!

  4. Source: scooblog by josh ledgard Software Testing 6: Good Tests for Bad Parameters

  5. KS says:

    It is a very good article. However lot of these cases might not occur because the property files or file names or location of the files can’t be changed to test. As this is almost changing the code.

Skip to main content