Using customer feedback to alter testing plans

  • Article

 

The past few months have been pretty busy for the test team, but busy in a good way.  We just finished shipping the Office 2007 suite last fall, and even though it is finished, we still have plenty of work on our plate.

 

In unprioritized order, here is a partial list of our tasks:

  1. We have Office 2003 SP3 testing (various tasks to complete)
  2. We have Office 2007 SP1 work to define and test
  3. We have ongoing automation tasks
  4. We are overhauling our OneNote automation to remove dependencies on the UI. This involves a wide range of tasks including planning, training, updating our automation framework and testing our new code.
  5. Start planning for the next version of OneNote and give developers and program managers testing based feedback on new ideas.

 

I want to write this time about this other commitment we have, and show how it applies to testing:

  1. Maintain a customer connection.

 

Each tester is tasked with taking on this commitment.  She is free (interesting trivia: OneNote test is 70% female) to define this role on her own and gets management to agree.  Then she goes out and does it.  For instance, those of you that read our discussion group (here) may have seen Irina Yatsenko step in and respond to questions when she gets some free time.  

 

One of my commitments as a new tester on the team was to simply get in touch with OneNote users.  In addition to this blog, meeting users around Microsoft and beginning to respond to the discussion group, I was given the chance to visit a local user of OneNote.  These "customer visits" are always informative and this was no exception.  It is always eye opening to see how people actually use your product, and the way they work around problems they encounter.

 

I visited a lady who uses OneNote to track her research for articles she writes.  The most fascinating aspect of her workflow to me was her refusal to have more than four notebooks showing at any time.  She felt limited by her screen real estate and 4 notebooks were all that would fit without text getting truncated.  When she hit this limit, she would start archiving and closing notebooks.

 

Using this information as a tester is my next step.  Some obvious things I will do will be to note her screen size (1024x768, 96 dpi) and ensure all my test cases have this as a configuration so I can guarantee clear names with 4 notebooks.  Next, ensure I have other DPI (dots per inch) cases: at least 120, 144, 192 and whatever the max resolution the next version of office will support.  More subtle testing techniques are verifying any new features we add will be presented well onscreen with just a few notebooks as well as users that have 20+ notebooks.  I'll look for things like extra whitespace in any new dialogs which may appear if a user has a small number of notebooks.  Then I will review our entire hardware configuration test plan to see if there are any changes I will need to make to apply to this particular case.

 

Those are a few examples for "UI" (user interface) testing changes I will consider when I test.  For those of you who installed my addin for importing text files, you will notice I give you a notebook chooser to select where the imported files go.  The idea to use a folder tree to let you choose where to send the information came from her.  She never specifically said to do that - it was watching her combine her notebooks to archive them to get down to four notebooks that gave me the idea.  I was watching her manually move pages around and thought that if I could make it easier to create them somewhere other than Unfiled Notes.  So when I created the importer, I had a goal to allow for a way to select where to put the pages.

 

As always, comments, questions and concerns are welcome.

John