It's automation focus week in OneNote Test

  • Article

 

We started another automation push yesterday and are turning out new scripts, fixing old, clearing automation system bugs off our plate and generally trying to get some breadth of coverage in place using our new white box testing system.

 

The surprising thing some new testers don't expect is that we file bugs against ourselves to get our automation system working properly.  Broadly speaking, testers are responsible for 2 facets of the automation system: the scripts themselves which run to test the product, and the underlying framework which lets the system run.  New testers are expected to be able to write automation scripts as a minimum requirement.  Once you can get your mind around that task, we expect you to contribute to the framework.  The framework itself (named MOTIF in Office, for "Microsoft Office Test InFrastructure") is built and maintained by testers.  When he hit bugs in that product, we file bug reports that we are responsible for fixing.  Think of MOTIF as a product we use (the Office group) and we have to maintain, add features, fix bugs, respond to customer requests and the like. And since the test team owns the product, we own the bugs in it. 

 

One of our tasks is to work through the backlog of OneNote specific bugs to reduce the number of active bugs, increase the quality of the system and ensure that when automation runs complete we get a valid picture of what the state of OneNote is.

 

One of the bugs I had to fix was this (and this is a very trivial bug, but shows the process of fixing bugs in the automation system pretty well).  The "legacy" napkin math script has a test which computes interest rate payments using this formula:

pmt(0.05;36;30000)=1813.033713614259   

 

So a $30,000 loan at 5% interest for 36 months gives a monthly payment of ~$1813.

 

The previous method of verifying the math was computed correctly used a String.Contains() method to look for $1813.  It was something like this:

string answer = "1813";

String formula= Compute("pmt(0.05;36;30000)= ");

If(formula.Contains(answer)

Log.Pass("PMT function works!")

Else

Log.Fail("PMT did not work.  The answer was " + answer);

 

I did not like this verification when I rewrote other parts of the script.  It seemed too "loose" to me:  if I am testing the number of decimal digits expected, then I need to be more precise than expecting the answer to Contain the result.  In this case, I want to verify the result of the equation is exactly 1813.033713614259, and not simply "near" 1813.

 

So I rewrote the verification routine to be something like this:

String answer = "1813.033713614259";

String result= Compute("pmt(0.05;36;30000)= ");//change the Compute function to only return the answer

If(result.Equals(answer))

Log.Pass("PMT function works!")

Else

Log.Fail("PMT did not work.  The answer was " + result);

 

Next I had to rebuild the script to ensure I'm not breaking anything.  Then I run the test 10 times to make sure it passes (some tests we run hundreds of times to ensure reliability before we enable them).  It passed.

 

Once all that is done, I submit my changes for a code review.  Another tester on the team will look through both the syntax of my change and the logic of the verification. Once she is done with that, she sends me mail to let me know the code is approved, and I can check it in.  Future automation runs will include this more rigorous verification.  Once it actually is running in the lab, I can close the bug I have on my plate which says to update this test.

 

It's the same process for more thorough bugs as well.

 

Oh, and I also added an equation test for 850*77.1=65,535.0 :)

 

Questions, comments, concerns and criticisms?

John