I got a report that the OneNote printer automation test I own was not passing recently. I doubted the printer was broken and looked at the test. One of the verifications it has is a check of the number of embedded files that get added to the offline cache after a document is printed (the test prints the text of Patrick Henry’s "Liberty or Death" speech). the failure was that a negative number of files was found.
That was a sure sign the test was faulty as opposed to the printer breaking..
The way the test was designed was to verify that a single text file was added to the offline cache folder after the print operation (and 3 PNG files, and an XPS file, but they all had the same problem). The way the test was designed was to start a counter at 1, then decrement it each time a text file was found in that folder. If, at the end of the test, the text file count was exactly zero, the test would pass. A little bit of thinking made me realize that if there was already a text file in that folder before my test ran, the counts would be off, and the decrementation of the counter would go negative. It turns out that is exactly what was happening.
So the fix I prepared would check the folder before invoking the printer and get a count of the files already there, and then verify the correct number of files was added to that count. This works fine.
But the other part of the fix was a bit more confusing. Our automation system is supposed to give me a "clean" machine for my test. In this case, this means the offline cache folder should be empty. The machine was being created cleanly, but files were getting put there somehow. Once a machine has Office installed, it gets checked out for test 1, then when test 1 is done, gets checked out for test 2, and so one. This saves hours of time since the machine does not have to be recreated for each test. Each test is responsible for "cleaning up" after it is done – delete any registry keys you created, delete any files you copied, etc… So there was a test somewhere what was leaving these files in the offline cache folder.
As I was looking up the name of the function that needs to be called to clean the machine, I noticed that the printer test I was fixing never had this call. It never cleaned up from previous runs, so if it ever ran twice on the same machine, it would fail on every test except the first. It would never clean up those files.
So the second part of my fix is to add the clean up (named "CleanUpAfterTest") to this script.
Lastly, we will need to make a check of all of our tests to ensure this is always being called.
Just a small work item that came my way last week.
Questions, comments, concerns and criticisms always welcome,