I did NOT hit 81 bugs in OneNote last night

  • Article

We have a tool that runs whenever we kick it off to test OneNote - edit pages, adds new, etc.. Basically it just mimics a user working with OneNote. Yesterday when I left work ( at about 5PM) I kicked off a run of this tool. When I came in this morning, I had 81 reports of bugs in OneNote the tool had found.

When the tool detects an error state, a few things happen. It gathers logs and some other troubleshooting information, copies that to a location I specify, then boots a debugger and gets a full memory dump with heap. This last bit of work can take 6-10 minutes depending on the state OneNote is in, the speed of the hard drives involved and other network speeds. So I was disappointed at first. A little quick napkin math shows 81*6=486 minutes of my run was spent on getting memory dumps. That is almost exactly 6 hours. 6 hours of a 14 hour run is 6/14=0.4286, or about 42% of the time was spent just gathering these memory dumps.

That would normally be fine. After all, this is a tool we use to save testers time doing all of this manually, and it can spend as much time as it needs to gather this information. Run time of tools can be cheap so I'm not concerned about needing to run this again (I will run it again before I leave today).

The down side to all of this was that I did not update a file to ignore known bugs which we already have filed and are in the process of fixing. So the machine got into one of these error states (technically, it thought an item on the page was a little larger than it actually was), and so it stopped the run, gathered the logs and memory dump (for 6-10 minutes), then went back to testing. Then it noticed an item on the page was a little larger than it actually was (again), and stopped, gathered logs and memory dump, and then continued.

Losing that much run time is still slightly annoying, though. But at least I know why it happened and can prevent this silly mistake from happening in the future. I really just wrote this up to reinforce in my mind the rum time costs (or manual testing costs) of gathering all the information we need for bug reports.

Losing that much run time is still slightly annoying, though. But at least I know why it happened and can prevent this silly mistake from happening in the future.

Questions, comments, concerns and criticisms always welcome,

John