A new process for checking in tools
Our test team is responsible for making changes and checking code into our repository for a variety of projects. We obviously own our test automation framework and test scripts. We also can make changes to OneNote itself to enhance its testability. Additionally, we can make changes to a "tools" project we own.
Tools differ from test scripts in several ways. First, our test scripts build each time OneNote builds and tools typically do not. While the tests need to reference OneNote (to exercise its functionality or to export page content via our XML API for verifications), tools usually are stand alone applications. Log gathering tools are one example - they just scour the machine and copy files found on it to a central location and do not need OneNote at all. Second, the tools can be written in a variety of languages - C#, C++, batch files, etc… Or they may simply be collateral files - registry keys, text files, images, etc…
But making changes to them needs to be as thoroughly tested as changes to any other code we own. Even the log gathering tool can't be missed for a day - we need the logs uploaded for analysis and don't want to miss any of that telemetry. And changes to images or text files need to be broadcast to the rest of the team to prevent confusion ("Why did my test insert a green square? It was supposed to be blue - there must be a bug somewhere." is a line of thought test should prevent.
So I sent a mail earlier today to ensure that any changes made to tools - and can be shown to only be in our tools project - need to:
- Get a code review.
- Get a PR test complete. "PR" means private release, and is our lingo for the person making the change building a private version of the changes and having a tester install/use/verify the changes perform as expected. For our tools, this may be as simple as ensuring the image has been changed as expected and letting stakeholders know. For the log gathering, the testing may be more complex depending on the scope of the change.
- Kicking off an automation suite run is always recommended. For some changes, though - like changing a text file that describes how to use a tool - this requirement may be too much overhead.
- Finally, some adminstrivia for the needed command line syntax to submit your changes.
I'll keep my eye on this and make sure it is not too much overhead and also prevents regressions (bugs caused from bug fixes) from getting into our tools.
Questions, comments, concerns and criticisms always welcome,
John