Notes about testing the ink colors we added

Last month we shipped an update to the Windows 8 store that had added functionality for OneNote. I wanted to take a little time here to write broadly about what we did to get this update shipped. I won't focus on details but instead walk through the test process for signing off on this change.

Specifically for ink support, we added the ability to change the color of the ink strokes you can make on a page. When we first released this version of OneNote, we more or less had the Henry Ford approach to colors - "You can have any color you want as long as it is black." We knew we needed to change this and made the decision to ship an update with this functionality.

As far as features go, this was not a large change. We have code in OneNote 2013 to change pen color so we just needed to port it over. We also were able to draw ink on the page so there were not many changes needed there. The test problem here was also well understood so we got started on this pretty quickly.

Testing started with manual tests - verifying the new colors were actually being drawn in color, ensuring that if you change the pen size then it persists and remains that size when you use it again and checking that the radial menu had the proper text and icons for the UI. This went pretty well and there frankly weren't many bugs with the implementation. Adding in some other tools we have to simulate shutting the application down, running out of memory and other stresses produced one total bug - that's a sign the feature is on the right track. If you spend days stress testing and only get one bug for a small feature like this, you can take that as an indicator that your methodology is sound.

Then we pronounced this ready to ship. Test broke free to work on some new functionality and tool creation (you can never have too many tools in your test toolbox) and we sent the final UI off to our localization team to get the strings localized into all the languages we support. Once we got that back, we had one final check to ensure those resources were correct and functioning properly, then put the app in the store.

Even then we weren't done. I checked the app immediately to ensure it had our changes (it did) and then we started to monitor the reviews on the store, on the forums and elsewhere on the internet. Thanks for all that feedback. We really do take it to heart!

Now back to testing and adding more features.

Questions, comments, concerns and criticisms always welcome,

John