Here at MSFT, specifically at DevDiv, we have specific testing categories called “Exit Criteria” – basically gates that you need to pass through in order to ship a release at any milestone. These criteria represent different flavors of testing that your product has to go through in order to be pronounced “fit for release”. For eg: localization, accessibility, performance. At DevDiv, we have 26 different exit criteria that we need to pass through before we ship!
Now, as is the case with any “process”, this one has also been the subject of passionate debate among the MSFT denizens as to the value this really brings to our product quality. I have heard arguments around how this is a lot of overhead at a very busy time in the ship cycle. Well – I have to concede that these set of tests really do not bring in a lot of bugs. Especially, criteria like “branding and copyright” – where we just test to see if the product is carrying the right trademarks everywhere, the tool names are mentioned correctly etc. Also, when criteria like “configuration matrix” is in question – this is basically the gate where we ensure our products run ok on different operating systems, different versions of CLR, different versions of IE, different bitness…some of us testers tend to go berserk and many times, products get overtested. And of course, there is this eternal danger of manual testing (which is still the best source of the most important bugs) getting undermined and sacrificed at the altar of checking off items in a checklist.
In spite of all of this, I still opine that the exit criteria proc is one of the best engineering practices I have seen here at home. If you are a tester testing a product with a customer base numbering tens of thousands or more, I am sure you appreciate the sheer breadth of things you need to cover in your test planning. The number of dimensions is really mind boggling – security, performance, localization, globalization, backward compatibility, accessibility, stress, endurance, limit…it sure gets crazy. Exit criteria really bring a semblance of sanity in this whole universe of testing. I would term this the distilled essence of the different testing dimensions that go into testing a world class product. It serves as a map to help testers navigate the maze of testing requirements and builds confidence into the quality of the product step by step.
The Globalization EC for instance, is so very well thought out – we have different configurations like Italian, Turkish, Arabic; with each configuration defined by what is different in that config and what are the kind of bugs you can expect to hit. For instance, ITA is one of the “most localized” OSes, which help catch issues where we hardcode stuff like security group names. I remember Team Build had this bug in 2005, where we had hardcoded to check if user belongs to “Administrators”! The test passed on JPN, because the admin group was still “Administrators” on the JPN OS, but it failed on German OS, since the group was “Administrateren” on DEU! Turkish has the infamous sort problems with the quirky dotted/undotted I’s. Testing on Arabic configs throw up specific problems rendering bidirectional text or right-to-left UI mirroring etc. To me, this is a treasure trove of info put together by several years of testing experience. Who wouldn’t want to use this cheat sheet? 🙂
More on exit criteria in later blogs.
As we get closer to shipping, it is getting more exciting here…hope you guys like Dev10…