Spec reviews for world class testers

  • Article

Hi - I have created a new category called Testing in my blog to track all my testing related posts, of which I am going to post in plenty, you bet! We are sooooo close to shipping, that I can almost see it around the corner. That motivated me to get my butt going on putting together a presentation on how testers would spend the very first milestone of a product.

Let's talk about spec reviews. The phase where your product feature specifications will be presented to the entire feature team for review.

So, what would a normal tester do in this phase?

  • No spec? Here's your bug - First you need to ensure that all necessary features have a spec doc. Yes – it sounds elementary, but I say this explicitly since testers will have to bear the brunt of non existent specs. Devs code whatever they want and define buggy behavior to be spec and tester has no proof against that without a spec.
  • You are an inspector! – Inspection of specs need to be done before going into a review. It is necessary to have sufficient context armed with the right set of info to push for in the review. There is not enough time in review meeting to do this study - you have to inspect the spec before hand for spec bugs with a handy checklist to check the spec's conformance against.
  • Customer, customer, customer - Look at all features as a neutral customer would. Who is the target customer? How useful is this to you? Is this prioritized in the right order - must have, good to have, can't ship without? Think through customer scenarios – different production environments, what sense will it make? Any flexibility required in feature? Missing options, information overload? Is there an overlap of functionality between this feature and another? Is there a better way of offering the same functionality? Have customers ever mentioned these scenarios before in blogs, forums or any other feedback? Will these feature be a costly embellishment? Are there chances for extensibility that you are shutting off right away?  
  • What if? Make a tree of all possible different situations. Product behavior must be clearly documented in each case. Restarting, restoring, jumping states – all possible scenarios to be thought about. Error messages – are they defined per error state? What happens when different platforms are used? App interoperability, low end machines – what happens? For instance, if your component is generic types, think of all scenarios around generic types – will it work for a generic type of a generic type, what about user defined data types, complex data types, data types that are not defined, data types that are not resolvable etc.
  • Testability is YOUR responsibility - You have to build testability into your product to facilitate testing innards of the product. Think ahead about what testing will need to be done on a feature and check if testability is built into that feature. For instance, a system that does complex calculations on a set of data and gives a single output at the end will need to have test hooks built within the product so that intermediate calculation results can be verified against an oracle. Now is the time to think about building testability into the product. Inserting test hooks into the product later on is going to be a messy affair.
  • One spec bug is worth 10 code defects – Bugs found at this time have greatest impact and least cost to fix. Miss finding a bug now and it is going to result in a dozen code defects that are a dozen times more expensive to fix later in the product cycle. Evaluate spec carefully and file bugs relentlessly.

That's about a normal tester. But, you don't want to be just a normal tester, do you? You want to be a world class tester. What would you do other than what is mentioned above?

  • Duh, where's the button to shut this off? - Think of usability aspects of the feature spec. Does it sound intuitive to a user who has never seen the product before? Is the interface simple and understandable? Are the UX guidelines followed while designing this spec? Is the functionality way too obscure or undiscoverable. For instance, are the menu options in an app well arranged? Are they logical and sequential? Are they named appropriately enough for the user to know which functionality falls in what bucket? Track with your UX guys and pound on the usability aspects right away – this saves you feature redesign later in the game that will cost hundred times more than what it would now..
  • I love Japanese, German, Arabic... - Think of international aspects in the product spec. This does not mean start testing the screenshots for date time format. Think of how the spec will lend itself well to international usage. Are there icons with letters embedded in them? Are there pictures with letters embedded? Is there a feature that may not work for international characters? For instance, is there a command line feature that takes only command line input and cannot be passed a file? That way the user wont be able to enter a string with chars spanning multiple code pages.
  • Kaleidoscope - Look at the feature from all other respects too – security, privacy, user interface experience, accessibility etc. Are there any security holes like your app being required to send passwords over unsecure channels or has the feature been designed to store user data at a common place where accessibility to that data is cheap and easily hack-able? Is the UI pleasant and usable? Are all UI guidelines being adhered to? Are the controls on the UI looking ill chosen? Are there better substitutes to these? Are all controls designed to be accessible?
  • You broke me? - Backward compatibility must be ensured for a V2 or more product. Will the feature work on all OSes that you have promised your users it will? Does the feature break existing functionality of the previous version? For instance, does your feature rely on functionality provided only by Win2K3? Will your previous users be able to upgrade to your next version on Win2K?
  • Piece me together - Integration scenarios. Your product can have several PMs – each involved in writing her own feature. How well do these features work together? What kind of interaction do these pieces have with each other? Are they seamlessly knitted together or do the integration scenarios look like an afterthought? What about external components outside of your product itself? How well do they integrate with the features you are building? For instance, consider SQL server and windows designers. When you are reviewing a spec talking about data binding with controls, you have to think of what are the different features that SQL server offers and what are the best structures to use to bind data.
  • It's my domain, beware - Domain expertise is a crucial need for testers reviewing a spec. Without full knowledge of the technology being used, you cannot fit into the customer's shoes and think of all relevant scenarios. For instance, reviewing the feature "Building C++ projects with Team Build" entails that the tester should be an expert in C++ projects and must be aware of all the different compile options and build environments that support these.
  • OOF: Gone camping in my PM's office – Unarguably, the most important of all – it is your responsibility to ensure that the spec is complete in all aspects before signing off on the spec. All of the above points must be thought of and documented in detail by your PM in the spec. Do whatever it takes to get this done. File spec bugs relentlessly. Go, sit in your PM's office and stare over her shoulder until this has been completed. Refuse to sign off on the spec until you are completely satisfied with the amount of detail in the spec. Remember – testers are at the end of the product cycle – YOU will be the worst hit if a spec is incomplete

Finally, at the end of all this, hope we would have had a great spec review! :)