One area of software development I’m really passionate about is Quality Assurance. Why? After all, if it compiles successfully, it’s going to work, right? Besides, there’s always time to fix it in the next release.
Well, when you ship software that people pay for and your the sole developer, you can’t afford have that mentality. When I wrote DLL Checker, I thought what I labeled Version 1.0 was good enough to ship because I had handed it a couple of colleagues who did Windows configuration and troubleshooting for a living to beta test for me. Their feedback was it helped them resolve issues they had to search for manually which was a major requirement. So I put it out on the Shareware web sites and a friend of mine who wrote for PC World at the time wrote about it in his column.
As you can guess, tons of downloads and emails reporting issues from installation issues to usability to too many DLLs than my array could handle later, I needed to have much more rigorous testing strategy in place to make sure that I didn’t break features that previously worked and ensure that the new features and enhancements worked as designed/requested. I used to think this all sounded good in theory and in the classroom, but having to experience first hand as the sole developer, marketer, technical support, etc., this now become a mantra I carry with me to every position I’ve held and every client I’ve consulted with.
Enough about my trials in QA. What I really wanted to talk about is the Test Case Management feature coming in the next release of VSTS.
One of the most used tools for documenting test cases is either Microsoft Word or Excel. Personally, I like Excel because I can make forms that are easy to update but also run metrics off of. So when it comes to writing test cases, typically you have header information like Test Case Number, Title, Created By, what requirement is it suppose to test, and the steps a tester will have to take to verify the feature is working as designed. Nothing revolutionary. Because the tester might not be the same person who wrote the test, you need to be pretty explicit in your test steps as well as explain what they expect to see, plus provide an area for feedback if the test didn’t pass.
Coming in VSTS 2010 is a new work item template called Test Case. This work item template is available in both the Agile and CMMI templates. What’s nice about the template is that it covers the basics like most test case worksheets (Title, who’s its assigned to, and steps it takes to satisfy the test), in addition, it also has fields for requirements traceability and automation candidacy. To link a test case to User Story or Requirement, associate the work item under the Tested Stories. For automation candidacy, it’s as simple as a drop down called Automation Status with values: Automated, Not Automated, and Planned. Why is this imported? As nice as automated tests are, not all tests can be automated or should be automated. This drop down can at least provide development, QA, and management if a test case is going to be automated or not.
The Excel screen shot is an example of the test case template I used, while other screen shot illustrates what the Test Case Work Item looks like.
For traceability, test cases should be linked to User Stories (under Agile) and Requirements (under CMMI). VSTS 2010 ships with Team Queries to check for this. Under the Agile process, it’s called User Stories without Test Cases. For CMMI, it’s called Requirements without Test Cases. Basically, you link the User Story / Requirement work item with a Test Case work item, similar to how you do it today in TFS 2008, but unlike TFS 2008, you can’t write a Work Item query to get a traceability matrix, you need to use SQL Reporting Services for that. With TFS 2010, you can write a Work Item query that can join on the Links fields.
To check this out for yourself, you can download a copy of the VSTS 2010 CTP VPC for yourself and experience first hand.