Why do I do what I do
As a testing consultant for Microsoft’s Premier Support for Developers organization (and a former consultant for Microsoft’s ITSM customer testing group), I often get asked about several aspects of my job, like “Where do you start looking for issues in a system?”, “How do I deal with the pressures of delivering messages to customers about their applications?”, “What causes a test to behave like this?”, etc. Luckily for me, the answers are easy: “I let the system TELL ME where it is hurting“, “I am paid to tell the truth, so I simply deliver the FACTS“, “I am not sure why it does this, but I can find out by testing it“.
The three questions and answers above sum up my attitude toward testing in general. The reason for testing is to understand how an application will behave. To test different behaviors of the application, you use different testing methodologies, for instance Benchmark Load Testing to learn about application response times, Stress Testing to see where and how the application breaks, functionality testing to see if the application is functionally accurate, User Acceptance Testing to see if the application was designed well and offers the proper functionality, etc. So what happens when you want to understand how the test harness behaves? Well, I say “Just test it!” By doing this, I can better understand how the harness works, I can better understand what changes can be made to the way I test to make the test results I present more accurate, and I can help the testing tools become better by passing on information to the Visual Studio Product Team and to the consumers of the tool.
How do I do what I do
Ever wonder why forensic analysis is so prevalent in today’s world of science and engineering? Because using facts is the most reliable way to understand and answer questions objectively (assuming the answer is not supposed to be subjective). Forensic analysis is a fancy term for “collect and study facts, then make a conclusion using only these facts.” In order to collect the facts about an application, we cause the application to be utilized (unless we are doing a static code analysis, which is beyond the scope of my blog), we collect data (using any number of different tools and logs), then we review the data to determine what happened.
If I want to understand what Visual Studio is doing, I design a test that will exercise the area in question. Then I add appropriate data collection objects, exercise the system, collect the results and analyze them.