How Test Case result is computed in Test Runner?
Note: Cross posted from Rubel's Blog.
“I’ve NOT marked any test step BUT test case result is auto-computed as passed sometimes and failed sometimes.“
“I’ve marked a few test steps to pass / fail but auto-computed test case result is not what I had expected.”
Have you also facedsimilar kind of issues around auto-computation of test case result? Do you find test case auto-computation logic confusing sometimes? if yes then it won’t be any longer confusing for you after reading this post :)
Before understanding test result auto-computation logic, you need to understand difference between the simple test steps and validation test steps. While creating a test case in Microsoft Test and Lab Manager you can list the action and the expected result for each test step in your manual test case:
- Under Action, there is action that the tester takes to perform this test step.
- Under Expected Result (Optional) , there is result that user should expect after the action has been performed.
Simple test steps just have an action assigned to them and expected results is NOT set for them. If you set expected results for a test step, it automatically becomes a validation test step (shown by the red check mark on the step icon). E.g. in test case shown below, 2nd and 4th are validation test steps whereas 1st and 3rd are normal test steps:
In Test Runner also validation test steps are differentiated from normal test steps by an extra red check mark on the test step result icon as shown below:
So now as you understand the difference between validation test steps and normal test steps, this is simple algo used for auto-computation of test case results in Test Runner:
- If your test case has ZERO validation test steps then your test case result will be auto-computed as “Passed” unless you’ve explicitly marked one / more test steps as failed.
- If your test case has one / more validation test steps then your test case result will be auto-computed as “Passed” if and only if:
- ALL validate test steps as marked as passed
- NO other test step is marked as failed
In short, it is mandatory to mark Validation Test Steps. If you don’t mark any validate test step to pass / fail then it is considered as FAILED so please do mark ALL validation test steps in your test case to ensure that test case result is auto-computed properly.
Note: Cross posted from Rubel's Blog.