Section 2: UI Automation development is an important, unique, and effective bug finding process
Section 2: UI Automation development is an important, unique, and effective bug finding process
If you think UI Automation is no more than repeating the manual testing automatically, you are completely wrong.
Many people expect to record the manual testing, and reply. They think tester just designs the test case once, records it once, then all the boring repeating jobs are auto! They try all the best to get there and they believe reducing cost is the only benefit Automation can bring. No, they are wrong. First of all, please check my previous post. This kind of recording/reply Automation is not a good Automation. It is not flexible. It cannot handle various inputs to verify. It cannot suit when resolution or window starting position changes. This kind of UI Automation is a typical failure. If your assumption on UI Automation is above, UI Automation will definitely hurt you.
In this post and the next, I will continue to discuss the meaning of UI Automation. I will focus on UIAutomation development here. Next I will focus on execution.
UI Automation development (before executing) is a unique bug finding process. It is NOT a combination of any existing testing mechanism plus automation. It is NOT derived from manual testing. It is unique, high effective and great coverage test mechanism which can compensate all the other existing test methodologies.
1. Accessibility and localization bugs are exposed naturally when developing the UI Automation.
The first step for any UI Automation is to access the target UI elements. The tester needs to find a proper way to probe, identify the target UI element, and then read the information from UI to verify. If the tester cannot find AutomationID, AutomationName, and shortcut key to execute UI task, bugs are found! Most of the accessibility issues are exposed naturally in this stage.
A good UI Automation should work well for localized build. The tester needs to write code to work with product resource file and resource key. If some string is hard coded in product, the bug is also exposed naturally.
Actually, Accessibility is very serous for Microsoft products. If an Automation name is not well defined, it is a S2 bug, the same as major function break. In early stage, our dev team did not know UI Accessibility requirements very well. The test team could help a lot in this stage to prevent hundreds of S1/S2 bugs.
2. The tester needs to solve the same problem as developer does. More people working on the same problem means more coverage. (This is not limited in UI Automation. It applies to all kinds of Automation tests)
In manual test, the tester starts the UI and plays with it. Most of the functions are visible and the tester will be influenced by the UI. The tester will try different patterns and different data to break it. Probably the tester will pre-define the test cases. However, these cases are usually based on scenarios, patterns, boundaries and other test metric.
It is great. It is important and valid test scenarios. We should continue to invest and apply these tests. However, UI Automation development brings another test strategy which compensates a lot.
In the manual tests, or for the pre-defined test cases, the think pattern is usually: can the product meet this kind of corner requirement? UI Automation development brings another think pattern: how can the problem be solved and all the requirements be met correctly?
To make a good and flexible UI Automation case, the tester cannot rely on static input/output. The tester has to solve the same problem just as the developer does. When the tester starts to work on the solution, the tester faces the same set of issues the developer needs to handle with. It includes the corner cases, exception/error handing and assumptions. It makes the tester understand the problem well, to really think about what a good solution can be. It helps test coverage because the tester is suffered with the same problem, and it also kills invalid test cases. In this process, the tester may want to proactively review product code, so it leads to more benefits:
l Effective code review and design review with specified goal in mind. It is the most effective way to find bugs.
l Complex measurement like performance can be easily covered in early stage from review.
l It helps the communication between dev and test
Put notepad as an example. Notepad reorganizes ANSI/UNICODE/UTF8/UTF16 formats automatically. UI Automation for this case should operate notepad to open different kinds of file, and check the string showing in the window. How could the test know the expected string? The tester has to write the code to reorganize formats just like the notepad.exe does. (Of course you can generate string first, and save to different formats. But you know what I mean)
Microsoft defines a special field in bug database to identify how bug is found. “Automation development” is a very frequently used value for me.
3. The tester has to triage the severity for UI Automation test tasks. It makes the important features tested first.
You can play around for everywhere in manual testing. However, you do not have the resources to create Automation to cover all the features. You have to put important thing first when creating the Automation.
Managers may like this. Manager cares about the priority and severity. They can bear tons of S3/S4 bugs but they hate surprise in major functions. However, testers trend to show how strong they are to find corner issue no matter how corner it is!
With UI Automation, the manager can assign challenging tasks for ICs. ICs are happy to take the challenging and interesting tasks. Manager is happy because testing goes with severity. ICs are happy because they are finding bugs in a challenging and effective way!
In previous post, I discussed the meaning of being able to create UI Automation before start. In this post, you see the meaning of creating UI Automation before executing. UI Automation can give you so many even before it starts.