Section 3: UI Automation execution covers test hole which normal test cannot

  • Article

Section 3: UI Automation execution covers test hole which normal test cannot

 

In previous sections, I discussed the UI Automation preparation and development. I want to deliver the message that UI Automation is not test repeating happens automatically. It is unique. UI Automation execution does not only compensate human flaw and costs, but also cover the test path that other testing cannot.

 

The benefits of UI Automation are obvious. Testers do not need to repeat the boring tasks. Regressions are found earlier. No human mistakes and the test results are accountable. It is cool but not limited above.

 

1.      There are bugs that can be easily found by Automation, but difficult by manual

 

Real case one:

 

Our project is WPF based. The feature is something like Window explorer. It contains the address bar and when the user inputs the path, the application loads the data from that path.

We have a test case to start the application, type some path, and check the data loaded. In manual testing, it is perfect. It works every time. We developed the Automation case to perform the same flow. The Automation crashes the target application when it executes the first time.

 

The root cause is about threading. When the application starts, it creates a worker thread to perform some initialization task. In manual testing, as the user type speed is not so fast, the worker thread exists safely before the application starts the data loading from the path. UI Automation makes the whole execution pretty fast.  The application starts to load the data before the worker thread exists, so the initialization does not complete and leads to data corruption with crashe. When the root cause is identified, we start the application and paste the path to load instead of type, and the crashes happens manually.

 

Real case two:

 

In our UI Automation, we wrap the Automation task in test helper classes. We have create, delete, rename, move, add group, lock/unlock and other tasks. With Automation, we can easily generate flows to execute these tasks, with specified pattern or total random.

 

The auto generated test flow find two special bugs. It generates a random test case like

 

a) Create userA

b) Add userA groupB

c) Add userA groupB

d) Add userA groupB

 

The test execution fails at step d)! The problem only happens when you try to perform the same add group task the 3rd time! The root cause is, adding user to the same group more than once leads error. The error flag is not handled correctly in step c), thus the problem happens in step d).

 

In manual testing, 99.99% tester stops at step c). Nobody will test by “adding to the same group” once, once again, and again and again.

 

Another similar issue is, object move task works fine in 99% cases, except move the object from current location to grand parent.

 

Add an example from One Note team:

https://blogs.msdn.com/johnguin/archive/2008/09/08/investigating-an-automation-failure.aspx

 

The UI Automation execution pattern is different from human. It is unique at detecting threading issue, randomization and strange combinations.

 

2.      There are test path and coverage can be easily reached by Automation, but difficult by manual

 

We have a test pass to measure the memory usage by running daily task for the UI application 24*7. Are you going to hire a monkey?

 

Stress testing is a typical example for test path which can only be covered by Automation.

 

Another real case is permutation. Our application is designed to handle different types of object. However, it is difficult to predict that all the types are known for us. The only way is to go through every folder and sub folder, to load every objects in the whole organization. With Automation, it is very simple to do it recursively. Eventually, we find a bug which corrupts the whole navigation feature. We made an incorrect assumption that the indicator between child and parent in path goes with a static pattern. It turns out the pattern is configurable. I cannot image if the product gets shipped with this kind of bug.

 

Summary:

 

I skip other obvious benefits Automation can bring. If the three topics give a clear definition of what UI Automation is, what it brings and what you need to prepare, it is unnecessary to discuss when to apply manual and when to use automation. Triage by yourself.