GUI schmooey


I answered a few questions this week about automating GUI tests. One question was about recommendations for GUI automation tools for non-coders, and the other was about how much time to spend on the GUI in an MVC (model-view-controller) application.

The answers were easy. In the first case, I said that they weren’t going to get ROI from the effort, and they should just test the GUI manually. In the second case, I suggested that they do all of the automation ignoring the view/GUI, and test the GUI manually.

I could expand an entire post on why I gave those answers, but it doesn’t matter. I’m going to go out on a limb and make the following statement.

For 95% of all software applications, automating the GUI is a waste of time.

For the record, I typed 99% above first, then chickened out. I may change my mind again. The point is that I think testers, in general, spend too much time trying to automate GUIs. I’m not against automation – just write automation starting at a level below(*) the GUI and move down from there. I’m not saying that you shouldn’t test the GUI at all – I just don’t see why you wouldn’t want to test it manually, and get people knowledgeable in user experience to help. I just think that in most cases we are wasting our time when we try to automate GUIs, and wonder if anyone has the guts to stop.

 

* What I beam by a “level below the GUI” is automation that works with IAccessible or an object model rather than interacting with UI elements directly.


Comments (16)

  1. Shrini says:

    >> For 95% of all software applications, automating the GUI is a waste of time.

    Bold statement indeed … How does this work if you were to make the same recommendation in IT outsourced environments?

    Most of the automation in IT outsourced environments happen mostly at GUI layer. They call that as "GUI regression automation". This is space, by far the biggest in terms of spend, reach and practice when it comes to automation. If you say "automation" – they will immediately jump and say "QTP" or some other COTS tool.

    Why "automation under the hood/skin" is not popular in IT?

    Reasons I could think of are —

    1. Automation Engineers (called "wrongly" so) are erstwhile black box testers with only "tool knowledge". Less than 5% of this crowd is programming knowledge. Some of these are even mediocre testers.
    2. Applications are legacy and are in maintenance mode. Application devleopment is outsourced to another vendor hence really there is a WALL between GUI automation and development team

    3. Whole concept of automation under the hood – likes of TDD, xUnit frameworks etc is alien to IT testing/automation world. Very few have now started to look beyond GUI COTS automation tools due to cost aspects.

    So if you were to write this post for IT folks – what would be recommendation on GUI automation or simply automation (keeping in mind the possible challenges of IT world in general)

    Shrini

  2. Alan Page says:

    I’m writing a post for testers – whether they’re in IT or not. Regardless of what you’re testing, chances are that if you’re writing automated GUI tests that you’re not being productive.

    I have seen good UI automation that tests with button clicks. It can be convenient to run tests like this sometimes, but in most cases, it just isn’t worth the effort for an all out investment.

  3. Corey says:

    Alan,

    totally agree.. 99% might be too bold, but I’m with ya at 95% 🙂

  4. I wanted to expand on the main point of my last post – specifically on what a “layer below the GUI” means.

  5. Rajasankar says:

    From my experience, GUI automation will work if you write the maintainable scripts. Otherwise you need to redo the whole thing for every release which is waste of time and effort. And automation won’t find any new bugs but will ensure that new code didn’t break the old features.

    Rajasankar

    errajasankarchem at gmail dot com

  6. My experience suggests that while we should be automating at object interface level, as Shrini suggests, the option simply isn’t there for many applications, and where it is, it is not accessible to the tester.  While it may be inefficient to automate through the GUI, in many instances it is still the most pragmatic approach and can work very well with a good framework.  Saying that it is a waste of time in 95% of cases is IMO very far from the truth, not to mention pretty brash as it assumes that you are aquainted with those cases.  Now if you’d said that for most applications that are still at design stage, there is a much better way of planning and executing automated testing than via the GUI, I’d be with you all the way.

    Ideally, I’d like to have a recorder that recorded at interface object level rather than GUI level, and have pestered a couple of the COTs vendors to allow hooks into their apps to do this.  For many graphically intensive domains, such as CAD, GIS, etc… the user interface is still the best place to generate the test cases.  Not much response so far…

  7. Brent Paine says:

    I tend to agree with Shane on this one.

    The whole 95% theory would work great if we all agreed that an application is the sum of all it’s individual parts. However, I don’t think it leaves space for some important elements of testing.

    As someone who worked for a year and a half on a single project, and then be asked to automate it, I might look at your statement and say, "GREAT!" and go off on my merry way and automate the application without issue. That’s because my domain expertise will be sufficient enough to tell me when what I’m doing is stupid.

    For someone who doesn’t have adequate domain experience, the problem that presents itself is they will begin attempting to automate things that would never interact, their automation flow would be completely irrelevant.

    So as far as I see it, the 95% line is probably going to end up being the smartest thing anybody has ever said, to some, and the stupidist thing anyone has ever said to others.

    One point I do agree with you on, though, is that I find it very difficult to justify automating GUI to the point of verifying that buttons are in the right place or the font is correct, etc. However, I think that the GUI is a great vessel for someone who has little or no experience on the application to use their QA knowledge to estimate how the program should flow.

  8. Alan Page says:

    I stand by my statement. IN fact, I shuddered a bit when Shane used the word "pragmatic" to use fragile tools.

    I have written automated gui tests for entire applications using some of these frameworks, and they all break down in future revs of the application, or when the underlying application changes.

    If your application isn’t designed well (well designed applications never put the logic in the GUI  – if you’re stuck with one of these applications, you’re going to have bad quality regardless of whether you write automated tests or not, so why bother?

  9. So if you’re presented with an application that has been written without an object level automation interface, you deem it badly designed, and hence not a candidate for automation?  Certainly not in my experience.

    Like yourself, I have written automation frameworks to drive testing through a GUI, which years later are still running just fine.  One of the principal purposes of the framework in this instance is to abstract the GUI to logical object style interfaces, such that the tests are not fragile.  It might not be as robust in most cases as using a COM interface, but that doesn’t render it either unworkable or a waste of time.  You simply have to be aware of, and cater for, the pitfalls.  

    It is also worth noting that object based interfaces can change from release to release, breaking object based testing.  It is bad programming, sure, but what is the point of testing if not to pick up on bad programming.  Similarly, the GUI shouldn’t contain any logic, or work differently from other interfaces, but it isn’t always the case.  Good designs can be badly implemented.

    Don’t get me wrong, testing through a programmatic or object based interface is way superior if its available.  If it isn’t, IMO, GUI is a viable alternative in many cases.

  10. Alan Page says:

    To me, part of good testing is having sufficient early influence in the product to ensure I never had a poorly designed application plopped on my lap. However, I realize that for many testers, this isn’t a possible solution (see my post on the Test Test for more information).

    The point I didn’t make strong enough is that whiole, of course, it’s possible to drive an application purely through GUI automation, this is one are of automation where I worry that the time investment isn’t worth the effort vs. just performing manual testing on the GUI in the first place.

  11. Brent Paine says:

    I don’t think that Shane is referring to record and playback tools. I don’t think that any of these tools are all that useful, for the reason you state above. In general, the application will always break.

    What Shane does mention is havign a solid framework. I can completely agree with this. I actually designed the majority of my most recent project at the GUI layer. However, I used my experience in automation to avoid the pitfalls that often plague automation projects. In the end, we’ve managed to create a project that has actually just successfully bridged two separate versions of the software.

    This automation suite was built entirely through the GUI, using manual test cases, with the exception that we understood that things happening at this layer were likely to change. We used a record and playback tool, but one with a robust scripting engine also. In the end we probably used the record and playback 1% of the time. Maybe for identifying objects, maybe to see how the scripting engine wants us to express something.

    In the end, we had a substantial suite which included 3 separate layers, the global framework, the project framework, and the test cases. Built in a heirachal system, the global framework was designed to be portable to other projects. It can be used across both Windows and Java platforms (which was the requirement of our organization) and also acts as a communication medium with the automation tool. The project framework utilizes the global framework, but also implements simple elements related specifically to the project itself. At the test cases layer, I expect to see function calls over tons of scripting. When I do see scripting at the test cases layer, I question it.

    In the end, the project has survived the transition from version 1 to version 2 of the software, even though each test case is designed right off the GUI elements.

    So, maybe I misunderstand what you mean by testing at the GUI level, but in my experience, if you don’t plan ahead then you will ultimately fail anyway. Sometimes, even the best laid plans will fail. However, I think that a poor plan will fail regardless of where you decide to put your chips.

  12. TheUIGuy says:

    I am having this discussion at work right now, even after 7 years of only having manual smoke testing. The pattern is this: java or OOP engineers don’t write the proper unit tests for something, and the customer sees it. Then we scramble to figure out what went wrong, and bolster the tests – meanwhile, the UI area (presentation layer) gets the "radioactive fallout" from these discussions and is forced to consider automated testing – when all the while it is not a paradigm that the UI ever even considered, or for that matter, would ever need to do.

    Then we politely explain that the UI never introduces bugs that affect the customer in any meaningful way, and if so, occur with much less regularity – where most of them are caught during smoke testing. So, of the average 1% of bugs that come from the UI side, we need to integrate a slow, murky, painful automated testing framework? To reduce .07% of bugs to what? .02%.

    Because we all have eyeballs and fingers. This is the best way to test your UI. It is now, and it always will be.

    Not to do this begs the question: do you care about your users? Do you care about usability? Because automated tests, as error-prone and time-consuming as they are, do not care, and could never cover the use cases that fingers and eyeballs can.

    The major issue people fail to get over is that there ALWAYS will be bugs.. the question is, how can we minimize those bugs?

    Relying on a testing framework, where the engineer has to "guess" what may or may not be testable, and is often wrong, is like saying that testing is simply not important enough to be given the same level of human interaction that coding did in the first place.

    So, its all this work, for very little benefit. Thats the viewpoint from the UI side. Java engineers really do need this, because their environments are homogenous and have many different consumers touching it all day long. If a build fails, it fails. They need unit testing and smoke testing.

    The ui can benefit from change control procedures, modularization, tested frameworks like mooTools or jQuery, and a solid smoke testing environment.

  13. Brent Paine says:

    I totally agree with you UIGuy. There are certainly limits to what I think should be tested. We haven’t, and won’t, automate 100% of our application, for the exact reason that you mention.

    What I’m trying to illustrate, possibly poorly, is that if we had modules X, Y, and Z, they might all function properly as standalone modules. We might test every possible combination on each of these modules, individually. However, even though we know what they are meant to do, without a GUI we’re never really sure how these modules will interact. That is, of course, unless we are allowed source-level access to the modules. Then it becomes an organizational question of what level of access to automation staffers have?

    At the same time, it could be a matter of semantics. If we’re talking about automating a GUI to verify the GUI layout, button location, etc., that’s one thing. If we’re talking about doing all of our automation within the layers underneath the GUI, then I think that is also lacking. Then there are X-Factors (I don’t know why I always capitalize that) like how much of the app are you planning on automating? Are you being asked to automate test cases or automate a module? Do you have access to teh application source code? Are you treated as a developer or a QA person who is acting like a developer ( 😛 )?

    So is Alan wrong? Meh, probably not. I think that the number might be misleading to a lot of people only because I think that there are fewer companies who would be ok with you trudging through their source code than those who don’t mind. Not to mention, I would expect that most companies would not allow juniors or even many seniors to work at this low a level. That’s why they buy the tools to work on top of those layers. For many, the GUI layer will often be their only access to any low-level items for automation. Now, this might be my own assumption. Maybe I’m wrong. In all my time, though, I haven’t granted someone full source-level access, even if I had it myself.

  14. Alan Page says:

    UIGuy – you’e posted in a comment exactly what I menat, but failed to communicate – thanks.

Skip to main content