The value of Manual testing (or…The value of Automated testing)


I’m skipping #8 for a minute so I can get to #9.

I hope we all agree that it’s silly to take a perfectly find manual test and automate it. It’s also silly to take something that’s highly repetitive and easily automatable and NOT automate it. So, as is the theme for this series, let’s discuss what happens when your org doesn’t get it.

On one extreme, your org may want to manually test everything. Depending on the situation, it may be a reasonable choice. If it’s not, do something about it.

I’m reminded of one of my first test assignments. I was kind of a programmer. I could write simple C programs, but I kicked butt writing automation with MS-Test (the precursor to Visual Test). I wrote a massive number of questionably designed, yet very_cool_to_watch automation at my previous job and “thought” I was hired to do more of the same.

So, I walked into my boss’ office, he handed me a list of test cases and told me to get to work. I glanced through the list, asked a few clarifying questions, then asked when he wanted I should finish automating them. He replied “oh no, we don’t have time to automate – you just need to run these every day”.

I was young and a little scared to rock the boat, so I did what he asked. That was Monday. By Wednesday I was bored, and by Friday I had automated most of the test cases. Not only did I not miss any regressions during the remainder of the product cycle, I actually had time to do testing!. Eventually, I clued my manager in on what I did (he had grown to trust me by this point), and he liked it. In fact, he asked me to automate some fairly complex scenarios (which I did), and we eventually ended up with a good mix of manual and automated tests.

I guess the moral of the story was that I just did what I thought was right and got away with it. A safer solution would be to ask first, but a small pilot project  – even if it’s on your own time could probably help here.

Sometimes, the opposite happens. Teams are so focused on automating everything that they don’t actually do much testing. In this case, just do the opposite of my example above. Stop writing automation for an hour or two every day, or for 6 hours on Friday, or whatever, and use the product. If you find cool bugs (and if you’re any good at testing, you will), note that they were found by some technique other than automation. Eventually, people will notice that you’re finding lot’s of cool bugs (don’t worry, you’ll keep up on your automation work too), and they’ll ask what your “secret sauce” is. Tell them the truth. Soon, they’ll be enough sauce flowing to actually whip your product into shape.

Cool stuff.


Comments (7)

  1. Ashley says:

    Another approach is to ask your manager for assistance coming up with realistic numbers for CALCULATING whether automation is really justified or not.  I’ve used this spreadsheet as a simplistic way to show at what point it makes sense to automate:

    http://www.primarygoals.com/Technology/automation.xls

    The biggest shortfall has been when one thinks "we’ll only run this test maybe 10 times," but because it keeps failing, it actually takes 20 more passes before it really works.  This creates the pre-automation view that it would not make sense to automate, but the post-testing realization that it would have made sense to do so after all.

  2. Shrini says:

    There is some good Advise at this post of James Bach …

    http://www.satisfice.com/blog/archives/58

    A good manual test can never be automated (due to its intrensic structure and reliance on sapient human tester)

    If you have managed to automate (the execution part) a test – that is probably a WEAK manual test.

    It is probably good idea to write automated test separetely !!

    One of the thing that amazes me is people compare some units of manual testing work to same number of units of automation testing. Hence talk about ROI, economics of automation, to me looks "absurd" because in real terms manual testing and automation testing can not be compared at all.  So, extending James Bach’s notion of good manual test – I would say, if you can compare (equate) say one hour of manual test and one hour of automation testing – your manual testing is probably POOR and WEAK.

    I see that most organization do not get automation right simply because they think about automation first then about Testing. So approach is to "retrofit" Automation in testing.

    In my opinion, given a test – we should first ask what kind of test is this? who perform it better – computer or human? What are the costs involved in doing this test repeatedly? What are defects automation is likely to uncover and what kinds of defects will not be seen by automation?

    A sound automation strategy is based on exploiting relative strengths of human and machine. The automation strategy should be based on testing mission. Automation fails when testing mission follows it instead of preceeding it.

    Shrini

  3. Alan Page says:

    Thanks for the comments.

    I absolutely agree that the key is to have a good automation strategy that makes sense (although to me, Bach’s post falls into the ("please tell me something non-obvious" category). The real point of this post is how to fix your automation strategy if it’s broken (as many seem to be).

    So, don’t tell me what a good automation strategy is, tell me how to fix it. Shrini – I hear you complain about bad automation strategies often – other than flat out telling people they’re wrong, what do you do to fix it?

  4. Shrini says:

    >. So, don’t tell me what a good automation strategy is, tell me how to fix it.

    I am sure that there are no easy answers more than that there no "context free" solutions or recipes for fixing a broken automation strategy. More I dig into this, I see lots of social and business angle to this than pure "technological solution".

    Whenever I cribbed about "bad automation " – I always mentioned that autoamtion strategy should follow test strategy/mission not other way round. In my universe, testing is at the center automation revolves around it.

    How do I propose to fix it ? I would say "refocus on testing" – analyse each testing idea/task by asking who/what is best positioned to perform the task – human or machine?

    Another part I reiterate here (important element of "fix" strategy) – Never ever attempt make stories about ROI on automation by comparing one unit of manual testing to similar unit of automation testing (execution).

    As I said earlier, there are no simple fixes for a problem that has so deep technological, business and social impacts… Its a long journey and we have just begun. Spreading awareness about wrong strategies and education – may be are the first few steps

    Shrini

  5. Alan Page says:

    No organizational change is easy – especially if you’re anywhere but at the top of the organization (and even that’s not a guarantee).

    But I’m not sure that refocusing on testing will make a difference. You need to show proof, then you need to show it again. Merely explaining why one approach is better than another is never going to change anyone’s mind.

  6. Alanpa on The value of manual testing (or…The value of automated testing) Charles Sterling on New…