Hiring Great Testers – Tester Roles


In my mind, there are basically three roles on a test team.  These three roles are: developers, scripters, and those who execute the test cases (runtime testers).  In reality there is a spectrum of capabilities on any team but I think most roles will be closely related to one of these three roles.

Test Developers are the heart of a modern test team.  There was a day when you could get away with hiring a few people to just use the product and call that a test team.  This is no longer the case.  Products are becoming more complex.  The lifespan of products is increasing.  More products are being created for developers instead of end users.  These have no UI to interact with so simple exploratory testing is insufficient.  To test complex products, especially over an extended lifespan, the only viable solution is test automation.  When the product is an API instead of a user interface, testing it requires programming. 

Test developers are programmers who happen to work on a test team.  It is their job to write software which executes other software and verifies the results.  The test developers need to be at least as skilled as the developers of the product.  They need to know whatever language their product is written in.  Their job is often as difficult or perhaps even more difficult than writing the product.  Sometimes the job can be done in a simple fashion where each function is just called with various parameters and a simple method of pass/fail is monitored.  This could be a success result or some notification from the program.  A better test developer will not rely on a program to tell him if the right things happened.  Instead, he will monitor the actual behavior.  For a simple function this might just be the return result but more often than not, it requires monitoring the output of the program and making a complex decision based on it.  Test developers are also responsible for creating the test harnesses and systems for their teams.

Scripters are those who have some programming ability but are not as skilled as a test developer.  They will usually know a programming language but on that is less complex than the language the product is written in.  Scripters may know Visual Basic, Perl, Javascript, or even C#.  More important than which language they know, what distinguishes them from a test developer is that their understanding of programming is less advanced.  They will understand the language syntax but their understanding of algorithms and data structures is likely less substantial. 

Scripters will often play a role where they spend a lot of time living the product.  They will use their programming skills to script the product (if it has such a feature) or to utilize tools to drive the product.  The tools could be UI toolkits like Visual Test or something written by test developers.  Because scripters have an understanding of programming they will be able to have a deep understanding of the product.  They will be able to quickly determine the root cause of a failure and see relationships between seemingly unrelated bugs.  Scripters play an important role on any team.  They often handle the bulk of the test triage efforts and closely monitor the needs of the users of the product.

In some teams the responsibility for setting up machines, installing builds, and executing test cases falls to a group I’ll call Runtime Testers.  This role involves no programming.  Runtime testers usually don’t have any significant programming knowledge and if they do, they don’t often get a chance to utilize it.  They play a fundamental role, however.  Because they are the ones running all of the tests, they often have the best knowledge of the state of the product.  If you want to know what really does and does not work, ask one an execution specialist. 

Sometimes this role is relegated to merely clicking buttons.  They run an automated test and log some errors.  This is a waste.  They should be tasked with understanding the product and playing with it.  As I’ve stated many times in the past, there is a critical need for manual testing on most project.  These are the people best equipped to do that.  To get maximum value out of runtime testers, they should be encouraged to live the product.  To become experts in the functionality the way a customer might use it.  In doing so they can become advocates for the customer and find the usability issues and corner cases that automation cannnot easily find.

Each of these three roles is critical on a test team.  If you build a team with only one or two of the roles, you’ll be missing something fundamental.  Each project will require a different mix of people.  Some need more test developers, others more runtime testers.  The importance of these roles comes into play not only when you task your team but also when you hire.  If someone is to be a runtime tester, hiring a CS grad may not be the best idea.  They will likely be bored and quickly move on.  Hiring a self-educated computer geek may be a better fit for the role.  Likewise, if you need test developers, hiring someone who only knows perl is probably not a good fit.  Instead, hire someone who is truly a developer.  My rule of thumb is that if they couldn’t cut it as a dev, don’t hire them as a test dev.

Comments (14)

  1. I’m going to be doing a series not on testing but on the people that carry it out. This will be a post

  2. Niteen Yemul says:

    This is informative blog entry. I liked it. It describes different roles which are complementary to each other.

    ~Niteen

  3. Hi Steve,

    Interesting perspective.

    For the sake of readers who are less experienced in development and testing, I’d like to respectfully challenge some of the points made. I wouldn’t want, say, a student to read this and take it as gospel (although a good student never would)…

    I’m sure your reply, or the replies of others, will help readers who are less experienced in development and testing to gain a broader view.

    So… here we go.

    1. Roles vs. People

    Are you highlighting roles or people? For example, one of my "runtime testers" is CruiseControl (or some other continuous integration server). I.e. a role performed by a computer.

    I am also a test-developer, a scripter and a run-time tester when required. I’ll perform whichever role is needed at a given time. I.e. three roles performed by one person. Rarely am I static in one of these roles.

    Also, an application-developer may also perform the role of test-developer and run-time tester especially if they are employing Test Driven Development…

    2. Context

    I wasn’t sure if you were saying if this is what your (experience of a) test team is like or if you were saying this is how all test-teams are or just observing some common roles.

    I wouldn’t disagree that some test teams can comprise of these roles but not all need to all the time.

    I tend to work as part of the development team, as a testing specialist and testers who work with me are also part of the development team. Just because they are testers who are part of one development team, operating in close partnership with other roles (such as application developers),  does that make them distinguishable as a "test team"? In my mind, no… We’re all just part of the development team and essential in delivering a working, useful and shippable product.

    Furthermore, on some projects, this cross-functional development team may instead deliver to an integration test team that might not require a test-developer.

    3. Simple Exploratory Testing?

    Do you mean that "Exploratory Testing is simple" or do you mean a "simple form of exploratory testing"?

    Exploratory Testing is an approach that is far from simple if applied in a reasonably mature way. Some people may do little more than simple ad-hoc testing and call it exploratory testing, whereas others, like James Bach, illustrate how it is a deep-thinking intellectual activity that must be practiced and augmented with extensive learning. The type of Exploratory Testing that people like Cem Kaner and James Bach talk of is, for the average person, far from simple.

    If you mean "a simple form of" then it is true that I’ve often seen people do a very simple form of exploratory testing and sometimes that’s enough. But, it’s important to realise that exploratory testing is a lot more than a basic form of ad-hoc testing.

    It is one of those things that you can start simply but it is important to realise how difficult and challenging it really is. I’d argue that if it feels easy then, for most people, they probably aint doin it right 😉

    For example, basic ad-hoc testing that is mostly intuitive and with a small amount of logical thinking is the "hello world" of Exploratory Testing.

    In the same way that printing "Hello World" to the console is programming but a simple form of programming, basic ad-hoc testing is (IMHO) a form of exploratory testing but a simple form of exploratory testing.

    4. Value-weighting or Hierarchy?

    I also detect an element of hierarchy or perception of greater value based on programming skills. Am I wrong?

    A good tester is as valuable as a good developer. The point being is that there are few that can truly be good at everything so we balance the team.

    Furthermore, a good runtime tester performing mature "exploratory testing" would dedicate as much time and skill to studying and developing that skill as a good developer would spend improving their development skills.

    In my experience, having only developers (and no testers) would result in delivering acceptably high quantity of unnacceptable quality. Having only testers (and no developers) would result in delivering unnacceptably low qantity of code  of acceptable quality… In either case, the product isn’t shippable and as well not have been built… so both parties are symbiotic and equally valuable. Hierarchy should be based on capability within their specialisation, not based on whether they can write code or not.

    P.S. In my mind, a good test-developer should be able to cut it as either tester or developer. Some are better developers than they are testers and some are better testers than they are developers… Personally, I fall into the latter category.

    –In Conclusion —

    I propose that:

    1. Roles are roles and not necessarily people

    2. There is no one right and wrong way of structuring a testing team

    3. Mature application of Exploratory Testing is far from simple

    4. Programming and testing are both

     a) sufficiently deep that either one can consume one’s entire learning time and you’d still not know all there is to know…

     b) of equal importance in successful delivery of a project…

    …thus, value can’t be legitimately or logically weighted based on an individual’s programming skills

    I look forward to any discussion that follows…

    All the best

    Antony

  4. Anand Deshpande says:

    Surely, This blog information will help us to define the dedicated , long term Automation team strategy. It will help us to build/developed an Ideal Automation Testers Team.

  5. Funny you should bring up this topic.  

    I was at AWTA (Bret Pettichord’s Austin Workshop on Test Automation) this last weekend where we ended up discussing tester developers and developer testers in great detail.

    That wasn’t our original topic.

    We were there to talk about open source test automation tools.  But the more we talked about tools, the more we got into example test automation code, the more apparent it became that not just programming skills, but true development skills, are essential for good automated test development.

    Being able to crank out a for loop isn’t enough.  Knowing multiple languages, understanding good code design, refactoring, and unit testing are important skills for the Test Developer.

    That’s a far cry from the ability to edit the code generated by a record-and-playback.  And it’s making me think a great deal about what skills someone doing test automation really needs to have these days.

    I appreciate your post.  It’s added to my thinking…

  6. SteveRowe says:

    Thanks Elisabeth.  Are there any notes or papers or anything that were generated at this workshop that I might be able to access?  The subject of what it takes to make a truly good test developer is one I think about a lot and I’m always looking for more input.

  7. Steve asked, "Are there any notes or papers or anything that were generated at this workshop that I might be able to access?"

    You’ll find all the publicly available stuff here: http://awta.wikispaces.com/

    See http://awta.wikispaces.com/Role+of+ToolSmith for some details on the tester-developer role…though not all the conversations were captured, so that’s just a subset.

    Perhaps the most significant thing (for me) that may come out of AWTA is…

    …another conference.  

    Chris McMahon and I have started talking about what it would take to pull together a small conference for developer-tester-tester-developers focused specifically on programming interesting tests.  Not on tools.  Certainly not on commercial tools.  But on good programming practices/tips/techniques for tests.  (Our motto: "Show me the code!")

    We’ll see what happens.  We’re just noodling around with the idea for now…though if we get a bunch of emails along the lines of "Yes! Do it!" we might be persuaded to stop noodling and start planning…

  8. Phil Kirkham says:

    As a programmer turned tester I enjoyed the 3 articles you linked and really looking forward to reading what else you have to say

    Sadly where I work we’re not in a position to hire great testers – or any more testers full stop – so any advice on getting the testers we do have to up their game would be useful

    ( and Elisabeth – stop noodling and Yes, Do it  !!!)

  9. Phil said, "any advice on getting the testers we do have to up their game"

    What do you wish they knew how to do really well?

    BTW, Brian Marick’s book "Everyday Scripting with Ruby: For Teams, Testers, and You" is due out any day now.  I read a pre-release copy.  Great stuff.  http://www.amazon.com/exec/obidos/ASIN/0977616614

    Elisabeth

  10. I forgot to say… good post. It’s raised a subject close to my heart.

    More and more testers are finding that in order to compete, they need to develop their coding skills… Hopefully, your post will help spark even more discussion on the subject.

    Competing aside, the demand for tester-developers is growing simply due to necessity. Personally, I think it’s a good thing since it is opening up new career paths for many.

    All the best

    Antony

  11. Glenn Halstead says:

    Thanks for an interesting post Steve.

    I generally work between all three roles you identified depending on what my project requires at any given time.

    When working as a Test developer I find that I make much more effort to use an automated unit tests and automated build/test approach than many of the full time developers I work with.  The majority of developers I work with do not use automated unit tests or automated build tests.

    Glenn

  12. SteveRowe says:

    Thanks for the in depth response Antony.  I’m glad you got something out of the post and also had a lot to add.  I agree with some of what you said and disagree with other parts.  Let me try to address some of your points.

    1)  In my mind I was highlighting people but I suppose most of what I said could apply to roles as well.  Certainly when viewed as roles people can slot between them as the needs of the team vary.  It is important to have people on a team willing to switch up roles.  However, as I stated, trying to slot someone with a lot of development knowledge into a runtime role for a long period of time will usually cause discontent.

    2)  This is based on my experience but I suspect that most of what I say is applicable to all teams.  It doesn’t really matter to me whether the people doing the test development report to the dev manager or the test manager.  They might even be people that take on the role of a dev at other times.  I do have a disagreement with the concept of having the same person developing the code doing all of the test development.  I’m not against unit testing done by dev.  In fact I’m for it.  However, I don’t think unit testing is sufficient.  Perhaps I’ll post on that in more detail someday.

    3)  I had in mind more ad hoc testing when I wrote this.  Exploratory testing is most often done in a simplistic, ad hoc manner.  That isn’t to say it is restricted to that, but more often than not this is how it is carried out in the real world.  I didn’t mean to imply that it was only simplistic.  It can be done in great depth.

    4)  I didn’t have an explicit hierarchy in mind when I wrote this but I will admit to a bias and thus an implicit hierarchy which probably pervades my writing.  I don’t think that there is necessarily a valuation difference between the roles.  Certainly I state that each role is necessary on most teams.  However, I believe it requires more training to be a test developer than to be a scripter and more training to be a scripter than a runtime tester.  There are scripters and runtime testers who are highly specialized and highly skilled but the barrier to entry into those two roles is less than the the barrier to entry into test development.

      I certainly agree that a team of only test developers is not optimal.  Likewise a team of only runtime testers (or scripters) will not work as well as a balanced one either.  I also agree that a good test developer should be able to play the role of a good runtime tester.

  13. Bill Wert says:

    I am unconvinced that all roles must be filled by a test team; the context of the project is important. I don’t think a compiler back end test team has a lot of use for a runtime tester as you’ve described them.

  14. SteveRowe says:

    A compiler back end test team probably still has an automated suite of tests that have to be run every day.  The person responsible for making sure that they all run, that the machines are in the right shape, etc. is the person I see in the runtime role on that team.  You are right though.  It’s not a huge need and it might even be small enough that another person on the team could absorb it.  This falls under "Each project will require a different mix of people."