“Experience Quality” and the senior tester


I’ve spent some time over the last few weeks investigating the role of test and the “experience” of quality software. Between the Customer Experience Improvement Program, Windows Error Reporting, Send a Smile, and connect, there is ample opportunity to get relevant customer data to help in this effort.

I was thinking about the role of test in processing this data (design guys LOVE the CEIP data, and developers…ummm…tolerate the WER data). The main idea is that customers like it when we fix the stuff causing them problems, and these programs give us a chance to do better. Anyway, as I was thinking about this, I almost forgot that I already described the role of test in dealing with this data almost a year ago when we created the tester personas on http://testercenter.

Here are some excerpts from “Alecha – the product line customer advocate”

Alecha is skilled at knowing the needs and attributes of her customers.  She has a knack for seeing things from the customer point of view, but also works closely with program management and marketing to verify her assumptions and clear up ambiguity. She analyzes data from various tools that track customer data and makes sure that the testing effort focuses primarily on customer scenarios and customer pain points. She is also one of the key drivers in determining how these tools are used for her product.

She is well versed in customer focused design techniques and works with the appropriate cross discipline owners to apply these techniques across the product. She is concerned with all aspects of the customer experience including usability, reliability, security, performance and general product effectiveness. Alecha spends a lot of time working with other team members to ensure everyone maintains that customer connection that enables her team to create a quality product that the customer wants and needs.

Doesn’t Alecha sound likes she kicks butt? The problem is, that I don’t think we have enough Alecha’s in test. We have testers who can analyze data, and we have testers that recognize customer pain points, but we have relatively few that can put it all together and make a significant impact.

Alecha – where are you?


Comments (5)

  1. Daryl Welsh says:

    Alan,

    You asked ‘Where did the Alecha’s go?’  I’ll tell you – they pretty much disappeared when the STE (Software Test Engineer) discipline was eliminated across the company.

    People are driven by two primary factors whether they realize it or not – by how they are measured and by how they are rewarded.  These two factors drive what testers focus their energies on, and what their management teams decide to invest in (or sell to their own management).  How are the testers/qa engineers at Microsoft being measured or rewarded?  Figure the answer to this question out and you’ll help to understand why we don’t have more Alecha’s at the company.

    The majority of commitments I’ve seen for SDETs around the company are hyper-focused on empirical issues (code coverage percentages, percentages of test cases automated, cycle times per test pass, etc.) SDETs thrive on this, and can spend thousands of hours per milestone writing insane amounts of automation to verify that what the developer wrote lines up with what the PM specified and can be repeated a gazillion times inside of some unwieldy harness.  How can this be a bad thing?  Code coverage is high, the vast majority of test cases are automated, and we can run thousands of them each night on every build.  This is the panacea we’ve all been striving for isn’t it?

    Unfortunately no, as the focus on delighting our customer has been lost.  Nobody is being measured or reward for stepping back to question whether the features/idea makes sense or will be something the customer actually wants or can use.  A really good ‘STE’ is worth their weight in gold because they have an amazing ability to string a bunch of conceptual issues together in the spec stage to find problems before teams go too far down a rabbit hole.  More importantly they are able to wear the clothes of the customers and find the seams where things did not make sense or fit together correctly during the development cycle.

    We no longer place value for the role that Alecha represented at the company and consequently we have very few of them around anymore.  This is extremely bad for us long term.  Think about some of our very large projects (Vista for example) that were tested extraordinarily well in the empirical sense I mentioned above, but have been received unfavorably by the actual users of the product.

    With many of our products moving directly into the consumer space and competing against Apple, and Sony and Google, the need for testers who can harness the Alecha persona will be even more important than it is today.  Unfortunately we have little framework to identify, develop and reward them; so we are guaranteed to continue to stumble here.  The ones who show some aptitude will look at the CSP levels for the Test Discipline and quickly jump ship to a discipline that values the focus on delighting the customer more (and that’s assuming they decide to stay at Microsoft.)

    The more focused our system is around rewarding and promoting the technical aspects of being a tester at Microsoft the more assurance you can have that we will be shipping products that are highly automated but be extremely unfriendly to our consumers.

  2. Alan Page says:

    Thanks Daryl – every point you make is on the mark. One clarification I want to make is that I don’t think all STE type folks can be Alechas – but Alecha is the role they should aspire to.

    It’s one thing to be that tester who is great at flushing out customer issues, but the role needs to scale. I want to see more great testers use customer data to get even better at testing – then find a way to help the rest of their team get better at testing too. One of the things I always tell senior testers is that their job is to make their team better – for tne "non-coder tester", this seems like the logical way to do it.

    I could do an entire post on mismanaged rewards systems – but I won’t.  Yet.

  3. gstaneff says:

    I started working with an Alecha a few months ago; always has SQM or WER data to back up any given user scenario and has a handle on how we use the product in addition to how we happened to have designed the product.  This person isn’t in Test anymore; this person switched to Development a few years back.  

    There is a great deal of organizational inertia standing between even the current CSPs and day to day expectations of a given role.  Skills and characteristics from the Alecha role are called out in the Test CSPs, jumping to a different role may have more to do with differences in enforcement and application of the employee review policy in a given discipline rather than any particular detail of that policy.  The CSPs weren’t around yet when many of the Alechas were jumping to other disciplines, afterall.  Perhaps more good could be done by calling out example testers in each role we want to encourage so managers can stop trying to shoe-horn senior test into the same mold.

    Perhaps a better solution is to look at where an Alecha has to spend resources to be effective.  Alecha needs customer data to function, that usually means working with Service Packs and Legacy Apps because that is where the data exists.  The cool work is typically in V.Next product development and in V.Next Alecha doesn’t have much empirical data to draw from as well as a pile of garden variety feature testing to get out of the way before anything else can be considered (does a budding Alecha get to analyze Beta 1 feedback directly or attempt to catch up with whatever just got checked in to meet code complete?).  A developing Alecha may, then, be drawn away from the ‘front line’ of development in order to pursue activities that Alecha finds more compelling.  If one buys into the promotional advantage of serving on the front lines you get into a situation where Alechas end up in situations where it takes them longer to reach senior levels of their discipline, increasing the odds they’ll jump displines or leave the company before reaching the testing end-game.  Is it possible, then, that we’ve just shifted specific focuses to different parts of the company and need to encourage diversification not just by experience but by focus?

  4. alan says:

    I disagree that Alecha works primarily in sustained engineering. I don’t want Alecha there – I want Alecha working on the mainline product. I want her involved in the super cool innovative product, and I want her examining all of the data from beta and partners to influence product and test design in (almost) real time.

    Alecha has the potential to be one of the biggest influencers on a product – or even a division. She needs to be center stage.

Skip to main content