Testing And Customers: Getting Answers from Customers

Bwill asks “Do you have any favorite techniques to obtain answers to your questions
about customers? How do you convince yourself that your answers to those questions
are ‘good enough’?”

My favorite techniques include a variety of data gathering tactics that include both
1st, 2nd and 3rd hand techniques. 

1st Hand includes my own understanding of the feature, how I want to work with and
interact with the feature.  Here I make the assumption that I’m a potential customer
of the product and have some interest in it or else I wouldn’t choose to work on improving
it.  Although its important to recognize that your use/perceptions are skewed
from your personal involvement in the product itself you can use your own assumptions
as starting points for further investigation.  Your own opinions however could
prove to be weighted least in the end as you start to understand work-styles and perceptions
that fall outside your own patterns.

My 2nd hand data gathering involves anything that creates a quality connection between
myself and real (intended target) potential customers that don’t work first hand with
the product being tested.  I read newsgroups about my own product as well as
those for competing products.  People that are loyal to competing products are
more willing than you’d imagine sharing with you what key scenarios you might be missing. 
I will create my own usage surveys targeted towards the feature in question. 
These, for example, led to a bunch of changes to the design of the VS Start Page. 
These surveys, in addition to gathering lots of raw data, served as great conversation
starters with customers that care about your product.  In these threads you can
dig deeper than you intended with the broad survey to see why people answered one
way or another.  Eventually you form, what some people would call, your customer
counsel of close key influencers you can bounce ideas off of.  A lot of these
people have a direct connection to me through IM.

3rd hand is information about customers that may be spoon-fed to you through designers,
usability engineers, co-workers, the media, etc.

I find the 2nd hand information to be the most valuable of the three.  This is
because I can use these techniques to be my verification method for information gathered
through 1st or 3rd hand approaches. 

Good Question,

Comments (4)

  1. Bruce says:

    Great ideas, thanks! I especially like that second one; I’ll have to give it a try.

  2. J.P. says:

    [The following thoughts are my own, and don’t necessarily reflect those of my employer]

    Our team had a "Joint Development Program" where we invited 14 or so companies to experience the development process with us and run early release versions on their side. Meanwhile while we would buid though, we could talk to our customers directly and ask them what they needed, what they wanted, and my favorite, what they thought that they were getting. This of course always helped in really getting the frame of mind of the customer. Of course this system is not perfect (really what system is?), since there were many companies that expected preferential treatment when it came to their (very) specific feature requests, and there were also companies that did not understand the concept or quality for that matter in a beta version, and did not find acceptable when there were bugs…..and then refused to upgrade later since they said they had finished their rollout.

    Even better was that through this program, I got the opportunity to even fly to visit a customer site, and talk with an administrator as he tried installing our software. It was enormously helpful to understand the frame of mind that he had (which was much different than I would have imagined) during this process. It was also helpful as a part of this trip to see all of the battles that have to be fought inside a company, even after they get our software.

    Anyways, I suppose my who comment is that, the source I find most usefull, is our customer directly. Then again, I suppose it might work differently when our customer base is large enterprises vs. any and every developer with the money to shell out for an IDE….