My Thoughts on the MSDN Feedback Center



Overall I know it’s a great thing and there are some issues that need to be improved over time for it to be truly successful.  I may be a little close to the project since my co-workers are responsible for the Feedback Center, but I think I’m still allowed to have opinions and air them in this space. 🙂 Recently I was asked some specific questions that do a good job framing my opinions on this project.




Do you like this kind of transparency?
I love this transparency and think it’s only the first step towards a more transparent future of developer products at Microsoft.  Why do I like it so much? 




In case it’s not clear… when you enter a bug through the feedback center it is also opened directly in our internal bug tracking solution alongside of bugs being found by our own testers.  Internally we can’t escape these bugs since they are in the database we scrutinize every day on our way to shipping the product.  You get to see when, how, and who fixes your reported issues and you get notified of its progress along the way.  We also require that team members provide quality responses to every reported issue to further encourage accountability and close the feedback loop.




Historically we’ve had crash reports and statistics that helped us know which crashes we should focus on based on the number of reports.  Triaging non-crashing issues has always led to debates over how we think customers might feel about an issue versus what rating and how many customers might actually be affected by an issue.   This will eventually help us triage bugs much more efficiently and gain an even better customer understanding of the impact of bugs we find internally. 




We will only get more transparent.  Soon you may be able to see our actual internal bug counts and even see internal bugs alongside customer reported issues.  When we are in a world where you might have access to any build of visual studio this will be even more important.  




How does this compare with the closed, semi-private, bug reporting on beta.microsoft.com ?
I’ve used both and the MSDN feedback center is much better simply because you can search, see, and vote on bugs reported by other users.  You also get to add your comments and workarounds to issues as well.  With betaplace (beta.microsoft.com) you only get to see issues that you report.




Microsoft products will still have private beta’s for a long time so there will always be a need for a non-public site, but there are teams talking about creating closed group instances of the Ladybug application moving forward because it is seen as a step up from existing solutions.   So you can imagine that beta.microsoft.com could eventually be leveraging the Ladybug solution. 




What do you think about the quality of submitted bugs?
It could be improved.  I’ve spent a lot of time in my career here triaging bugs as a Test Lead and I’ve also been involved in triaging the bugs from the Feedback Center.  In general feedback center bugs require more “translation” time to understand what customers are trying to report and what they may have been trying to do when they encountered the bug.  Some teams have suggested that the triage time per bug is up to 10 times as much when compared to bugs reported internally.  I’m not sure I buy that number (given the statistics you’ll see below), but it does take more time.  The extra time required is probably because of the following reasons:





  • Lack of Additional Background Info: There is a set of information that each team feels should be collected when a bug is logged on their feature. When this information is missing the issue requires an extra round trip with the customer to attain it.  We would rather not burden every customer with these specific requests, since we want the bug reporting to remain easy, but we may eventually run an active X control that automatically fills out portions of a longer form for users when they attempt to file a specific bug. This information would include what profile settings you chose at first launch, what other versions of VS you might have installed, what type of project you were working on, etc.
  • Lack of Clear Repro Steps: We should provide you guys with better bug report examples.  I’ve seen a bugs that look like “Sometimes I seem to get an error message that says Foo” without explaining what they were doing that led to the error message.  The feedback center is a community in its infancy and the unspoken rules/expectations of a community have not been fully explored yet.  Over time, with better examples and additional practice submitting bugs it will improve.  The quality here is not unlike the quality of bugs you might get from a new-hire testing the product and, like the new-hire, I’m sure it will improve over time with a little guidance. 
  • Lack of Expected Behavior Reports: Good bugs should not only contain the description of the behavior experienced, but also the expected behavior from your perspective.  Instead of just saying “When I click X, FOO occurs” it can be helpful to say “When I click X, FOO occurs, but I was expecting BAR because of Y”.   I also believe this will get better over time. 
  • Lack of Good Microsoft Responses: Yes, this something we can improve on that will help quality.  A running joke on our test team was that we should have all been filing our bugs through beta-place or Ladybug as customers because we’d get better responses about why issues are “Won’t Fix” or “By Design”.  A tester internally will often just see these two word responses and it would be up to the tester to push for a better explanation if they felt the bug was important.  Strangely enough we felt we should encourage people to treat the community members with a little more dignity than our own employees. 

    • Providing clear explanations about bug resolutions is something that is new for a lot of people.  Speaking as a former tester I’ll say that it is about time.  Currently we’ve had some rough spots and bits of information that are Lost in Translation.  Eventually we will adapt by improving our accountability, learning better “customer-speak”, and finding more common ground within this community.  This is new for us too.  We are investigating providing a direct “Contact the Developer” form for MSDN Feedback center bugs so you don’t get the rather blind “Resolved by Microsoft” text.  You would then see who resolved it any be able to contact them directly.




What do the results look like so far?
So we’re listening and guaranteeing a response.  IMO this is only part of the solution.  In the end we’ll also be judged by what we do with what we hear.  It’s unfortunate that this feedback mechanism was opened so late in the cycle of Visual Studio 2005 because a lot of this great feedback will only start to get leveraged for the next version.  This is especially true for suggestions that would require more than a trivial amount of new development work.  In the mean time I’ll share some raw statistics since late June when the feedback center became public that I took a snapshot of today. 




For comparison I thought it might be interesting to look how these percentages compare to internal bugs and suggestions opened during the same time period.  For these, unfortunately, I don’t have a good measure of time to response so the raw % will have to do.  Also, just assume the total numbers for internally reported issues are larger. 🙂




MSDN Feedback Center Bug Stats




  • Total # Opened So Far                1679
  • Responded to                             75%
  • Avg # of Days to First Response  6 Days
  • Still Active                                   50%
  • Resolved Fixed                           24%
  • Resolved No Repro                     9%
  • Resolved By Design                    7%
  • Resolved Postponed                   6%
  • Resolved Won’t Fix                    4%




Internal Bug %’s for the same time period




  • Still Active                                   55%
  • Resolved Fixed                           20%
  • Resolved By Design                    8%
  • Resolved Won’t Fix                    5%
  • Resolved No Repro                     5%
  • Resolved Duplicate                      6%*
  • Resolved Postponed                   1%





MSDN Feedback Center Suggestion Stats




  • Total # Opened So Far                1174
  • Responded to                               79%
  • Avg # of Days to First Response  6 Days
  • Still Active                                   34%
  • Resolved Postponed                    27%
  • Resolved Fixed                           16%
  • Resolved Won’t Fix                    12%
  • Resolved By Design                    9%
  • Resolved No Repro                    2%





Internal Suggestion %’s for the same time period




  • Still Active                               52%
  • Resolved Won’t Fix                 17%
  • Resolved Fixed                        11%
  • Resolved Postponed                10%
  • Resolved By Design                 5%
  • Resolved Duplicate                   5%*
  • Resolved No Repro                  <1%




*If an MSDN Feedback bug is considered a duplicate it is associated with the primary internal bug so customer bugs can only be duplicates of other customer bugs.  That number has been very small since customers can “+1” an existing bug rather than opening new entries.  So the % duplicate of MSDN Feedback bugs was not worth reporting. 




What’s important to note is that the jury is still out on around half of these issues that are still marked as active.  Because of this the resolution percentages are very likely to change over time.

What I learned through this is that, so far, we’ve actually fixed a higher percentage of customer reported issues than internally reported ones!  It will be interesting to see if this keeps up over time.




Now that you’ve read this far I’d like to know some things:




  1. Are these numbers interesting to you guys?
  2. Would you like to see more?
  3. If so, what other information would you like to see?
  4. How would you like to see top contributors to the Feedback Center rewarded by Microsoft?



 

Comments (18)

  1. AT says:

    Thanks Josh for reply.

    I’ve found % numbers very interesting and somewhat unexpected for me.

    I’ve expected that Internal Bug % Duplicate be about 30-40%. Why? Because there was no way for users on private betas to check if bug was already submitted.

    What this number show? These numbers show that either number of private beta testers too small or number of bugs in product is really big. Only under those conditions probability of duplicate reports will be so small.

    Can you confirm this? Or there is something wrong with my logic? For example how bug reports reported against another duplicate but fixed bug will be resolved?

    Taking in account current repro steps.

    Is there any way to promote submission of automation test cases from LadyBug users?

    I’ve proposed this several months ago:

    http://blogs.msdn.com/fivetestersfromvc/archive/2004/03/25/96488.aspx

    Possibly some kind of contest with small rewards or simply a best-practice and HOW-TO ("How to record automated test cases for VS.NET IDE using macros", "How to create tiny source code example for ICE", "How to create sample code for API changes suggestion", etc.. )

    I’ve submitted FDBK12274 ( http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=FDBK12274 ) suggestion a few days ago about lame repro steps 🙁

    BTW, What do you think – it was correct decision to move as many as possible products to Product Feedback center? Or it was better to test-drive it initially for single product (for example ASP.NET ;o) and only after start using it for all others?

    P.S> How about numbers you collected will be possible to regenerate on ProductFeedback site ?

    Select date/product/version/area and see numbers of bug reports filled/resolved.

  2. The numbers themselves were ok; I think your analysis was more important. I’d like seeing more of this sort of thinking aloud. Ladybug is a good move, and it’s interesting to hear how it affects the way you work.

    I don’t think you should start systematically rewarding people at least yet; that leads to unhealthly competition on a field where the "top" is not very easily measurable (quality vs. quantity etc.). If somebody is an excellent reporter, invite him to a job interview – rumors will spread and people will consider that rewarding, but not in a way that would lead to competition 😉

  3. Great post Josh! It’s good to read that even though we on the outside don’t have a complete view (yet?) of the bug database, at least bugs filed through the feedback center are treated as first class bugs internally.

    The numbers I’m a bit surprised about though are the No Repro percentages: 9% on the feedback center and half of that on internal bugs. 1-in-11 and 1-in-20 is quite a lot considering the absolute number of bugs submitted, which means that there are quite some bugs that, although found, just linger because they can’t be reproduced.

    I understand that developers can’t spend their time looking for a fix to something they can’t reproduce, but still, as a developer creating mostly applications for end-users in the past years, I would say that it’s very common for me to go looking for a bug that’s been reported that I can’t reproduce. In fact, since the feedback center currently only allows bugs to be reported on developer tools, I would say that the reports you’re getting now are going to be the best you’ll get across the full line of Microsoft products 🙂

    For instance, I remember a bug about not being able to get your mouse pointer out of a tabcontrol in the designer under certain, unfortunately unknown, circumstances (http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=44fc0b2e-8cbd-49f0-a3a1-a96d4f72764d). It sucks that there are no complete repro steps, but at least the behaviour is clear and this could at least lead to an investigation of for instance the code governing these movements. Perhaps this was done, but the responses make it seem like no repro = no action, which would never fly with any of my end users 🙂

    And regarding your questions at the end: the numbers are very interesting and I’d like to see more. Personally I’d be interested in seeing the areas where a lot of bugs occur, for instance which controls/BCL namespaces/etc. although that might be a little hairy internally for you to publish. Oh and I agree with Jouni above: don’t reward people directly in top lists or whatever for submitting/validating/etc. because it’ll only lead to people submitting crap or validating everything they see just to get in the top lists.

  4. >when you enter a bug through the feedback center it is also opened directly in our internal bug tracking solution

    This is not as good as it sounds. This way more pressure will come to close these bugs as soon as possible to get them off the internal bug tracking solution (which is in most companies primary benchmark of the state the product is in).

    A bit relaxed attitude could be better, and then devs would not rush to close so many items without proper explanation.

    But I see you have good ideas about how to cure this – make sure every tester/dev goes through at least one bug using ONLY ladybug. Then hopefully we will get more meaningful explanations on closed bugs.

    That said, you are going in the right direction – please, keep up the good work.

  5. You could start publishing the email address of the employee who resolved a bug or suggestion (at least to the person who submitted it). That way, the customer could continue the conversation, as internal testers can.

    I’d be interested to know the browsers people are using to access the feedback center (keeping in mind that you can’t submit a bug in Firefox).

    Also, things like number of bugs in each category, average and distribution of importance of bugs.

  6. I don’t know *what* the reward for top contributors should be but I have a pretty good idea *how* it should be given: without fanfare. Quietly mail them some thank-you gift and be done with it. Keep it unofficial.

    This would prevent useless competition and attempts at "gaming" the system.

  7. Cade Perkins says:

    Yes, this information is very interesting. As for what information, I say just keep posting whatever thoughts you have… don’t make it too formal.

    How about a free version of the MS software package for the most helpful contributors, especially the next version with the bugs fixed. This would inspire better bug reporting, especially since the reward would be directly related to the effort. ("Fake" rewards would only cheapen the feedback.)

  8. josh says:

    AT: I don’t think I explained the duplicate logic well. The small number of reported duplicates is due to the fact that when a customer bug is reported as a duplicate the customers don’t see this and we just move their association to the original bug that we found internally. So if 100% of the customer reported issues had already been found the customers wouldn’t see one duplicate because we let them track the original bugs so they know when their issue is fixed. If we did it another way there would be no way for customers to know if their “duplicate” was ever fixed or not.

    You can attach files to ladybug issues to demonstrate the bug, but we aren’t yet at the stage where we are set up to allow customers to write our tests. Not that you couldn’t, but I’d just like to win the bug reporting war first.

    We actually did test drive just VB with just MVP’s for a while before the larger public beta. We’ve had requests from several non-VS teams to add their products, but we consider the service to be still in beta as we make adjustments. The way the features are spread out amongst all the different language teams it would have been hard to go public with just Asp (for example) and not have the VB or C# teams as well since they also deliver some key components.

    We have been thinking about posting numbers like this onto the site. You could almost think of this post as a way of exploring what goes over well. We want to give you guys access to more stats, but each chart or calculation required is more dev time and we want to get the system stable and deliver on the core value proposition of responses and bug reporting before deciding what new code we want to add.

  9. josh says:

    Jouni: WRT job interviews. I’ve actually been wanting to see this happen for a while. In a world where so much of our process is being opened up, why not hire the people who work with it the best. I’m not sure that the hiring managers are all thinking the same way, but it won’t be long I’m sure.

    I hear what you are saying about rewards. It’s a common debate internally as well about how to reward testers for their bug numbers. We do, however, want to do something to acknowledge that the community is doing great work here (even if we don’t hire them all).

  10. josh says:

    Jeroen: Sometimes the No-Repro is a good thing. The bug may have been caused by something unrelated and no longer reproduces with the current builds. Rather than believe that they were fixed with no action we generally call these No-Repro to have the tester who owns the feature re-verify that this was the case. So there is always action on these bugs. Generally the triage teams will try once, and then the tester who owns the area will do a deeper investigation before closing out the bug. The high percentage reported on the feedback center is most likely due to the latency of builds customers receive.

    WRT rewards. We would want to scope the reward down to people who submit bugs that we fix. We’ve also been thinking about different categories so its not just a volume game like.. most interesting bug, strangest behavior, etc.

    WRT Where bugs occur: We’ve been thinking about this as well. The tricky part is the context that also shows how much code is written or the complexity of the area. There are very few bugs found in the DTE when compared to the IDE, but when there are bugs found they are more serious when compared to a bug about a misplaced pixel in a custom drawing.

  11. josh says:

    Drazen: We do want to deal with the bugs quickly. In the past our feedback mechanisms (MSWish, etc) have been far to disconnected from the development cycle and this has led to a HUGE amount of feedback being ignored. We may have turned the dial the complete other direction now, but you can see from the stats that we are no longer ignoring it.

    One important note is that at every turn (as we’ve scaled the system larger and larger) our estimates for the incoming rate from customers has been way too high. There are many who felt we would have been swamped by now, but it’s been reasonable. I also have an unproven belief that the search before posting model is working well. I could be wrong though.

  12. josh says:

    David: E-mail publishing is under consideration. Thanks for your feedback on things you’d like to see.

    Stephane: I also think some “offline” thank-you system is a good idea. Everyone like being rewarded differently.

    Cade: Thanks for your feedback. Software products are interesting, but a good number of customers already have MSDN Universal so the benefit of getting another copy of VS would go down for those people. It could be an option for those that don’t though.

  13. I have been quite impressed by this new bug tracking system (I would like to see some of the products I work on go down a similar route). I added my feature request (http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=78fbe39f-954d-494f-b35c-9a7cad5dd245) which was looked at soon after I reported it and although it was marked as ‘Won’t Fix’ I am glad someone has taken the time to respond (even if I don’t entirely agree with their reply I am sure they are right).

  14. josh says:

    Jonathan: Thanks for your additional comments and good example.

  15. AT says:

    josh:

    Duplicates vs. resolving all bugs with Fixed. Ohh .. I see.. I’ve assumed this in my comments.

    Can we get real numbers ?

    What is average number of reports per unique master bug in old semi-private beta.microsoft.com reporting site ?

    BTW, On WindowsBeta there was additional information for bug reports resolved as duplicate. Each bug report additionaly listed master bug resolution. Thus instead of 100 duplicates resolved as Fixed – you will have 1 master bug report as Fixes and 99 reports resolved as Duplicate with ability to see master report status.

    Just a FYI, I’ve participated in initial test-drive. I’ve found ProductFeedback not usable and proposed a few improvements. But unfortunaly they was not all implemented before June public anouncement.

    I believe that ProductFeedback is on long way to collecting productive feedback. I’ve not expected such a little and feature-less site created by Microsoft for world-wide use.

  16. Josh Ledgard says:

    Not sure what you want mean by "real" numbers. A customer bug is a "valid/real" bug regardless of whether it already existed in our database because of the build latency. If you mean, how many of these we actually end up re-directing to other internal bugs… it is probably around 12-15%.

    Regarding the windowsBeta implementation. I think this is an additional level of complexity we don’t want to add in the public space. It’s better to just re-associate the bug for the customer so they don’t have to worry about anything else.

    Regarding your feelings of the site being unusable… Without specifics i can’t explain the trade-offs, but I know there are a LOT of people who think this is a HUGE improvement compared to never hearing back from your feedback report.