Product Feedback Systems

Several people on the Windows Internet Explorer team have written blog posts about our feedback mechanisms (our use of automated telemetry in Windows, how to write a great bug, how to submit bugs, etc) for IE9.  After looking at the many similarities and obvious differences between manual feedback systems and projects, we decided to use Microsoft Connect for IE9 and eliminate the invitation requirement for filing bugs.  In this post, I want to take a step back and talk about how we made that decision.

Every Internet Explorer user is a Windows customer.  Listening to customer feedback is vital to the success of any business.  As communication methods have evolved, how a business listens to its customers has changed as well.  Our customers have always been at the center of how we envision and design IE but the way we listen has evolved both with technology improvements and with the way our customers communicate with each other.

Our Internet Explorer 8 beta customers asked us to look at different options for how we listen.  We took their feedback to heart and looked deeply into all aspects of our feedback systems as well as those used by other software companies and in other industries.  We also looked into research regarding the effectiveness and efficiency of various ways of listening and responding to customers.  Most importantly, we listened to real customers, real enterprises, and real developers.

Each time we start a new project we begin by stepping back and looking across the feedback from the previous project and the engineering process by which we collected it.  We challenge all assumptions, especially those based on speed, size, or connectivity.  We also talk with people to see what they like and dislike about our competitors’ product and engineering process choices to see if Microsoft’s customers might benefit from something similar or more advanced.

One of the assumptions most software organizations appear to take for granted is the need for beta releases.  In fact, some companies have even taken this assumption well beyond its commonly understood meaning. So we asked ourselves, “Should we do betas and, if so, how many?  Should we do nightly builds?  Should we continue to use Microsoft Connect or move to something else?”

A group of us sat down and thought through the customer benefits and drawbacks of various models.  Ultimately, there are two main benefits to public releases:

  1. Feedback – customers reporting bugs on what is not working correctly
  2. Readiness – testing sites with the new browser, adding newly possible features to pages, preparing product support, writing documentation and books, adjusting tooling, etc.

There is a continuum of feedback models that ranges from completely closed to completely open.  Examples of completely closed betas include new PC models, cars, toasters, etc.  Examples of completely open models include some academic open source projects, school PTA projects, etc.  We’ve chosen something near the middle so we can get broad feedback on quality events but not completely anonymous or unstructured either because we believe in responsible engineering.

From our point of view, the most important thing about working on a consumer technology like a browser is respecting people’s scarce time and energy.  You can do this while still achieving your goals of getting everybody ready (from web developers to support professionals to corporations) for product launch and getting feedback on quality issues that only surface in unique configurations.  I want to call out a few important aspects about feedback systems:

  • Having a distinct start and end to the pre-release period is important.  Without this, people would gamble their business or personal browsing on the product always behaving in a predictable way.  Having beta customers running betas forever also neither respects their valuable time nor focuses that time on finding and reporting real defects. 
  • How often you send out builds depends heavily on your own ability to find defects.  We looked at the pros and cons of releasing “nightly builds” to beta testers.  Because we have a professional testing organization and a broad range of hardware on which we test, we find defects daily (many prior to checkin), fix those, and look for more.  When the IE team starts running low on new types of bugs to find, we get thousands of other professional Microsoft testers to start using IE9 to report any defects they find with IE9 as their daily default browser.  We call these folks “Self-host Testers” and they find an additional set of interesting bugs.  Again, when it appears that this group is running low on new bugs, we expand the scope even further and publish things like IE9 betas.  The risk of putting out daily builds across all audiences is that multiple beta testers waste their valuable time finding the same bug because they’re simply testing incomplete code. 
  • There is a decreasing effectiveness in the feedback as you expand the group of testers.  We looked at IE8 bug reports that were actionable versus those without enough data on which to make an effective code change to determine a signal vs. noise ratio.  For IE8, the signal to noise ratio in our bug database went from 3.1:1 to 2.6:1 when the code moved from Development to Test.  It dropped to 1.5:1 when other Microsoft testers started using it and reporting bugs.  It dropped to 0.73:1 when the rest of pre-release Windows users inside of Microsoft used it.  Finally, it dropped to just 0.64:1 across all the IE8 beta testers.  Closely related to the actual effectiveness of the bug reports is each group’s ability to produce a volume of effective bug reports.  The IE engineering teams produced more than five times the number of bug reports at six times the effectiveness of the beta program.  In other words, the IE engineering team was 30 times more efficient at finding actionable bugs than the IE beta testers in-aggregate over the same period of time.  Make no mistake though, the IE beta testers filed some great bug reports and we and Microsoft’s customers are glad they did!  For instance, bugs found on websites which require account login, special credentials such as smart cards, regionally-locked IP addresses, etc. are very important to us.
  • We really want to know if something doesn’t work as expected.  If you think there’s something wrong with a feature, please file a bug so we can make adjustments before IE is done.  We will manage that in a way that respects everyone’s time by helping people target their time and bug reports on reporting actual defects we haven’t yet found at Microsoft.  Bug reports come in all flavors and sizes.  They range from “I want feature x” to “Please add a whole new mode so I can use feature y this new way” to I tried feature x in the way you said it should work, but it doesn’t work that way.  Reports similar to the last example are the most useful.  When features don’t work like they should, we definitely want to know it

We also looked at various feedback tools when we started the project.  These included things like Bugzilla, Mantis, Launchpad, etc.  We compared them to Microsoft Connect, which is what nearly every Microsoft product uses to get beta feedback from customers.  It turns out that these tools are all amazingly similar.

We looked more deeply at Bugzilla and projects using it because it’s a tool that at least two other browser vendors use today.  We looked at the tool itself and how it handles bug workflow both from a beta site perspective and a product engineering perspective.  We found some really interesting similarities and differences:

  1. All of them require some kind of login.  Mozilla’s Bugzilla needs an email address to log-in. WebKit’s needs an email to log-in.  Both of them warn you of being spammed if you actually use your primary email account so you should get a new email account if you want to file bugs.  By contrast, Microsoft Connect uses a Windows Live account with either your current email account or a free new one.  In either case, your email address isn’t visible to people or web-bots mining the site to get email addresses for spamming campaigns.  We actually went one step further and require LiveID login to query the bugs, further reducing the risk to our beta users.  Connect also works for multiple Microsoft beta projects with a single account.
  2. They all handle entering issues by area, feature, or symptom.  All have some bug entry form where you fill out the details, including repro steps, title, etc.  They provide a way for you to give it a priority.  They let you provide great feedback (Bugzilla, Connect) or poor feedback (Bugzilla, Connect).
  3. Bug management differs considerably across projects though.  With IE8, we took action on every issue that was reported and we staffed our beta with Microsoft professional product support engineers so they could learn how to support the product while we built it.  By contrast, if you filed an issue with Firefox 3.5, it would sit at Unconfirmed until someone with “Can Confirm” privileges saw it, which may take years.  If a privileged person does happen to see it, it may eventually be seen by the engineers. 
  4. Bug closure also differs across Bugzilla and Connect.  For IE8, we took action on every bug and drove it to closure.  By contrast, as of 4/26/10, Mozilla’s Bugzilla had 12,779 unconfirmed bugs.  Webkit’s had 2,616 unconfirmed bugs across all versions but very few bugs against older versions.  Just like unemployment will never be 0%, it’s important to note that in-development projects will always have a non-zero number of bugs just as a normal part of bug management.  Each project team needs to understand and manage their inventory of bug reports to ensure a reasonable response time for their project testers.  For a project as complex as IE, that can be as large as a couple thousand issues active at any point during the project.

We made some engineering decisions on how to get the best possible feedback from our beta customers after doing all the research into different release cadences, feedback models, tools, and bug management processes. We want to respect our customers’ time and energy so we’re going to distribute more focused Platform Preview builds when there are new platform features ready for people to test drive. 

Thanks for all the great feedback in IE8. We’re looking forward to building a great IE9 release!

Jason Upton
Test Manager – Internet Explorer

Comments (27)

  1. Anonymous says:

    Your post doesn’t actually seem to provide a rationale for why Connect-the-tool was chosen; the model you seem to be criticizing on Bugzilla is a people process that you could have fixed, using your professional support professional professionals.

  2. Anonymous says:

    You’ll notice that none of the 5 files are preview-able/downloadable.

  3. Typhoon87 says:

    This is a very interesting post. You do explain why you chose not to opt for nighly builds which is understanable. But what about something in between nightly and two months like maybe two (2) releases a month or maybe a weekly build?

    It seems like instead of choosing nighltys wich is fine, you just went a little to far to the other side of the spectrum.

  4. gkeramidas says:

    i filed a bug on rss feeds in ie8 during the windows 7 beta, over a year ago, and it’s still is not fixed. i was told by an engineer that they thought a fix was checked in back in september. it is still not released.

    every time i wake my pc up from sleep mode, i have to run:

    msfeedssync disable

    msfeedssync enable

    to get my feeds to update. if i don’t run this, my feeds will not update for over 24 hours. and no, it’s not my settings. i have installed windows 7 multiple times on multiple pc’s and they all have the same problem.

    since the issue was acknowledged by an engineer, i know it’s not me.

  5. GregM says:

    I find it interesting that Mozilla and WebKit each have more unconfirmed reports than IE 8 has of overall reports (2437+65).  Also, all of the first 100 bug reports were closed as "Not Reproducible", "By Design", "Won’t Fix", or "Postponed".  Are there any stats on how may of the 2437 reports were actually closed as fixed?  There doesn’t seem to be a way to filter by closure reason in Connect.  

    Even more interesting is that all of the Suggestions were closed as "Postponed".  (There is one Suggestion reported fixed, but it was really a bug reported in a newsgroup, and had already been fixed.)  Is this a result of the quality of the suggestions, or from not having the feedback area open early enough to actually be able to do anything with the suggestions?

  6. Mitch 74 says:

    @Jason: a bug may sit for years on Bugzilla. But, at least, when you wake it up, you don’t need to completely recreate the bug submission.

    Asking people that saw their bugs "solved" as "we acknowledge the bug but won’t fix it" and then needing to resubmit AGAIN for the next version is probably the part that pissed off most bug reporters in IE 7 and 8.

    Another part that is uncomfortable is the lack of attachments management: screen caps, code snippets etc. are often used to document a bug, but Connect requires that we include the code directly in the comment.

    Next, bug interdependence is absent from Connect: it’d help if we could know the bugs our issue depends on (that definitely helps when trying to refine duplicates) or that depend on it.

    At least in Bugzilla, when a bug is closed, you know it’s done for – except in the "WONTFIX" case, and even then it could be reopened if the conditions for "WONTFIX" aren’t valid any more.

    What I’d like to say is that, in fact, Bugzilla allows better communication as it provides many information on a bug’s progress – that Connect doesn’t.

  7. Sven says:

    Your post doesn’t actually seem to provide a rationale for why Connect-the-tool was chosen; the model you seem to be criticizing on Bugzilla is a people process that you could have fixed, using your professional support professional professionals.

    Still, it’s nice to see someone from IE throwing stones at the competition. Bravo.

  8. TheCycoONE says:

    Other peoples points are quite valid; the number of unconfirmed vs fixed vs won’t fix etc. is up to the methodology of the team and has nothing to do with Connect, Bugzilla, Mantis, Fog Bugz, SpiraTeam or whatever other bug tracking system you might use.

    Microsoft seems to treat each IE release as distinct with distinct bugs and closes any that won’t be fixed in the current in progress version.  While this keeps the bug list small it doesn’t please your testers who would be happier with a "We can’t get your changes into this release but they will remain under consideration for the future" than the cold slap in the face of "WON’T FIX" which sounds like "We don’t care".

    I don’t think, as Mitch suggests, that Bugzilla allows for any better communication, but I do think that some projects are better at providing feedback than IE.  The reason for this is more likely that the bug tracker IS the forum used to discuss the viability of fixing the bug; for IE these decisions are made internally and the results are posted to the bug tracker afterwords so we miss out on the private discussion and any chance to contribute to it.

  9. @Jason Upton [MSFT]

    A lot of what you wrote does not make sense to me.

    1- The "we listen to customer feedback" motto. If Microsoft really had done this, then an IE 6.5 would have been released in 2003 or so with lots of improvements in all areas (security, HTML 4, CSS 2, DOM 2, accessibility, etc) I can think of.

    "Listening to customer feedback is vital to the success of any business." The absence of listening during years reached a summum with the Apple "Hello, I’m a Mac. Hello, I’m a PC" tv advertisements.

    During the whole IE8 beta development cycle, an unanimity of people complained in this blog about the annoying DHTML flying-in tooltip at IE beta feedback and nothing changed during the whole IE8 beta development. So, were you really listening back then?

    > "we listened to real customers, real enterprises, and real developers."

    It is because your company did not listen to real customers, to real enterprises, and to real developers that IE has lost (and still today continues to lose) browser market shares since 2005. I personally remember posts back in 2000 where web developers were clearly expressing their anger and frustration at Microsoft’s decision of not implementing all of CSS 2.0 and not fixing known flaws and bugs in its web standards implementations.

    Microsoft listens to big share holders first. To begin with. Then Microsoft listens to lawyers, prosecutors, judges and juridical instances like EU commission. Then Microsoft listens to big corporate customers. The other "real" customers being listened are clearly last.

    2- > Mozilla’s Bugzilla needs an email address to log-in. WebKit’s needs an email to log-in. (…) We actually went one step further and require LiveID login to query the bugs

    Having the ability to view/read a bug report, to link to a bug report does NOT require a log-in at Mozilla’s Bugzilla and at WebKit’s bugzilla. Nor at KDE, neither at launchpad, not at, etc,etc Only at IE beta connect is this required. If the IE8 beta development was based on openness, then this IE9 beta development is not. I asked in this blog why a registration is required just to be able to read a bug report, read the details (replies, data) and never got a reply.

    3- > With IE8, we took action on every issue that was reported (…) For IE8, we took action on every bug and drove it to closure.

    IE8 bug management has not been perfect; a bunch of bug reports were mishandled by the IE Team. Most of the time, unfortunate errors/misunderstandings were corrected but not everytime.

    In one bug report, I asked what was exactly the difference between "by design" and "wont-fix" and never got a reply… and I was quoting the very words of explanations provided by IE beta feedback webpage. If the comment confirms, validates the bug but its report is resolved as "wont-fix", then such resolution is certainly illogical and incoherent.

    Closing (or resolving) a bug report as postponed is not what I would call driving it to its closure when the bug is valid, sound, correct, confirmed.

    4- > For IE8, we took action on every bug and drove it to closure.  By contrast, as of 4/26/10, Mozilla’s Bugzilla had 12,779 unconfirmed bugs.  Webkit’s had 2,616 unconfirmed bugs across all versions but very few bugs against older versions.

    What you do not mention is that IE8 beta feedback was based on an invitation only and on a selection of beta testers. Good/better bug reports will dramatically reduced the number and percentage of non-confirmable, incomplete, needsmoreinfo types of bug reports. At, anyone can register and file any kind (good or bad, useful or not, valid or not, incomplete, indecipherable, poorly written, not-confirmable, not-reproducible, etc) of bug reports and as many bug reports as (s)he wants to. And, during the whole IE7 beta development and IE8 beta development, they could do so for components such as DOM 2 Events, SVG and any other web standards which IE7 and IE8 had not implemented one bit. You compared 2 browser development organizations where IE with its IE7 version and IE8 betas were clearly behind Firefox 2, Safari 3, Opera 9 in terms of correct implementation of many web standards (DOM 2 HTML, DOM 2 Core, DOM 2 Events, SVG, etc.) during the whole IE8 beta development.

    Gérard Talbot

  10. Martin says:


    "We actually went one step further and require LiveID login to query the bugs, further reducing the risk to our beta users."

    Was THAT the reason it required a login to view a bug?????

    Here is a much better solution: Don’t show me(The person trying to view the bug database) other users email address, unless i am logged in. Problem solved, and no need to hit all the bugs

  11. Matt says:

    Gerard– there are many things to complain about and examine critically, but you’ve picked the wrong ones yet again.

    The current IE team is in no way responsible for the fact that they **didn’t exist** as an ie team in 2003 and thus didn’t deliver the mythical 6.5 browser version you wanted then. You can whine about it, but it’s a waste of breath unless you have a time machine around somewhere. Move on.

    As for the tooltip: the IE team doesn’t actually own Connect either. I think it’s a steaming pile myself (and they are to be criticized for choosing it again and then glossing over its massive deficiencies) but again, whining about the tooltip is a waste of breath.

    Whatever you "personally remember" (as opposed to what, exactly?) about the state of the web in 2000, you seem to forget that IE was the best browser (by a landslide) back then. Sure, MS can be criticized for abandoning it after that, but we’re back to point #1– the current IE team had nothing to do with that. They’re all likely glad that competition sprung up so that they could join the new ie team made for IE7.

    It’s neither illogical or incoherent to agree that "yes, there is a bug, and no, it wasn’t by-design", and still resolve it "Won’t fix." Bugs that aren’t important enough to fix (ever) have this happen to them, at EVERY software shop in the world.

  12. James says:

    It is posts like these that make feel that one you do not listen to developers, two are trying to spin the past as happening in a way it did not. Nothing is worse to read here than PR talk whitewashing why issues that people have to spend real energy to make applications work correctly in your product that are not an issue elsewhere remain to this day and will for years.

  13. Mayweather says:

    So far I use IE to latest IE released by the microsoft, I never experienced bugs nor problem that may cause your internet hindrance.

  14. Typhoon87 says:

    @matt – instead of pointing out the issues with connect on this blog, post a bug/suggestion on

    This is the connection for the team that acutally owns and builds connect. They may be able to help out with improvements. They FINALLY added public attachments in the most recent site update.

  15. says:

    I`ve 2 suggestions that ie 9 could  have.

    When I go to imdb and I want maybe chack jodie riveras page I type address bar: imdb jodie rivera and press enter.. That should be standard feature in every browser.

    Second feature could be information window like view page info in firefox. Internet explorer lacks easy way to get information

    from webpage like rendering mode, information images from images, properties, scaled or not..

    I as a developer check those informations from different webpages to compare.

    I do care about standars support but I also want information from page.

    bottom line ie lacks webpage information !!

  16. jim says:

    A great "PR" post on bug tracking.

    Does bugzilla for mozilla have a lot of unconfirmed bugs? yes! but with public access to a great browser, lots of people will dig deep in the application and submit bugs.

    Are there "poor" bug reports? yes! sometimes you have to deal with the reality that every submitter might not be the best reporter or even a technical user.  This is why the ability to filter down the report to a simple, repeatable test case is critical (reduce the noise, increase the signal)

    Is Connect the best system for MSFT to use? Of course it is, and they have no intention of moving off it (as noted, this is a PR post)

    What is needed though is improvements to Connect – even if just for the IE connect portion.

    1.) All users ******** M U S T ******** be able to see all attachments (unless a bug is reported as private) – You can’t enlist a volunteer army to do the bug triage, refinement, and verification ***IF*** they can’t see the test case code.

    2.) Last time I checked, the search in Connect for IE was horrible. If this has been fixed, great – if Not this is needed.

  17. brett says:

    Who is responsible for the lack of a TABBED WINDOWS EXPLORER in Windows 7?

    Why have a tabbed browser but no tabbed file manager?

  18. Fiery Kitsune says:

    The IE team would use Bugzilla or Mantis in a heartbeat if they could lock down either platform to be as restricted and code-hostile as possible.

    I’m also pretty sure that Microsoft finds the idea of using an open-source bugtracker or open-sourcing their own to be absolutely abhorrent.

    The "WE MUST DEVELOP EVERYTHING HERE!" mentality is astounding and stupid.

  19. ThePlaz says:

    Great post.  

    I can see how nightly builds is too much – but I just wish you would move to monthly builds at least.  Every Microsoft product it seems does a beta and 3-6 months later an RC and then 3-6 a release.  Huge changes can be made between version.  More problematic is that people report bugs fixed months ago.

  20. JustinSC [MSFT] says:


    Would you mind providing a link to the bug that you filed?  It would be helpful to be able to check on where it currently is in our system.

    @Mitch 74 & @jim

    Thanks a lot for your feedback!  As @Tom Stack points out, many of the challenges you describe are the results of the current design of the Connect system.  The Connect team is dedicated to providing the best feedback system possible, and I’m certain they’d very much appreciate hearing your input on how to make the process better.  This includes the challenges of re-activating a bug from release to release (which isn’t possible if a new bug form is used for the new release), having to use ZIP attachments for code instead of the source files themselves (which is due to appropriate security concerns), linking bugs, and the search experience.  I encourage you to add your voice to their feedback program at


    Currently users can in fact see attachments, unless the submitter filed it as private.

  21. gary k says:


    the original bug was 437631, but is not accessible now. it was filed on 5-4-2009. worked through email with reza on the issue.

    refiled the bug in february of this year. here is the link.

  22. Hey Jason! 🙂 I’m one of the primary developers of Bugzilla, and I really appreciate and found interesting the analysis you did of Bugzilla vs. Connect! 🙂

    It’s true, a lot of bug-tracking systems have a lot of things in common. That could at least partly be because Bugzilla is pretty much the oldest bug-tracking system in common use today, and so to some degree was a model (or at least something for comparison) when building other systems.

    There are a few points I wanted to respond to:

    1. As of Bugzilla 3.4, Bugzilla no longer displays email addresses to logged-out users, so the note about being spammed isn’t really true anymore, unless spammers start creating Bugzilla accounts (which I suppose is possible, but we don’t have any reports of that yet). With some minor modifications, Bugzilla could actually even protect email addresses when the user is logged in, if that was important.

    3 & 4. As far as there being a lot of Unconfirmed bugs in Firefox, I’d say that that’s more of a function of the Mozilla Corporation and the way that resources are allocated, as opposed to an aspect of the tracker itself. It sounds like Microsoft has some people being paid to actively look at incoming bug reports, while at Mozilla it’s a bit more hit-or-miss as far as whether or not any paid person will actually be there to see the bug reports. However, do you have any particular differences between Connect and Bugzilla that you think makes it easier to address incoming bugs in Connect? We’d love to improve Bugzilla if you have any thoughts on it.

    You know, one thing I’d love to see in Connect that is available in Bugzilla is the ability to *read* bug reports without having to log in. Bugzilla was built from the ground up to be an open bug tracker, so that is probably an advantage it has over Connect, in some ways, although I don’t know much about the history of Connect.

    Any how, good luck with IE9 and the feedback process! I’d love to hear any responses you have to my comment, if you get it. My email address is


  23. Oh, also, I forgot to mention, but getting Bugzilla to authenticate against Windows Live would be really easy–Bugzilla has a pluggable authentication system.


  24. steve_webdev says:

    @JustinSC [MSFT]

    re: "@jim – Currently users can in fact see attachments, unless the submitter filed it as private."

    This should have been announced!  Attachments in Connect have NEVER been publicly viewable, a point that developers and testers have complained about continuously since IE7 first entered beta development in Connect.

    Here’s a screenshot of a bug I submitted (marked as public), with in this case, some screenshots attached.

    You’ll notice that none of the 5 files are preview-able/downloadable.

    however, I tested a brand new bug submission and yes, you can attach files that are downloadable!

    However, there’s a hidden link tooltip that indicates you can’t upload HTML, JS, or SWF files. (extremely annoying)

    You’ll need to zip those files first I guess.

    There is a definite delay in seeing the files show up though, so don’t be surprised if it takes 10minutes for the files to show up!

    Great news on this ability being added – I look forward to reviewing other attachments to reproduce bugs – and others being able to view my attachments (at least on the new bugs filed – as the old ones are still locked)

  25. Jason [MSFT] says:


    Thanks for the update on the Bugzilla 3.4 behavior.

    I’ll pass your suggestions and contact info to the Microsoft Connect team.  Thank you.

  26. IE8 has been completely unusable for me and I am known as an extreme enthusiast of Microsoft products. The yellow security info bar was super-annoying and now my co-workers and I have the following error (for which I have not found a proper available forum to submit feedback on)…

    Security Warning Pop-up Box

    "Do you want to view only the webpage content that was delivered securely? This webpage contains content that will not be delivered using a secure HTTPS connection, which could compromise the security of the entire webpage."

    I’ve taken steps to disable the pop-up, but it still persists…

    Internet Options > Security tab

    Enabled Protected Mode (requires restarting Internet Explorer)

    Checked for Internet Zone & Restricted Sites Zone

    Unchecked for Local Intranet Sites Zone & Trusted Sites Zone

    Security Settings – Trusted Sites Zone

    Display mixed content = Enable

    Securit Settings – Internet Zone

    Display mixed content = Prompt

Skip to main content