IE Public Bug Database – Connect


I’ve been following the comments over on the IE blog about Molly’s post, particularly the comments of Chris Beall, thacker and steve_web, and I figured I’d post a bit of a follow up to what they were saying and ask for feedback from the community.

In the IE7 timeframe, with the connect database open, we took in close to 10,000 bugs.  I don’t remember the exact number, but it was closer to 10,000 than 5,000.  Yes, you read that right.  The problem is that realistically, the number of real customer bugs out of that 10,000 submitted was tiny.  I’m talking fractionally tiny. 

So what makes a bad bug.  We got bugs that basically said “IE sucks”.  That was it. No more detail than that.  Every bug like that, takes time for someone to look at, triage, resolve, close, and so forth.  It’s not a short process.  For each bug that we outright close, it’s probably about 3 minutes work. 

Then there are the bugs that say something like “x does not work.”  Great, so the bug gets handed off to a developer or a tester and asked to see if they can repro the problem.  They’ll likely spend 10-15 minutes on it.  If they can get a good repro back to triage it goes with comments.  If they can’t, it still goes back to triage, where someone will end up closing it.

Then there are the diamonds.  The great bug reports.  Someone who includes a HTML and CSS snippet, an screen shot from IE and a screen shot from another browser.  In cases like these, triage looks at the bug, makes a decision on how serious the problem is, how many users it will affect to fix it, what the implications are to fix it, and then either assign it to a developer, or reject it.

But wait, there is still another category, and this one is the most trouble some.  Duplicate bugs.  Say Chris Beall files a bug, and then thacker files another bug that but because of the way Chris filed his bug, and thacker filed his bug, don’t find each other.  Well, now, we’ve got two bugs filed for the same issue.  Now triage looks at those two bugs, and tries to figure out if they’re the same.  Most likely, they’ll assign them to a tester or developer to see if they are or not. 

So what is the solution for today and the next version of IE?  There are people on the IE team who are very much opposed to opening connect again.  Primarily because we can’t handle the volume of feedback that you have to give.  Outside of Windows, I’d be willing to guess we’ve got the largest user base of ANY application.  I think to not reopen it is not the best decision, but I completely understand the cost involved in reopening it.

I’m about to present a proposal to the IE management team (shh, don’t tell them, I haven’t told them yet ;)).  Instead of opening Connect to every single web developer and web designer in the world, we open it to a select group of people.  Those who have proven themselves to file good bugs and understand the problems that we’ve faced in the past.  They need to be completely connected to the web world, and know and understand that the bugs we’re looking at can’t just be “IE SUCKZ!!!!111OMGBBQ!!!!111!!!1!”  (and yes, we did get a bug similar to that).  We need to find a way to make them accessible to you, so that you can tell them about your bugs, and they can look at it. 

And finally, and I think more importantly, we need to find a way to make our bug tracking system more transparent to you.  Let you know if we know about a particular bug, and if we’re going to fix it in the next version, if we’re punting it, or what not.  I don’t know today how we do this, but this is the thing that I’m trying to find the answer to today.

This post turned into one MUCH longer than I was expecting.  Hopefully I’m coherent enough to have made some things clear and explained the difficulties faced by the IE team, and why the Connect database isn’t open today.  We need to find a good solution that works for all of us soon, and I’m working on it.

Comments (21)

  1. Rosyna says:

    Why not do something similar to what Apple did? http://webkit.org/

    Aside from opening the source code, Apple allows anyone to file a bug (that makes an account). They then give trusted people that have shown a history of giving good bugs, good reductions, and good comments on bugs the ability to triage bugs, set priority, and mark them as duplicate.

    "Instead of opening Connect to every single web developer and web designer in the world, we open it to a select group of people."

    This comment alone could be interpreted by trolls in such a way that it makes Microsoft sound like they only want to hear from people that are approved by Microsoft and don’t have anything negative to say about IE (Standards support or otherwise).

  2. Joku says:

    You could try community+incentive approach:

    Advertise it on the front page of the bug reporting system that eg.

    "Top 100 people who have highest score (s/n ratio & amount) get free upgrade from say Vista Home to Vista Ultimate/free Xbox HDD upgrade/2 years of Xbox Live Gold/….

    The s/n ratio would mean having more of validated (by yourself) bugs that were internally triaged to be fixed/won’t fix and less of validated (by yourself) bugs that were determined to be non-bugs."

    That should give some incentive for community based bug validation to work to atleast sort out most of the useless reports as your score would only increase if you did a good job (as later determined by the Microsoft team looking at the bug).

    Now whether or not the suggested incentives are good enough is hard to say. The amount of lower end rewards should be such that most of those who put a decent effort have a good chance of getting something. Maybe something nicer for those who got a score well above average?

  3. fr says:

    Only opening it to a few select people will still leave you with lots of people complaining and with no easy way to report their bugs.  Surely if mozilla and apple can handle a public bug databases, ms should be able to.

    Carefully picked community moderators as already suggested would be the best way to maintain a public database, and you have to make the database public if you want to improve web developers opinion on IE/MS/

  4. steve_web says:

    Pete,

    I appreciate the candidness of this post, and actually like your idea (somewhat 😉

    I would like to consider myself as one of those uber web developer folks you mention, that can and would provide only 100% authentic bugs, complete with test cases.

    I’m not sure how you might determine an eligible user, hence my minor skepticism with a not-so-public database, but I do believe that this could work.

    Looking forward to an invitation to ‘prove’ my web savvy.

    Oh, I wanted to suggest something on this… you might want to consider giving "responsible" bug submitters, a "power" to vote/validate other bugs/users… this would take a load off your hands, and help weed out duplicates, and non-bugs.

    Similar to say Digg (I’ll presume everyone knows the reference here), a user with "voting" power, could analyse bugs as they came in, and "validate" the bugs.

    E.g. new user ‘xyz’ submits a bug that: "IE doesn’t render page ‘test123’ properly"  as a responsible user, I could view the page in question, and determine that… <tabel> tags were used, thus the browser rendered correctly, based on what it was supplied. (typo above was intentional)

    In this scenario, the bug could be marked as "resolved" (with an explanation as to why) before it even reaches the MS team.

    More importantly, a submission of: "IE SUX, BUy Firefox and replace IE engine" could be immediately flagged as: "Ok, this is lame" (i.e. troll/spam)

    I think you will find that the more involved submitters would like to help.

    From old IE Feedback, Aedrin, steve_web, that guy/girl with all the funky characters in his/her handle, patriotB, etc. would be the kind of users in the system that would be able to really assist in helping manage/control the new and improved IE Feedback site (TBD).  🙂

  5. I see I’m not the first person to suggest something like this but:

    As I understand it the way the Mozilla team handles the same problem is by allowing anyone to file bugs but initially puts them into an "unconfirmed" state which the team themselves don’t even look at usually. Then there’s a large group of triagers (who in your case *wouldn’t* be in Microsoft, but rather would be external people who’ve shown the ability to submit good bugs, find duplicates, etc) who can move a bug from Unconfirmed to Confirmed at which point it will be looked at by the team.

    A closed system ensures that you’ll never find the *next* person who’s able to submit a "golden" bug report. A system where anyone can submit but only trusted people can forward the submissions all the way to the team provides an excellent way to continue to increase the size of your trusted group. When someone submits a bug that ends up making it to the team, as well as reviewing the issue itself, you can also review the person’s submission history and decide whether they’re worth admitting into the trusted group.

    Your problem is that you’re still thinking about too tall a "wall" between Microsoft and the outside community. Microsoft has limited resources to triage bugs, the community has unlimited resources to submit them. Of course that’s not going to work. But there’s no reason you shouldn’t use the community’s resources to do the triage too!

  6. Mike Johnson says:

    I am dissapointed in what this post says about MS and its relationship to the community. As you state, IE has probably the second largest user base in the world (second only to windows) and with the profitability and resources (50k+ employees) its sad to hear what amounts to "whining" about a bug database.

    IE has an image problem all around and that you are getting feedback in any form means they are still engaged in using the product.  Right now most developers I talk to love ASP.NET but the problems with IE are almost to a tipping point of no return  –  you are going to have to go several extra miles to win back the developer community and ultimately the user base at large.

    you really need to focus on these things:

    0) IE 6 is a turkey, IE 7 is improved but still a promisary note on a future browser we *HOPE* will appear someday that sovles compliance, security and general bugs. People are frustrated and will lash out with meaningless reports. get over it and move on.

    1) out of the gems, let the community determine what needs to be fixed. its not your call to determine what the user/developer community values and needs. if your customers dont feel valued/important they will migrate to where they do and for all intents and purposes that means Firefox.

    2) dup bugs are a fact of life in any bug base. I am sure the internal windows bug base has them too. lots and lots of them. the community can help with this but wasted effort is going to happen. get over it and move on.

    3) Step up the release cycle and deliver true value in terms of features or improvements. Firefox is kicking your ass in terms of releases and features and add ons. Look at Jeff Atwoods comparison of the IE developer bar vs. the firefox Add Ons on his blog: http://www.codinghorror.com/blog/archives/000780.html

  7. thacker says:

    Pete–

    What I know, when compared to the experts, wouldn’t fill a damn thimble.  

    Structural and operational aspects of a bug reporting system that addresses your stated concerns, including efficiencies, are beyond my scope for the most part.

    What I do know is this, "It is not important to remember everything as it is to remember where to find it when you need it."

    Consolidate IE7 resources into one easily navigated, user friendly area.  This includes links to all pertinent Blogs from the IE team.  Hell, I never knew yours existed.

    1.  Include notable outside Blogs and Web sites that pertain to Internet Explorer.

    2.  Include links to outside resources of workaround resources.  Molly Holzschlag is currently gathering some of those.

    3.  Provide developed content for known issues and workarounds, gathered both internally and externally.  Provide usable information in the same manner to the user/designer/developer that you expect within an ‘exemplary’ bug report.  This includes screen-shots of the problem with the problem and the appearance after the workaround.  Keep the language as simple as possible.  Imagine that you are speaking to me [an eleven year old kid].

    4.  Incorporate the IE forum, possibly using it as a base for triage and movement to a more elevated level into a bug system.  Let  no reasonable post go unanswered.  Promote participation.  Offer a circa 1981 Gate’s photo dartboard. :::chuckling:::

    5.  Provide expectation. No two genetically identical mammals will react nor look the same.  There are instances where no two Web browsers will render code the same.  An example is print capability of a rendered page between browsers.  It is hand grenades — close is good enough.  If a developer/designer requires a zero reflex kill shot for print, then deliver a PDF file.

    6.  Expect that any forum or user participation area on the Internet will validate Darwin’s Theory of Evolution — in reverse.

    7.  Keep it simple and stupid.  If you need to drive a nail do not do anything fancy.  Simply, pick up the hammer and drive the damn nail.

    That is my two cents worth.  Consider the source.

    Maybe, IE.Next will be arriving within the year.  A lot of these workarounds, bugs, whatnot will become moot.  Okay, so a kid is allowed to dream — so what.

    Thank you very much for your passion, conscientiousness and commitment.  It is very much appreciated.

  8. Charles Bocock says:

    I think opening the bug system to only a select number of people is a dangerous idea.

    The problem is that you’ll only get those 80% of bugs which are obvious/easy in your list.

    Internet Explorer is the most important application in the world, because it is the most used application. There are so many tiny, niche bugs that you would never get reported if you lock down the system. Stuff that only a handful of people are hitting (e.g. browsing Korean web sites on an Intuktit localized version of Windows).

    I like the idea a poster above mentioned about how Mozilla does it. Everyone can submit bugs, then the community weeds out the wheat from the chaff. There are plenty of people who would be happy to do this, but it would be good to incentivize this boring task by offering freebies to the people who accurately filter the most bugs into the system.

    I think that would get you the best of all worlds.

  9. Charles Bocock says:

    Oh, and how about a localization of Internet Explorer into my native language – English? It would be great, after all these years, to see "Favourites" rather than the American "Favorites" 🙂

  10. Jason says:

    So you mean to say that window that pops up after IE crashes and asks me if I want to send this crash to Microsoft, doesn’t actually get looked at?

  11. John S says:

    Although Mike Johnson’s comments appear to be harsh… I’d have to agree.

    The IE and FireFox communities are completely different. People are helping out with the FireFox project because they love the browser. Whereas IE developers are just desperately hoping that some of the crap they dance around every day will get fixed – the really frustrated few take the time to submit bug fixes.

    If you want fewer "IE sucks" bug messages, then win back the community. Sure, XBOX 360 giveaways and other gimmicks will make happy for a while but ultimately developers just want a better product. I can only play a 360 a couple hours a day but I spend 8 hours a day battling IE – which one will make me happier? I’d choose an easier workday.

    Don’t get me wrong, I’m not anti-IE, but I use FireFox (I’m in it now) because it’s been my experience that it’s better. Until IE7 was released it wasn’t even worth comparing the two. At this point I’ll keep programming for IE, but only as long as it continues to be the dominant browser.

  12. Chris Beall says:

    Pete,

    As you requested, here’s my feedback.

    What I want:

    When I detect a bug in IE, I want to be able to report it to MS.  My purpose for making that report is to get the bug fixed, so that my web pages will look as intended without any IE-specific changes required to my standards-compliant markup.

    I want it to be easy to find the bug-reporting mechanism.  To me, the best way to do that is to add an entry under "Help" that says "Report a bug" and which takes me to the entry web page for the bug-reporting process.  [EVERY MS product should have such a link].

    I do not want to create an elite corps of bug reporters.  Anyone can find a bug; anyone should be able to report it.  [In the past I have, in desperation, asked for special treatment (without success, lest others be jealous), but it’s a bad idea.]

    Once I have defined my bug, I want to see if it is already known and, if so, if there is a known workaround.  I also want to know when it is likely to be fixed, so I can judge whether applying the workaround is cost-effective.  I also want to be able to add myself as an interested party to the bug report.  The count of such interested parties would provide MS with one measure of pervasiveness.  If I could leave my email address, that would allow MS to notify me when a workaround became available or when the bug had been fixed.

    I expect my bug report to include sufficient data to allow MS to reproduce the bug, without further input from me.  In return, I do not expect to be asked to "try removing the DOCTYPE" or the like; if developers can duplicate the bug then they can try all the experiments they want to isolate it in the code.

    And, of course, I want my confirmed bug to be fixed.  Promptly, meaning within no more than 180 days.  Without introducing another bug.

    Now some comments on the problems you have described.

    Excessive signal-to-noise ratio in what you call Connect.  Well, maybe Connect is too general.  Maybe Feedback (which is what the page was actually titled) is too general.  If you call the first page of the process ‘Report a bug in Internet Explorer’, that might help.  If you follow it up with an introduction that says "This process is intended to allow you to notify the IE Development Team of a defect in Internet Explorer.  To make suggestions for improvements or new functions in IE, go to http://yadda.yadda/suggestions.  For assistance in developing web pages, go to ……  Etc."  You have to TELL people what the process is for and, to keep out crap, you have to TELL people who have stumbled into it where they can find what they were really looking for.

    I don’t like having to register for things, but I think it should be required.  It HAS to be required if you are going to let people add themselves to an interested parties list; otherwise there’s no way to prevent someone from pumping the vote.  Registration means ID, password, and email, nothing more (um, and no spam in return).  This also allows you to notify the submitter (if requested) when a workaround or fix is available.

    Bug submitters should be required to document their bugs.  As stated above, that means providing enough data to allow the developer to reproduce the bug.

    How do you do all this?  Why not use…computers?  (The following scenario is necessary abbreviated and incomplete.  The point, however, is that IMO it is not an insurmountable challenge.)

    Help

     Report a bug

    Report a bug in Internet Explorer

    Indicate which area of IE is failing:

    _ Rendering (the appearance of a web page)

    _ Options, setting them or how they respond

    _    :

    _ Other (there should always be an Other, for the case you didn’t think of)

    Depending on the selection above, ask the next relevant question.  I’ll pick Rendering, ’cause that’s where I’m most likely to be.

    IE Rendering bug

    Which (X)HTML element is not rendered properly? [Select from a LIST of valid element names]

    Which, if any, CSS property applied to this element causes the failure to appear? [Select from a LIST of properties that apply to the selected element]

    Which CSS Media show the problem? (check all that apply)

    _ Display

    _ Print

    _ Etc.

    Please create a web page that contains only enough HTML and CSS to demonstrate the problem, and no more.  Enter the URL of this page ___________________.

    [That’s enough to give you the idea.  Several things are happening here:

    – The submitter is being prompted to make a case.  In the process, most ‘bugs’ will be found to be user errors and you’ll never see them.

    – You are collecting data needed to reproduce the bug.

    – Under the covers (or perhaps displayed back to the submitter; why not?) you are building a search string containing the symptoms of this bug.  At some point, when enough data has been collected, you smash this string against the bug database and then:]

    Listed below are some known bugs that may describe the problem that you are having.  Please click on each bug to read more about it and see if a bypass is available.  If your bug does not match any of those listed,  please enter into the space below a description of how your bug differs from those in the list, then click Continue.

    [The display of known bugs should include a way for the submitter to say "Yes, that’s my bug!" and then, optionally, add his ID to the list of interested parties and, also optionally, elect to be notified when the bug is fixed.]

    [Let’s talk about triage.  In addition to the interested parties list, you could ASK the submitter:]

    Based on your experience: [stroking the submitter a bit here…]

    – What percentage of web pages will be affected by this bug?

    – How many minutes per page will it take you to apply the suggested workaround? [If one exists]

    – If not bypassed, how severe will this bug appear to the end user:

        _ invisible unless you use a ruler and micrometer

        _ minor cosmetic effect

        _ increased difficulty viewing page content

        _ page is unreadable

    —————————

    BUT:

     IF you provide a pathway for developers to report bugs

       AND IF you screen bugs via a logical process like that described above

         THEN you will save so much time and money that you can fix ’em all!

    Chris Beall

  13. Chris Beall says:

    One other thought regarding reducing invalid bug submissions.

    When something flat doesn’t work in IE, such as a specific use of the CSS page-break-inside property, it’s hard for a developer to know if that constitutes a bug or just a piece of the standard that MS has not gotten around to implementing yet.

    To address this, I’d like to see a matrix that shows what IE does and does not support.  The format could be similar to the one at http://www.webdevout.net/browser-support.

    Chris Beall

  14. thacker says:

    Beall–

    Sounds like a plan to me for bug reporting.

    Microsoft–

    Hire that guy.

    Pete, Molly, Chris Wilson, Brian Goldfarb, Dean Hachamovitch, et al–

    What does the entire IE team want to accomplish?  What are your priorities?  What is important to you, individually, collectively and as a product? What do you need from outside sources to get it done?  How can we help?

  15. Al Billings says:

    It’s good to finally hear why my project was killed after I left. 🙂

  16. Al Billings says:

    It seems my comment was deleted. Why?

  17. PeteL says:

    sorry Al, it was marked as spam, and I had just noticed based on your comment.

  18. joel maxri says:

    I concur with everyone above, that thinks (a) that this system should have been in place years ago, and (b) understands that Microsoft loses credibility for every day that it does not exist.

    As a Pro Web App Developer, I know of about 50 known issues that are broken in IE (not missing implementations of specs, but flat out broken implementations of specs).  Armed with this knowledge and access to a public bug tracking system, I can not only file such issues, but I can help triage other bugs filed, determine if they are duplicates, develop simple test cases etc.

    With a half decent system in place, where the users of the system have earned "respect/authority", you’ll find that they will "self-police" the system, so that you are not overrun with useless bogus bug reports.

    I would love to spend say 4 hours on a weekend, filtering through, and cleaning up bugs in such a system.  The mere thought, that this would help isolate, and help prioritize bugs would bring me much pleasure, as one would gain hope, that if the bugs are neatly identified, reproduced, and even provide a test case (uploaded (not online, too many issues here)), then creating the fix, should be a piece of cake, and best of all, testing it would be a snap, since the test case is being provided for you.

    Waiting for your comments on when such a system will be up… or what specific features "we" are asking for.

    e.g. some of the Required fields of a bug report.

    – Mark as duplicate of xxxxx

    – Upload a test case (or patch if some developers have access)

    – Mark as [reproducable|cant reproduce|bad markup|not a bug|…]

    – Status  (MS field) – [initial|confirmed|fixed|resolved|works per spec|suspended|in testing|…]

    —-

    Note, that idealy, the system should be easy to use when a frequent reporter acesses the system.  e.g. in Feedback, you could only see the top 5 or 6 items, and you had to page through several screens to find stuff.

    Otherwise excited like a kid in a candy store, waiting to help in the process of development of Internet Explorer!!

  19. steve_web says:

    Just one note on your opening statements…

    "we took in close to 10,000 bugs.  I don’t remember the exact number, but it was closer to 10,000 than 5,000.  Yes, you read that right.  The problem is that realistically, the number of real customer bugs out of that 10,000 submitted was tiny.  I’m talking fractionally tiny."

    So, if the number was 10,000. then 10% would be 1,000, and 1% would be 100.  To be fractionally tiny, the # of valid bugs would need to be less than 100 to be a fraction.

    Considering that I myself filed at least 75 valid, reproducable, confirmed bugs, I highly doubt that the total # of real bugs was "fractional".

    That said, some of those "fractional" bugs, bite 10,000+ developers every single day!  so no matter how small the number is, they need to be addressed.

  20. Al says:

    That’s ok, Pete. I think the blogs.msdn.com system has had it in for me since I quit doing the IEBlog. None of my comments ever appear there either and whoever runs it doesn’t unspam them.

    I’ll be visiting campus tomorrow to have lunch with your peers. Maybe I’ll bring that up then. 🙂

  21. Thomas Tallyce says:

    I think a big part of the problem with the implemented Connect system was that its usability was not great, as far as I remember. Duplicate bugs is probably partly an indication of a poor reporting or searching system.

    Mozilla’s system does seem to work. Would it be unacceptable to use a bugzilla instance?

    Personally I think it’s useful for people to have some way to report obscure bugs, but in practice I’m not 100% convinced a public bug database is actually needed at the moment.

    There are already some good, public resources like the Channel9 wiki page [1] that contains a good list of problems, many of which would undoubtedly involve a *very substantial* amount of coding/refactoring work, but work which is surely necessary to get IE to catch up with the web. Perhaps developers need to let the IE team work through and stabilise all that implementation, and then open up a public bug database, to deal with the smaller stuff that then arises.

    [1] http://channel9.msdn.com/wiki/default.aspx/Channel9.InternetExplorerFeedback and http://channel9.msdn.com/wiki/default.aspx/Channel9.InternetExplorerBugs