Only Human

I’m sure others have talked about this, but I haven’t had a chance to check out the other blgos, so I’m going to throw in a little bit of information about what’s going on work right now.

On the road to shipping beta2 we’ve now entered a time called “ask mode.”  The name comes from the fact that before we’re allowed to make any changes to any part of the IDE we have to “ask” for permission from up on high.  Actually “ask” is a rather generous term.  What you’re actually doing is pleading, cajoling, bribing, or just pain arguing that the issue you’re trying to resolve is necessary for beta2.  In order to judge this the “ask mode committee” attempts to determine where the “bar” is in terms of the quality of VS and whether or not you’re currently below that bar and if the fix you have will raise you above it.  Of course, as you get closer to shipping this bar gets higher and higher as the damages that will happen if something goes wrong will get more critical.  Less time to fix bad bugs means a better chance of shipping crap out to you.  So they try to find a careful balance of determining which bugs are worth fixing versus the potential for worse bugs to be introduced.

Each team has their own individual “bug bar” as well which they’ll use to triage incoming bugs to decide if they should be fixed for beta2 or if they should be fixed for the final product.  Each team has their own set of criteria, i.e. hangs, crashes, critical functionality broken, but it’s pretty safe to say that the “bar” is defined as: something so bad as to be known to be completely wrong while *at the same time* making the IDE so unusable that we won’t be able to get good feedback about everything else because this will be bothering users all the time.  For example, a crash in a mainline scenario (like typing) is really³ bad.  It’ll make the product unusable and will prevent users from actually getting to try out more than 5% of its functionality.  However, a crash in some very bizarre and difficult to get into scenario might not meet the bar.  Something like a crash in the immediate window when you type a very special type of expression that has an associated debugger visualizer on it that needs to cross security boundries to a sql machine that then executes a stored proc with a managed-to-native transition.  That’s just something that we can deal with since it will probably hit a low percentage of users and one could still effectively use the rest of the product. 

What’s interesting is that one bug we brought to “ask mode” that was actually accepted was a problem with type colorization where the colors had accidentally been redefined so that teal was now showing up as light gray.  “What!!!” you exclaim “how could that possibly meet anyone’s bar???!”  Well, in this case we felt (and argued) that the experience would be so god awful for anyone typing that the literally would hate every moment of it and would feel so negatively against the product that they wouldn’t want to use it now or in the future.  So while it wasn’t something so obviously bad as a crash it was so completely wrong for the C# customer that we felt that that was the only issue they would give us feedback on at the expense of everything else we’ve worked on.

Now, for a dev this time is incredibly hard to deal with.  For months and months you’ve been happily coding at a great clip.  Your features have been getting better and better and you’re totally psyched that they’re going to get into the hands of the customers who’ve been waiting.  And all of a sudden it comes to a screeching halt.  Issues that would have once take 5-10 minutes to fix now take on the order of 4-5 days.  No, that’s not an exaggeration.  Now, a fix which you could have coded up, had reviewed and checked in after having all tests pass has to go through the following gauntlet:

1) You have to have your individual team triage and sign off on fixing the issue for beta2

2) You have to prepare the fix

3) You have to the fix reviewed

4) You need to create a build of the fix that you can hand off to QA where a far more complete test pass will be done over it including integration testing where it’s banged on a real world scenario to make sure that nothing has regressed

5) Once it’s been signed off by dev and QA you need to place it in escrow where it cannot be touched again. (if you touch it again you need to start at the beginning

6) You now need to bring the bug to DevDiv shiproom and convince then that it’s worth fixing.  The dev who reviewed it and the QA who tested it need to be there to confirm signing off on it as well.

7) You need to bring snacks as well to bribe shiproom.  There’s about 30+ people from multiple teams that you have to convince.

8) Sacrifice a virgin to the gods (optional)

Sigh…

Step 7 is why I often blogged about issues and asked you guys to tell me what you felt.  We really needed to understand the customer perspective on this so that we could convince others it was worth doing.

So productivity for a dev sloooooooows down by several orders of magnitude.  But I guess that’s ok.  We’re usually the ones causing the problems in the IDE so it makes sense to allow us to change as little as possible so that the product can truly stabilize and get into good shape for you.

Now, the title of my post comes from the fact that ended up completely screwing up this process.  Because of the drastic shift in policies I misunderstood the different roles that were played (especially the difference between the individual team triage, and the entire division “ask committee”), and I ended up checking in a bug fix without doing everything that I was supposed to.  This caused a whole lot of grief for many parties and ended up making the rules more stringent and more understandable so that other silly devs like myself wouldn’t rock the boat anymore.  In the end it was felt that the mistake was forgivable.  It was fixing a pri0 failure in Japanese Visual Studio and Pri0s are not considered “ask” bugs rather they’re “tell” bugs (i.e. you fix them and then tell the committee why you did it.

Anyways, it’s an interesting time now.  Working a glacial pace to get beta2 out of here.  Working on the bugs that have been pushed to the RTM release, and also putting a lot of thought and energy into the plans for the next release after this one.  We’re working on some kick-ass stuff and I think you’re going to love it once you hear about it.