Hmm, it’s been a few days of reflection. I don’t think I expected my post on the bugs we were fixing in Update 2.1 to generate so many comments about other bugs that people are hitting. Let me share some questions that have been on my mind and my thoughts on them…
Why did I get so many bugs reported as comments on my blog?
In the end, my blog isn’t the place I want people reporting bugs. It’s high cost for me (I’m just overhead because I route them to the right person to look at them). It’s not a great experience for you – you don’t have a great way to track updates. And it’s not a great experience for our engineers because getting additional details is difficult – blog comments often don’t have contact info. The system we have for this is http://connect.microsoft.com for bugs and http://visualstudio.uservoice.com for suggestions.
I have some hypotheses. 1) I kinda asked for it by putting up a post with a lot of bugs. 2) My blog is REALLY responsive. If you post something there, I almost always read it within an hour and forward it to someone on my team and, because it comes from their VP, they tend to drop whatever they are doing and get right on it. So that part of the experience for you can’t be beat. 2) connect.microsoft.com is a less than ideal (to be gentle) experience both in terms of the UX and, I think, our responsiveness.
But those are just my hypotheses. I’m interested in any thoughts you all have on this.
Don’t get me wrong, my blog is definitely available for anyone who feels they need to be heard. If you are frustrated about something and can’t get satisfaction (or really happy about something and just want to say so), I’m here. I don’t want people to ever feel like I’m discouraging discussion. At the same time, I’d like for people to find satisfaction in our “normal” channels. If they aren’t working, I want to know why and see what it takes to fix them.
I’ve asked someone to take all the bugs reported on my blog since Update 2 shipped and go look on connect.microsoft.com and see if they are there. Maybe they got filed and we just ignored them. If so, that’s something to fix.
With this additional list of bugs, what do I do with Update 2.1?
The schedule for the final checkin for Update 2.1 is today. We’re not going to repro, identify and fix every one of them by then (and I’m still seeing a couple of new ones a day). I’ve decided we’re going to go ahead and ship Update 2.1 “on schedule” with the fixes we have. Regardless of however many additional fixes we could make, Update 2.1 has a couple of dozen valuable improvements. Further Update 3 isn’t that far away. If I wait too long, Update 2.1 becomes pointless. We’ll finish “development” on Update 3 in about 6 weeks and ship it within a few weeks after that (I think). Given all that, we are going to move forward and ship Update 2.1 next week.
What am I going to do with the bugs that have been reported?
We’re going to continue to work on them. Not only that, I’ve asked the team to go back and do a thorough scrub of connect.microsoft.com, Send-a-smile feedback, etc. and identify all the other bug reports that should be and can be fixed for Update 3. So, keep reporting any bugs you find. I can’t promise that we will fix all of them but we’ll fix as many as we can.
What does this all mean about the quality of our development process?
Well, I’m not sure yet. It’s the first time we’ve ever wanted to do a .1 release on an Update. We’ve had bugs. There are always bugs. It’s nigh impossible to eliminate every single bug in a very large complex system. But, it’s the first time we’ve had enough of them that we’ve said, hmm, this isn’t good enough. We need to do better. We’re also doing a retrospective on all the bugs that have been reported to understand when they were introduced, what the root cause was, what could have been done to have detected them earlier, etc. From that we’ll make some adjustments and I hope we will not have this number of bugs again any time soon.
All that said, I’m not saying Update 2 was a *bad* release. Lots of people have installed it and lots are using it very happily. It has lots of good stuff in it that people are very happy about. But, it could have and should have been a *better* release. Once I’ve seen the retrospective, I’ll know more. Overall I am very happy with and proud of the work this team does, the quality and commitment of the people and the innovation in the process that allows us to deliver so much value in so many different ways. At the same time, everything is an opportunity to learn more and be better. The only question is whether or not you recognize the opportunity and act on it.