Thinking about Last Checkin Chicken
Raymond Chen’s post today started me thinking about “Last Check-in Chicken” again. Back in the says when we were close to shipping Windows Vista, I wrote about ”Last Check-in Chicken”. What I didn’t mention was who ultimately won the game for Windows Vista.
It turns out that the very last change to Windows Vista was actually made by one of the developers on the sound team.
When you reach the last few days of a project, the bar for taking changes is insanely high – the teams which approve changes to the product get increasingly more conservative about taking changes – every change taken is an opportunity for regression and resets some amount of the testing which has gone before. So the number of bugs that are accepted towards the end of a product gets smaller and smaller. You can think of the ability to take bugs as a series of ever increasingly high barriers – it starts fairly low – just about any bug fix will be accepted into the tree. This is the normal state during most of product development. As time goes on and the team gets closer to shipping, the bug bar gets raised and the bugs that are considered are only those that are going to affect customers directly (as opposed to those bugs found during testing won’t necessarily be encountered by customers). Then the bar gets raised again (and again, and again) until eventually it gets to the point where the only bugs that are accepted are “recall class” bugs[1].
The idea behind a “recall class bug” is that it’s is a bug that is so bad that we’d be willing to call the manufacturer and pull the product off the assembly line (at a cost of millions of dollars) to fix. These are the worst-of-the-worst bugs, and typically involve major scenarios not working. When the bug bar is at “recall class only”, there are typically only two or three bugs that are considered each day across all of Windows and even then most of the bugs brought up to the triage team aren’t accepted.
At some point the bug bar gets beyond even “recall class only” – this is when you’re REALLY close to being done (typically the last two or three days of a product). Normally builds of the product are done daily because there are one or two “recall class” bugs still being accepted. But eventually all those bugs are fixed and the build team stops doing daily builds because there have been no changes since the previous build. The test team is hard at work doing it’s final sign-off of the bits and everybody is on tenterhooks waiting for the final build to come out. When you’re at this stage of the product, every once in a while a change comes in that would be really nice to have because it fixes a critical issue with an important scenario, but it’s just just not important enough to justify cracking open the bits to take the change. Raymond calls these type of changes “Remora Check-ins”. The idea is that if another bug was discovered during the final testing phase that forced us to rebuild the system, we would take these “Remora Check-ins” along for the ride.
In our case, the change we made was a Remora check-in – it was an important bug, but it wasn’t important enough to justify resetting the final test pass. But someone else’s component had a critical bug that HAD to be fixed and our change came along for the ride (and no, I don’t remember exactly what either of the changes were, I just know that our check-in was chronologically the last one made).
Nitpickers corner: None of the information in this post should be particularly controversial – much of what I’ve described here is software engineering 101. There’s always a bar for taking bug fixes in every product – if there weren’t, you’d never ship the product (for example, the Mozilla Foundation shipped Firefox version 3.5 today (congrats!) and they still have several dozens of critical bugs active in their database – I’m sure that these are all bugs that didn’t meet their bug bar). Heck, there’s even a book that’s all about the process of shipping NT 3.1 that covers much of this information.
----
[1] In the past these bugs would be called “Show Stoppers”.