I've been meaning to write a blog post to tell this story for a while, and I figured that now that we've passed the 5 year anniversary of when it happened that I should finally take some time and type it all out. Before I describe the details of the problem and what we did to fix it, I need to set the stage a little bit with some background information.
At the time, I was working on Visual Studio .NET 2002. I joined Microsoft in 1999 and at that time, the team had already been working on VS .NET 2002 (then called Visual Studio 7) for about a year. When we got to the summer of 2001, we finally finished with VS .NET 2002 beta 2 and were finally feeling like we were getting close to shipping a new version of Visual Studio for the first time since 1998. This was a big milestone for the team because it had been so long since VS6 and a lot of the team members hadn't yet experienced shipping a product at Micrsoft.
To help get people excited about the upcoming release, our marketing team arranged to distribute 2+ million copies of beta 2 for free in upcoming editions of various programming magazines (such as Dr. Dobbs Journal, etc - I still have my souvenir copy of one of these magazines that I bought at a local bookstore becuase I hadn't yet had the experience of seeing a product I worked on available on a store shelf). We signed off on beta 2 at the end of June and were about to manufacture all of the DVDs needed for these magazines, which were going to hit the shelves in August.
As a reward for shipping beta 2, the teams were given a couple of extra days off. That year, the 4th of July fell on a Wednesday, so they gave the team members Monday and Tuesday off so that everyone could have a 5 day weekend. The setup/build team could not all take the same days off because some of our activities were mission-critical and couldn't be skipped for 5 consecutive days, so our team was given the choice of taking off Monday and Tuesday or Thursday and Friday. I happened to choose Thursday and Friday, so I was in work that Monday, July 2nd. Most of the team chose to take Monday and Tuesday, so we were working with a sort of skeleton crew.
Unfortunately, we came into work on Monday morning July 2nd to a mail in our inboxes from one of our team members in Japan asking why he wasn't able to launch Visual Studio. When he tried, he got an error message saying that the product had expired on July 1, 2001 and thanking them for participating in the beta program. A couple of quick repro scenarios confirmed that we had a big problem, and we literally had to stop the presses from creating more DVDs because otherwise there would be 2 million magazines on the shelves in August with useless copies of Visual Studio .NET 2002 beta 2 in them.
We did some root cause analysis to locate what was causing this timebomb that we hadn't seen in previous testing. We had recently implemented some new timebomb and activation scenarios and also created a new Professional edition (previous betas for VS .NET 2002 all shipped the Enterprise edition). As part of testing the timebomb functionality, we picked up some technology from the Office team that they used for their activation features. Part of that technology we picked up was a test licensing file that specified a beta timebomb that expired on July 1, 2001 (this was the timebomb date used for a beta version of Office XP from a couple of years prior to that time). We originally implemented this timebomb and activation logic in the new Professional edition because all of the teams in Visual Studio were focused on the Enterprise edition and we could work through setup-related issues without disrupting day-to-day development and testing for other teams in Visual Studio. Towards the end of the beta 2 ship cycle, it was decided to ship the Professional edition instead of the Enterprise edition that we had originally planned to ship. The team had not yet fully understood some of the settings in the licensing files, so we did not realize we needed to test some scenarios where we set the system clock ahead to check for inadvertant timebombs.
With all of the above information, we found ourselves in a situation where we needed to rebuild the Professional edition MSI with a new licensing file to eliminate this timebomb, then re-test and sign-off on the updated version in time to allow publishers to include copies of the beta 2 DVD in the next issues of their magazines. To top it all off, we needed to do all of this with a skeleton crew because most of the team was on vacation as a reward for finishing beta 2.
In order to do this, we decided to surgically patch the licensing files into the existing build and then repackage everything and try it out. We also decided that we need to change the product code so that we could block users from installing this non-timebombed build on a system that already had the timebombed build installed. This required some additional work because we had to locate the places in the setup data files where the product code was referenced and change them all to the new code, plus we had to add the old product code to the block list built into setup.
We tried a couple of scenarios and kept running into issues where we missed instances of the product code in one of the data files, so we decided we needed to re-run the post-build part of the build process so that the data files would be automatically updated. However, we had to do this in such a way that a full binary recompile did not happen because we wanted to try to not invalidate the product testing that had been done before the holiday weekend. This post-build process took a couple of hours so we basically had to launch it and sit and wait and hope that the resulting setup package worked. The first attempt showed us one place where we forgot to update one of the data files, so we had to make a fix, restart post-build and wait a couple of more hours. Finally, we produced a build that worked in all of the scenarios that we needed to support on late Tuesday afternoon, July 3rd.
From there, we had to figure out how to get the teams to test and sign off on this updated build, and then get it released to manufacturing again. My friend and I ended up camping out in the burn lab on the 4th of July producing gold master DVDs that the test teams could use for their sign-off testing, running through test scenarios and helping other teams with their scenarios. At that time, Visual Studio only planned to ship on CD, but we had to produce a DVD to fit in a magazine slip-cover, so we had to use our single, flaky DVD burner to produce DVDs one by one and hand them to teams for testing. We also had to create some detailed documentation explaining what the problem was, how we fixed it, and how test teams needed to exercise the new functionality because there was a lot of confusion as teams tried to catch up on their email from the holiday weekend once they realized the seriousness of the situation.
Fortunately, by the time the majority of the team got back to the office on Thursday the 5th, the heavy lifting was done and we only needed to gather the final sign-off and hand deliver the golden master DVDs to the duplication facility across campus.
Because of all the manual tweaks that myself and a couple of other people had to do, I basically spent the next month or so not sleeping very well and dreading reading my email for fear of some scenario cropping up that we hadn't accounted for. Everything felt so rushed, so I was sure that there must be something we would end up missing. In the end, it turned out fine, and I learned a ton about how our setup worked behind the scenes that I was able to apply to future situations (writing better tools, automation and test cases, helping customers find workaround to odd behaviors in our setups, advising the team about design decisions for future versions, etc).
Looking back, this scenario is when I really started to feel like I belonged at Microsoft and could fit in and contribute. It gave me a lot of confidence that I could really make a difference and help solve tricky problems on my own and not need to always ask for help from my manager or other more senior members of the team.