Where, oh where are my bug fixes?

The Visual Studio and CLR teams are huge. If we were to all develop on the same code base simultaneously, we’d kill a lot of time because instability caused by one team’s changes would affect the whole project. To remedy this, groups of developers develop against a specific “virtual build lab”.

There’s multiple virtual build labs, thereby allowing teams to develop against a code base without worrying about changes from other teams affecting the stability of their code base. Periodically, each build synchronizes its changes with the “main” build lab. This is when:

  • The main code base picks up your build labs changes
  • Your build lab picks up changes from the other build labs

The frequency of these integrations is on the order of a few weeks. See the release_team blog for more specific details of the process.

Conceptually, this Virtual Build Lab system works fairly well. However, sometimes you absolutely need a change from a team that’s in another build lab. In order for this to happen, both you and the other lab’s code have to be synced against the main build lab.

The upshot is that sometimes a fix made by one team won’t be propagated to another team’s code base for quite some time. Currently we’re waiting on some stack walking changes made by the CLR team a long time ago. The way around this (if you’re lucky) is to take just the changes you need from the other build lab’s code base and integrate then into your build lab. Often times though, a change can’t be confined to a small, reasonable set of files.

I can’t help but wonder: Is there some better way of doing software development on massive projects with many teams? Specifically, is there a way of getting important changes to the right teams faster, while not compromising the overall stability for all teams?

Comments (11)

  1. Matt Pietrek says:

    You can think of the virtual build labs as branches on steroids. Imagine branching and reintegrating every few weeks.

  2. DrPizza says:

    They don’t sound like they’re on steroids, as you’re apparently incapable of merging between branches; only between a branch and the trunk ("In order for this to happen, both you and the other lab’s code have to be synced against the main build lab").

    Otherwise, if you want a particular bugfix, go to a branch with the bugfix you want and grab the patch that makes up the bugfix, along with any other patches that that patch depends on (this information should be available because bugfixes are grouped into patches and dependencies inferred or perhaps explictly created). This solves the other problem ("The way around this (if you’re lucky) is to take just the changes you need from the other build lab’s code base and integrate then into your build lab. Often times though, a change can’t be confined to a small, reasonable set of files"), insofar as the change is as small as it can be, and available as independently as possible of other changes. I don’t think you can hope for any better than that….

    Now, it may be the case that your revision control system doesn’t offer that kind of fine-grained cross-branch patching. It may even be the case that no revision control system does (though BitKeeper and arch are probably close enough).

    But it’s surely the approach that one should aim for?

  3. The fairly brutal policy in the open source java land is: No Branching (normally), build against CVS_HEAD: http://gump.apache.org/

    It works pretty well, mostly. Any change in any project gets caught within 24 hours, be they bug fixes or bug additions. One big problem with the no-branch policy is that if project, say "log4J" removes a feature from the alpha of version 2.0 as it was deemed unrepairable, all projects that used it in the 1.x line suddenly break; this is exactly what happened last week. A flurry of emails ensues, and what usually happens -post recrimination- is that everyone works around, resulting in code that should work against versions 1.x and 2.x of the library in question.

    The other limitation is that it doesnt do functional tests, and is limited to a couple of system configurations. So problems that only surface on win98 with under 256MB of ram dont show up till beta test time. But overall, the "no branch" policy is pretty good. You just have to check your mail for gump warnings in the morning -before you update your local source tree.

  4. DrPizza says:

    I wonder if one of the consequences of that is the overall pisspoor quality of jakarta projects?

  5. IanC says:

    The granularity may be wrong. A single developer check in often consists of multiple features addressed over a period of time: fix for bug A, fix for bug B, speed up of routine X, new feature Y.

    Even if a single developer works on these separate tasks its often feasible to work on them in different work areas that are each a checkout of the current head.

    When check in time comes you can now see exactly which changes were for which fix/feature, can have a check in description for just that change (rather than a slightly confusing one describing multiple changes), and can far more easily pass just that change to someone else who is half way through a different change and suddenly realises they need yours by diff-ing the previous and resultant versions in your SCC system.


  6. At my previous employer, we were using subversion for version control, for about a year or so. It seemed to support alot of the branch and merge scenarios Dr. Pizza mentioned. Once I "got" the way subversion is all based on transactions, it became easy for me to conceptually understand what was happening in a merge or a branch. I never "got" that with other version control systems I used in the past.
    <br>As I understand it, Microsoft uses &quot;Code Depot&quot;, which is a homegrown version control system, simmilar to CVS? I have caught a few glimpses of some pre-checkin script tool or something. Obviously some time is spent maintain this tool set (the version control tool set.) Perhaps someone should make a tool to help with cross branch merges.

  7. DrPizza says:

    I would think that if a rule such that each fix had its own patch (or perhaps set of patches, if the fix needs a fix) were established and enforced then the granularity issue would go away.

  8. IanC says:

    re DrPizza 25 Jan 2005:

    I may have obscured it, but that was exactly my point. One work area per fix/feature is one (albeit of several) ways to get a per fix/feature patch if the SCCS has change set functionality or some sort of ‘auto label per checkin’ feature is available.


  9. Recently it was my turn to perform BFD duties. What is the BFD you said? Well, let’s start from the beginning….