Parallel Feature Teams working on multiple releases in development. Monthly releases to production.

I recently received a question from a customer who is dealing with the need to support multiple feature teams working on multiple releases in development. They asked how to structure branching to support this environment.

Background:

• My company is, for all practical purposes, a dot com.
• We have a single production website used by all of our customers.
• We do monthly releases of the entire site to production.
• These monthly releases vary in scope:
• One team consists of a couple of developers working on an immediate-need hot fix to production (vCurrent)
• We have many developers and testers actively working on a minor release going out in January (vNext)
• Currently, no one has started working on the February (vNext +1) release yet.
• We also have many developers and testers actively working on a relatively large release going out in March (vNext + 2).
• We have multiple feature teams (three (3) right now, will be going to seven (7) soon) working on different aspects of our product.
• I'm thinking of creating feature branches to allow feature teams to isolate their work until it’s ready for regression testing.
• Currently we have 3 branches - we have named them "Future", "Next" and "Current".
• We do merges (Forward Integration – FI) after each successful Development build and merges (Reverse Integration – RI) whenever a release goes into production. (???)
• Occasionally we create a feature branch for releases that can't go directly into the “future” branch due to active development going on in future.

Recommended approach.

I would characterize your development model as Feature Team and your release model as Monthly Releases. The following illustration shows key concepts of this environment:

Each month, you do a full release of your entire web-site. In the illustration above, these monthly releases are marked v0.1, v0.2, v03, and so on. In the timeline above, one Feature Team works on developing features for the next release (vNext, or v0.1). At the same time, a second Feature Team begins working on developing features for vNext +1 (v0.2). At the end of the first month v0.1 is released and becomes vCurrent. Suring the second month, development continues on v0.2 (which is now vNext). In addition, a third feature team begins development on v0.4. During the second month, there is no development happening on vNext +1 (v0.3). But since vNext +2 is going to be a major release, development on vNext +2 (v0.4) begins in the second month, for planned release at the end of the fourth month on the timeline. At the end of the second month, vNext (v0.2) is released and becomes vCurrent. The vNext release is now v0.3 (but development has not begun on this minor release yet). The next major release is now vNext +1 (v0.4).

During the third month on the timeline, a new Feature Team begins development of the minor release, v0.3 (vNext). A second Feature Team began work on the next major release (v0.4, vNext +1) in the second month and continues this work during the third month. A new Feature Team begins development on the vNext +2 release (v0.5). Note that during the third month, there are three development teams working in parallel on v0.3, v0.4, and v0.5 (vNext, vNext +1, and vNext +2 respectively). At the end of the third month, the minor release (v0.3) is released and becomes vCurrent. The next release is now v0.4 (vNext), and v0.5 becomes vNext +1.

During the fourth month on the timeline, one Feature Team continues its work on v0.4 (vNext), while a second team continues its work on v0.5 (vNext +1). A new Feature Team begins work on vNext +2). At this time, there are three parallel teams working on v0.4, v0.5 and v0.n (vNext, VNext +1 and vNext +2 respectively. At the end of the fourth months, the major release, v0.4 is released and becomes vCurrent. The next release is now v0.5 (vNext) and v0.n becomes vNext +1.

During the fifth month on the timeline, two development teams work in parallel on v0.5 (vNext) and v0.n (vNext +1). At the end of the fifth months, v0.5 is released and becomes vCurrent. The next release is now v0.n (vNext).

The process continues as above. New Feature Teams can be formed, multiple Feature Teams can work in parallel on one or more future releases. One key is that each Feature Team must be working in their own branch so that their changes are isolated from the changes being made by other Feature Teams. A branching model to support multiple Feature Teams working on multiple releases is shown here:

To support the timeline described in the first illustration, only key branching an merging activities are shown on this branching model. There are likely to be additional merges (particularly from the Main branch to its child development (Feature Team) branches that are not shown in the second illustration.

Each of the key branch and merge activities is indicated with a node containing a letter (from A to W).

This is a description of these key activities:

A. The Main Branch is created. Initially it may not contain any source code.

B. In the first month, Main is branched to create a development branch for Feature (Development) Team 1.

C. Also in the first month, Main is branched to create a second development branch for Feature Team 2.

D. At the end of the first month, code for the next release (vNext, v0.1) needs to be merged to the Main branch for stabilization and then release. However, before merging the Feature Team 1 branch to Main (Reverse Integration from child to parent), it is important to do a final merge from Main to Feature Team 1 (Forward Integration from parent to child). Note- this FI merge is not shown in illustration two. After merging changes (if any) from Main to Feature Team 1, test the integration of the code from Main with the code in the Feature Team 1. Do this integration testing first in the Feature Team 1 branch. When the code in the Feature Team 1 is ready to be released, it can be merged to Main.

E. After the code from Feature Team 1 is merged (RI) with Main, it is stabilized in Main until it is ready for release. At this point, when Main passes quality gates, Main is branched to create the Release branch(es) for the next release (v0.1) which is now vCurrent. At this point, the Main branch can be freed up for stabilizing the next release (v0.2, vNext). Note the color change in the Main branch from blue to red takes place after the Release branch for v0.1 is created.

F. Continue doing daily builds in Main. Following a good daily build, merge Main with the Feature Team 2 branch (Forward Integration). Development continues during the second month in the Feature Team 2 branch.

G. During the second month on the timeline, create a new Feature Branch for Feature Team 3. Development on this branch is for a major release. It will continue for a couple of months and will be released at the end of the fourth month on the timeline.

H. At the end of the second month, the code in Feature Team 2 branch for v0.2 is released. As with version v0.1, code should only be merged from a development branch to the Main branch when:

1. 1. The code in the development (feature) branch passes quality gates and is deemed ready to release (or ready to share with other teams)
   2. The code in the Main branch (the parent branch to the feature team branch) has been merged (FI) to the feature branch, and this integration has been tested.
   3. At this point code from the feature branch is ready to be merged with Main, and then shared with other teams or released to production.
   4. Note the color coding on the second illustration. During the second month on the timeline, the Main branch is being used to stabilize vNext (v0.2). The color of the Main branch during the second month is RED. The color of the branch for Feature Team 2 (working on v0.2) is also red. It is important to recognize that Main can only be used to stabilize one release at a time. During the second month Main is used to stabilize v0.2 (RED). You should never merge code from any other parallel Development (Feature) branches into Main. Doing so would bring features from vNext +1 or vNext +2 into the Main branch which is used to stabilize vNext.
   5. It is acceptable, however, to merge bug fixes (hotfixes) or Service Packs from the Release branch for vCurrent into Main. This is often done to bring these changes into the next development release. If you merge changes from a Release branch into Main (RI), these changes will be merged with each of the child branches of Main after the next good daily build in Main.

I. Once the code in Main for vNext (v0.2) is stabilized and passes quality gates, it is ready to be released. Create the release branch(es) for v0.2, which now becomes vCurrent. The next release is now v0.3.

J. As described earlier, the third release (VNext is now v0.3) is a minor release that does not require extensive development. A new Feature Team branch for vNext is created during the third month on the timeframe.

K. In addition a new Feature Team branch for vNext +2 (v0.5) is created during the third month. Development on vNext +2 will take longer than four weeks (the normal release cycle, so development begins during month three).

L. At the end of the third month, vNext (v0.3) is complete and ready to release. Follow the same steps described earlier for integrating changes from Main to this branch (FI), testing this integration, and then merging vNext to Main (RI).

M. After vNext is stabilized, create a release branch for v0.3, which now becomes vCurrent. vNext is now v0.4.

N. Main is now opened up for stabilizing vNext (v0.4). After each daily build in Main, merge Main (FI) with its child development branches (for example, vNext, v0.4)

O. During the fourth month, work begins on vNext +2 (v0.n). Branch Main to create a Development branch for this release.

P. At the end of the fourth month, development on vNext (v0.4) is completed, integrated with Main, and merged (RI) with Main.

Q. After vNext is stabilized, create release branch(es) for v0.4, which becomes vCurrent. vNext is now v0.5.

R. Main is now opened up for stabilizing vNext (v0.5). Merge Main with its child branches after each good daily build.

S. At the end of the fifth month, vNext (v0.5) is merged to Main (RI) for stabilization.

T. After vNext is stabilized, create release branches for v0.5, which is now vCurrent.

U. Continue doing daily builds in Main. Main is now opened up for stabilizing vNext (v0.n).

V. At the end of the month n, vNext is merged to Main (RI) for stabilization.

W. After stabilizing v0.n in Main, create release branches for v0.n, which becomes vCurrent.

As you can see, this is an iterative, incremental development process involving multiple feature teams working in parallel on multiple releases at the same time (vNext, vNext +1, vNext +2). At the end of each month, a new full release of the web site is released to the customer.

There are several keys to success when you are working on multiple future releases in parallel

• Although development can proceed in parallel on multiple releases (vNext, vNext +1, vNext +2), Main needs to be a stable branch used for stabilizing only the next release (vNext). Take care not to comingle code for vNext +1 with vNext before vNext is released (and its release branch is created).
• In this scenario, multiple feature teams are working in parallel. Each Feature Team works in a branch that is isolated from changes made by other feature teams.
• A key concept here is to ensure that a Feature Team needs to be working on an atomic set of changes comprising the feature.
• It is very difficult to isolate some of the changes within a feature team branch and hold them back from a release if part of the team is not ready when the team decides to release a feature.
• Either an entire feature (or set of features in the Feature Team branch) is released together or it waits until they are ready.
• Trying to cherry pick changes from a development branch to do a partial release is never recommended and is likely to be difficult if not impossible to accomplish without errors.
• A key to forming Feature Teams is to ensure that the entire team working in the same branch is not given more work than they can reasonably complete within the timeframe for that release. In our scenario we described major and minor feature releases. Major releases require longer time to build and test, so these feature teams began work two or three months before their release date. Minor releases can be completed in shorter timeframes. These feature teams began work within the month of their release.

The scenario described here has some similarities to Scrum. In this scenario there are monthly releases that somewhat correspond to Sprints in a Scrum project. There are some key differences:

• With a typical Scrum project, where incremental working code is delivered at the end of each Sprint. A complete production release may not take place until several after several Sprints have finished. This scenario assumes we are enhancing an existing web site with new features or bug fixes every month. At the end of each month a new, complete build of the entire web site is released to production and deployed to the customers.
• In a Scrum project each Sprint typically lasts from two to four weeks. Some Sprints may last up to six weeks. At the beginning of each Sprint, a Sprint planning meeting moves items from the prioritized product backlog, forming the Sprint backlog (the work to be completed during the Sprint). To be successful, a Sprint Team can only accept an amount of work into the Sprint Backlog that they can commit to completing within the duration of the Sprint. The Sprint Team should not have the option of delivering a partial set of features (Sprint backlog items). Likewise, the Sprint Team should not have the option of changing the duration of a Sprint to adjust for accepting too much (or too little) work in the Sprint Backlog.
• In this scenario, new releases are shipped every four weeks. Some of these releases are minor releases and can be completed within a single month timeframe. Other releases are major releases. Development of major releases may take longer than one month. For this reason, there are overlapping releases during development, but only one release is shipped each month.