Principles 5: End-to-End Development Process (for Large Projects)


Series Index

+++

This fifth post discusses and end-to-end development process for large software projects.

The only difference between large and small projects is that large projects require more “supporting infrastructure” (more processes, etc.) than small projects, due to the fundamentally imperfect communication between the team members. The “supporting infrastructure” is a tax we have to pay when we develop software. We can minimize it, but we can never really avoid it completely.

Larger projects require larger teams, which (due to accumulated communication imperfections) require larger process tax.

On one hand, we like short product cycles, because they get the product out in the market quickly; on another hand, the shorter the cycle, the higher the ratio between process tax and actual productive time. This is a classic case of an optimization problem, which can be defined as…

“Find the shortest possible project cycle, that keeps the process tax at a reasonably low level“.

Clearly, there’s a variety of other factors that we need to consider when we talk about project and product cycles. To list a few, we need to factor in the general product business model, pre-existing contractual obligations, specific calendar events such as conferences or trade shows, the competitive landscape, etc. For the purpose of this article we can assume that all of these additional factors have already been taken into consideration.

 

Single Project Rhythm

I am a big believer in having a single, transparent, well-socialized and strictly enforced project rhythm. All products that an organization delivers in my opinion should be aligned to this single rhythm.  The rhythm (or cycle) should be short enough, so that it can accommodate all products in development. In this model, different products can take different number of cycles. At the end of every cycle, one or more products get released.

Having a single cycle in my opinion dramatically decreases the complexity (and ultimately the process tax) that a team has to pay. It helps create well-aligned, synchronized and cohesive organizations, reduces dramatically the release management costs of a product, practically eliminates context switching for the individual engineers, drives ownership deep into the organization, improves communication with supporting disciplines (docs, localization, support, sustained engineering), and so on.

 

Agility

In this model, agility is achieved by having an appropriately-sized cycle. For example, for a desktop OS the right cycle may be 6 months or 12 months. In contrast, a web site would probably need to have a release valve every 2 to 4 weeks. Agility doesn’t mean lack of planning – agility means a reasonably short cycle capable of getting high quality products out in a predictable, controlled fashion.

 

Process Summary

The process of developing a large software project can be organized in 4 major phases (see Fig. 1 below):

  1. MQ
  2. Pre-planning and Planning
  3. 1 or more Development Cycles
  4. Stabilization and Release

(A) and (B) happen simultaneously. (A) is “Milestone for Quality” targeted at constructing a high-quality “software production factory”. This milestone gives the engineering organization a chance to optimize and simplify the tools and processes that it uses to product the product. (B) follows a predefined planning process. Planning is a whole other topic in itself. (C) consists of one or more close-looped development cycles.

Development Process Outline

Figure 1  Development Process Outline

Obviously this is just a general outline. The size of the different phases has to be determined by the development organization, based on deep understanding of what needs to happen in these phases and on the current state of the business and of the development factory. Some phases (specifically A and B) may shrink if there is no need for them.

Below I show an actual implementation of this general outline. Each small rectangle represents one week. Weeks in green are “stabilization weeks” (stabilization has different meaning for the different milestones – see tables below). Note again that the size of the actual milestones and phases that I show below are just a specific implementation of the general outline. You can have development cycles that are as short as a week, if you think that the ratio of dev time to process tax associated with managing every cycle is reasonable. You can also have longer cycles, although I personally wouldn’t recommend cycles that are significantly longer that 18 weeks or so.

Development Process - Detailed View

Figure 2  Development Process – Detailed View

With these sizes we have…

  • About 18 weeks (4 months) of pre-planning and planning activities.
  • One or more dev cycles of 15 weeks (3.5 months)
  • Final stabilization of 10 weeks (2.5 months)

I.e. a release with 1 dev cycle takes about 9.5 months; a release with 2 dev cycles takes about 13 months, etc.

The number of the actual development cycles can vary and is a purely business decision. At the end of every development cycle, the organization delivers shippable functionality. When management deems that the available payload makes sense as a next major version of the product, they switch the team to the RTM (RTW) phase, and drive to an actual release.

Before we delve in MQ, it’s important to note that all of these phases are divided in “red weeks” and “green weeks” (a block denotes a week). “Red weeks” are weeks of active development, “green weeks” are weeks of stabilization. Obviously the actual activities that happen in these weeks depend on the phase of the project.

 

MQ

MQ is the Quality Milestone. The role of MQ is ensuring that we have a working high-quality “production factory” before we embark on producing the next version of the product.

MQ and Planning

Figure 3  MQ and Planning

The breakdown of MQ is as follows:

Breakdown of MQ

Figure 4 Breakdown of MQ

MQ is all-inclusive collaborative effort between the 3 disciplines. The majority of PM, Dev and Test will be working on MQ activities during MQ. A few PMs and Devs will be working on planning / prototyping in parallel. MQ is planned and rigorously project-managed. MQ has a set of exit criteria that the org needs to hit before exiting the milestone.

 

The Dev Cycle

All features are developed during the dev cycle, which looks as follows:

Dev Cycle

Figure 5  Dev Cycle

There is a single prioritized feature backlog that captures all features that are being worked on in the current development cycle or can be worked on in future cycles. The list of features that comprise the current development cycle is stable throughout the cycle i.e. Management does not change the list beyond M0 – they have to wait for the next M0.

Any feature that cannot be completed is “cut” until the new dev cycle. Features that take more than one dev cycle to complete will be split into smaller manageable “featurettes” can be completed within the constraints of a dev cycle. Several featurettes may comprise the final feature. At the start of the development new cycle the feature is re-evaluated again against the other features in the backlog.

The start of every new dev cycle is a clean slate. All bugs and features are automatically “cut” and need to be reevaluated and brought back (can be evaluated and brought back during)

IMPORTANT: The development cycle is a contained unit of delivery of added value. We can have as many development cycles as necessary to build a compelling value for the “big release”, but every one of them ends with high quality features that are ready to ship.

The breakdown of the development cycle is as follows:

Breakdown of a Dev Cycle

Figure 6  Breakdown of a Dev Cycle

The M0 to M4 cycle can be repeated multiple times within a release. Every time in M0 (but only in M0), executive management gets to redefine the feature backlog.

 

Stabilization and Release

The final phase in the development process (disregarding the various post-release activities for now) is the “Stabilization and Release” phase or “RTM”.

Stabilization and Release ("RTM")

Figure 7  Stabilization and Release (“RTM”)

This phase is about fixing bugs, integration (if necessary) and managing the final quality of the release. The breakdown is as follows:

Breakdown of Stabilization and Release

Figure 8  Breakdown of Stabilization and Release

 

QFEs (Hotfixes)

Every large software project needs to have a mechanism to deal with hotfixes (also known as QFEs). The nature of these fixes requires reactive delivery (as opposed to the proactive approach described above). Teams typically maintain a separate source code branch (or branches) for QFEs and that is typically the same as the branch that contains the currently shipping version of the software.

Due to their reactive nature QFEs are delivered on demand with a release schedule that looks as follows:

QFE Release and RI Schedule

Figure 9  QFE Release and RI Schedule

Typically, fixes introduced in the QFE branch must make their way into the Main product branch, which is done with a series of regular reverse integrations (or RIs) as shows above. QFEs also get packaged as larger packages and released as service packs (or SPs).

Every QFE is subjected to a root-cause analysis, following a simple standard template, to help the team avoid similar situations in the future. 

 

Templates and Artifacts

Last but not least I wanted to provide a quick list with some of the fundamental documents that are required to bootstrap a development process for a larger product. Note that this list is nothing more than a reasonable starting point – at least according to me.

Key Templates and Artifacts

Figure 10  Key Templates and Artifacts

 

Conclusion

There are many ways to ship software products. Large products require more pampering and more scaffolding, at least until a critical mass of the team adopts a coherent product development culture. If you want to ship high quality products, you need to invest in rationalizing your product development cycle. The best way to do that is to plan, using a series of thought experiments (e.g. “imagine that we have achieved code complete, what are the steps from CC to release ready and how long are they realistically going to take”), and a guiding principle of lowest possible overhead costs and highest possible simplicity.

Comments (4)

  1. tobi says:

    It seems that active development only happens during M2-M3 which account for 6 of the 13 weeks. Does this mean that only about half of the project cost is devoted to dev, and the other half to planning? This seems like a very high proportion. Does the planning deliver that much value or can't it be done any faster?

  2. Ivo says:

    @Tobi:

    M1 which is a 4 week milestone is also a development milestone. The focus of M1 is design and spec-ing. My experience has shown that having a focused investment in design and spec-ing tends to dramatically increase the confidence of delivering a high quality product on time. Having a clear architecture and clear usage scenarios speeds up the implementation phase significantly.

    The general rule I tend to apply is that you can exit a given milestone earlier if you are ready i.e. folks can progress to coding, as long as they can demonstrate that they have done sufficient design / spec-ing work.

    Imagine for example that the goal is to produce a linked list class. In M1, you will prototype, design the interface, demonstrate through clear usage patterns that the interface makes sense, take various fundamentals into consideration (e.g. perf, etc.) and then check in the stubs for your class. You can even write a bunch of automated tests against the stubs (which will of course all fail). In M2, you'll go ahead and write up the implementation.

    Ivo

  3. tobi says:

    This is a great explanation which makes sense. This model must fit especially well to API design but I could imagine doing the same with any kind of user-facing front-end. You could create the UI and stub out all real events with messages boxes. The cost of doing variations would be dramatically lower.

    If you want any input on what to write about, I would appreciate a few comments on specifications. There is a lot of dubious advice around about this topic.

    Working inside of Microsoft seems very attractive because you get to learn to maximize your output and be really efficient. Amazing that such a big and old company can still get things done. Think of Oracle, SAP, …

  4. Ivo says:

    Ah, the good old debate on specs… 🙂

    My preferred approach there is:

    a. Standardize on a simple and concise spec template. Team members can extend the template as necessary, but the basic standard template should be very concise (no more than 3-4 pages).

    b. Move “how to use the template” information out of the template in a separate document. That document can be considerably longer.

    c. Check in the specs in your source control management system. Specs should be versioned and the simplest way to version them (and keep the chances of them being updated high) is to check them in right next to the code.

    d. What's in the spec doesn't really matter to me as a manager. What matters is that all team members who need to review and approve the spec have reviewed and approved the spec and agree to its sufficient readiness before proceeding to coding. This includes development, QA, UX, documentation, legal, architects, various fundamentals owners, etc.

    (a) and (b) help you avoid situations where the template is 40 pages and the final spec is 42 pages, and no one knows how much of that 42-page document is cruft and how much of it is valuable content.