Agile Vendor Management - removing waterfall from outsourced projects

The Tyranny of Waterfall in RFPs

When I was in consulting, we would routinely bid on RFP (Requests for Proposal) issued by government agencies and large corporations where the client would require the project to be performed in the worst possible model: Waterfall. (If you aren't familiar with the Waterfall Software Development Lifecycle Model, you are probably using it.)

In this model, all analysis is done before any design... all design is done before any construction... all construction is done before any deployment... and nothing ever ships on time or within budget. These projects are very difficult to properly estimate. A project size grows, so does the risk of delivering a late project with major features snapped off.

This was a problem as a consulting company, because we were routinely asked to deliver software for a fixed price with fixed requirements (fixed bid contracts). As a result, we were between a rock and hard place: asked to use a software development model that nearly guarantees failure, yet accept all the risk for failure.

Is it any wonder so many consulting companies go out of business.

How did we get here?

How does a bidder force a waterfall model in an RFP, you ask? It's easy! Simply tell your vendor that they must first deliver the analysis document, then the design document, then the code. In that order. Even better, require that each document must be formally approved by the agency before the next step may begin. (this is a recipe for suicide.)

If this nearly always fails, why do it? It's part of a vicious circle that comes from failure and lack of trust.  A project is described pooly and a consulting company, eager to get the bid, responds with a fixed price that is risky.  The project fails to deliver value on time or within budget.  So, as a way to cut risk, the business or agency asks for something of measurable value for each payment.  That's the idea anyway.

The problem is: what artifacts of software development have measurable value?  In the waterfall model, we would state that the analysis is valuable.  We put a dollar figure on it, and manage it as a project.

The same with the design.  It is documented to the hilt, with excessive details dealing with everything from architecture, to message definition, to class diagrams, to statements of how performance can be achieved. 

I won't be the first person to tell you that this process doesn't work.  So if the process doesn't work, how are these documents valuable?  The idea is that you could have company A develop the analysis, company B develop the design, and Company C could use them to develop the application.

This is nonsense.  There are two reasons why:

1. Forced rework and wasted energy.  You cannot overcome the flaws of the waterfall model by introducing competition.  It just fosters the disease.  If the analysis is flawed, then the design team will either have to recreate it, or will produce the wrong design (lose - lose).  Same goes with construction.  The only hope that a company has when they accept a contract of this kind is to build in the costs of performing the analysis and design again under the covers.
2. Nothing has been developed.  The things that really bite a project are the things you don't discover until later, and you cannot discover them by delivering a document.  As a result, this process simply pushes all of the changes (and their costs) to the point when the first deliverable is made, usually half-way through construction.  It is too late to avoid cost overruns at this point.

Therefore, if these documents fail to deliver a working project, then your controls don't work and the project fails.  This engenders more distrust because the agency thought that the vendor knew what they were doing... so they increase oversight and set up payment terms based on delivery of these artifacts, in this order.

Setting up contract terms means assigning a dollar value to each phase.  That creates an interesting question.  If this process is a sure road to failure, what value should the signposts of failure really have?  Not enough to pay for, I'll tell you that.

So why are most software development RFPs out there written to not only encourage this practice, but to create cash flow problems for companies that want to avoid it?  It occurs to me that many of my friends in the government and large corporate circles may simply not know of another way to manage a vendor.

My response to darkness: light a candle

Now that I'm on the buying side (again), I've crafted a new list of requirements for a vendor to fulfill. Of course, this list will be vetted by my organization. So, I decided to put a vanilla list out here, on the web, as a way of providing information for my former clients and other interested parties to see.  I fully expect that my organization will change this list, so this is my way of getting my opinion out unaltered.

The message I want to share is this: provide these criteria to your vendors, instead of the arcane requirements of the waterfall model, and you will encourage your vendors to follow BEST practices, instead of WORST practices... that will make a much bigger difference than you can imagine.

First off, let me say that not all vendors perform all tasks.  I am breaking down the tasks according to analysis, design, construction, and delivery, because it is the model that my former clients are familiar with, and because some vendor companies cannot do all of the steps.  So, when reading each section, realize that each section stands alone.  It can apply to a vendor hired to fulfill that role alone, or to a vendor that fulfills that role as part of a larger process.

However, if you do have one vendor creating requirements and another doing design, and potentially a third doing construction, you have to provide the facility (desks, computers, phones, server software licenses) for them to develop all of the artifacts in one place, because they will ALL be working AT THE SAME TIME.

This is absolutely key to success: the primary effort of analysis is not done until design has released an iteration and code is being delivered, and it has no positive impact if the developers don't meet, know, and trust the analyst(s).

Agile Requirements Analysis

If a vendor provides the analysis team, then add descriptions to this effect to your RFP.

The vendor will deliver a requirements document containing the items listed below. The requirements document will be delivered more than once, in iterations. With each delivery, it will be updated to reflect changes. Changes must also be tracked using a formal change management process. Note: The level of detail for each use case must match the use cases delivered. Therefore, if a code drop contains use cases 1-7 + 12 and 14, the we would expect that these nine use cases will be detailed and described in the requirements document re-delivered with that drop.

The requirements document contains:

1. A use case survey describing every use case envisioned for the final system (one paragraph for each use case in the survey, no more than would fit on a 3x5 card).
2. Detailed use cases for every use case to be delivered in the system. See Alastair Cockburn's book on Writing Effective Use Cases.
3. A detailed list of actors and roles that are used by the use cases, with defining characteristics for each actor.
4. A list of supplemental functional specifications, to include: Accessibility, Security, and Privacy as well as a discussion of the requirements surrounding: Availability, Reliability, Service Level, Scalability, and any Code Re-use requirements that were specified by the users.
5. A structured glossary and list of business rules defining concepts, formulas, variables, constants, and conditions that control the behavior of the system.

The use case survey (step 1) will be delivered before the design document. However, as each use case is detailed, it must be delivered to the design team as part of a weekly deliverable package. The design team is responsible for delivering the design components to the analysis team and the construction team in a joint weekly meeting.

The analysis team will meet regularly with members of the client staff, design team, and construction team. For larger projects or remote projects where the project manager may not be able to attend, minutes of these meetings must be produced and filed with the project office showing attendance and results.

If the analysis team begins a use case, it must deliver results for that use case every week until it is complete. The results must be understandable and useful, but do not need to be complete. The analysis team must be aware that design and construction will begin immediately after the first iterative delivery of a use case. Therefore, the use case should contain enough information to assist the design and development team as much as possible.

The vendor performing the analysis should expect that analysis activities will not complete until well after the first stable code drop is accepted by the customer. However, the headcount needs will diminish dramatically between the first proof of concept deliverable (at peak) and the release of the first stable drop, when a small fraction of the expected team would be required.

How not to do analysis as you would in a waterfall project:

If you notice from my text above, we don't require all of the analysis up front. In fact, don't require that any of it is delivered before the first code delivery. That's right... code first. Caveat: that code should be "proof of concept" code... completely disposable but eminently useful.

By letting your consultants write code before all analysis is done, they flush out further requirements, discover snafus in their ideas, get infrastructure in place, and push risk from the system, early. Wasn't that the reason we wanted all this stuff up front in the first place?

Agile Design by Vendors

If a vendor provides the design team, then add descriptions to this effect to your RFP.

The vendor will deliver a design document that contains the following items. The document will be re-delivered with each iteration, updated to reflect changes. Note: it is not important for this document to be 100% complete until the code complete milestone. It is, in fact, preferable, that only portions of the document are delivered with each iteration.

1. A logical data model and data dictionary illustrating the data types, ranges and the relationships between important entities.
2. A high-level component view, showing individual components and indicating which components communicate with each other
3. For systems involving multiple servers, a deployment view, showing which components are deployed on which servers in a fully scaled environment.
4. Depending on the complexity of the components: a layered component view, diagramming which classes or modules exist in each application layer or tier.
5. A short description of the responsibility of each component.
6. A short description of the interfaces that components use to communicate. Detailed descriptions are required of any XML or flat file formats used to encode and transmit data.
7. Screen shots or wireframe diagrams if the user requires them, illustrating how the functional interface will be laid out.
8. For web applications, an interaction diagram will be provided, illustrating the navigational flow that a typical user will use to flow through any groups of sequentially ordered pages (like wizards, form fill-in, and payment/checkout).

The goal of design is to progressivly develop an approach for describing the components of the system, their responsibilities, and how they interact. Therefore, with each delivery by the analysis team, a series of diagrams will be produced by the design team (within a week) that illustrates how the newly delivered requirements affect the current expected design. The first iteration of design begins after the use case survey is delivered.

The results of the design effort are presented at a weekly joint meeting with the developers, the customer and the analysis team. The goal is to flush out misunderstandings and see if the customer is likely to approve of the technical approach. After the most useful use case (as judged by the client) is delivered to the design team, and the design team develops diagrams to support construction, construction will begin.

The vendor performing the design should expect that design activities will not complete until well after the second stable code drop is accepted by the customer. However, the headcount needs will diminish dramatically between the first stable code drop (at peak) and the release of the second stable drop, when a small fraction of the expected team would be required.

How not to do a Waterfall design

Similarly to how we did the analysis, we will require that the vendor produce a design.  However, we will not require all of it at once. In fact, we do not require more than the mere framework to be delivered until a period of coding is complete (often a month of coding). Then, ask for all sections to have some detailed content. With each iteration, EVERY section should become more detailed until that section is completely specified. (It doesn't matter which section is completed first, just that they are all completed along the way).

Also, require that the developers doing the code are updating this document, but that they are not spending more than 10% of their time on it. That means, in a typical month, they would spend no more than two days working on the document. The rest of the time is to be spent writing code.

What if that's not enough time to do a "good job" on the document?

Guess what? The document is supposed to be a communication mechanism to allow developers to speak to each other... not to you. If the developers finish writing code, and didn't need the document to have more in it, then the document is good enough.

Really. It's OK to write down only what you need to convey and communicate.

Of course, the closer your users can get to the development team, the fewer words you have to write down. Users nearly always do a better job of explaining what a system should do than developers do. After all, they are paying for it, using it, and counting on it.

What to require during Software Construction

If the vendor is involved in the construction of the system, then place text to this effect in your RFP.

During construction of software, the vendor will be required to adhere to the following practices:

1. All code will be delivered with unit tests using either the NUnit, JUnit, or CPPUnit unit testing framework. The unit tests will exercise every step of each delivered use case that is part of the “normal case” or “sunny day flow.” Unit tests will test the handling of expected exceptions. Unit tests will verify the validations applied to the user interface by invoking assemblies used by the user interface. It is not necessary for unit tests to verify the mapping between object properties and database columns.
2. All code will be checked in daily and will be compiled at least twice a week. This is under the contractor’s control. However, the client may require build reports to be transmitted for each build as a verification that this is occurring.
3. As a best practice, the vendor should automatically invoke the unit tests for all checked in code immediately after the code is compiled. The results of the unit test, if this step is taken, should be included in the build report. It is not important that all unit tests pass in each build. Any failures that accompany the delivery of demonstrable code should be documented, so that we can verify properly installation.
4. The vendor will deliver a version of the application with demonstrable functioning code at least once a month. The version does not need to represent a complete set of features. In fact, we would expect that it would not, especially in early stages of construction. The vendor will demonstrate this code, on client servers, to client customers and IT personnel, within a week of each delivery. The vendor will be prepared to describe the functional additions to the system, or the refactoring that took place during the previous month.
5. Each iteration of deliverable code shall use a Setup project to install it. Copy and Paste installation is only acceptable where there are serious and known issues with using an Installer.
6. Each iteration of deliverable code shall include a full copy of the source tree.
7. At least once in every month, the vendor will perform a design review of code being delivered during that month, with at least one client employee present either by NetMeeting or in person. The design review may cover topics in data architecture, class architecture, module architecture, messaging, reliability/scalability, security/privacy, accessibility/usability, as needed by the application team during that iteration.
8. The vendor will either follow the client's coding standards document, or will deliver a coding standard to the client for review and feedback before commencing the construction phase in earnest. The code that is delivered must match the standard as agreed. If the code standard originates from the vendor, and the client determines that the standard is not sufficiently detailed, or does not sufficiently protect good coding practices, the client may require the vendor to conform to their coding standard.

How to avoid Waterfall construction

You can see the iterations in this process, much more so than in the Analysis and Design phase. That's because we assume you have to do SOME analysis and SOME design before the construction team delivers usable "final state" code. This means that each iteration contains an update to those documents. Code, on the other hand, doesn't get done until it's done. As a result, the iterations are much more stark.

All the same, each iteration must be time-boxed.  There is a fixed (short) period of time in which to deliver. If you have to choose between shipping all features and shipping on time, drop a feature. Pick it up in the next iteration. Do Not Slip.  

Even more importantly, each iteration must contain a representative of every deliverable artifact. In other words, if you are planning to deliver an online help document, you should deliver one with each iteration, even if the first three times you deliver it, it doesn't contain that many articles.

Don't be tempted to say "it's a short project... we only have time to deliver once!" If the project lasts more than a week, you have time for multiple deliveries.

Remember: deliver deliver deliver

Intelligent Software Stabilization

If your vendors are involved in the final delivery of the software system (nearly always true), then place text to this effect in your RFP.

The construction phase transitions to stabilization when all use cases required for the delivery of the system have been coded to completion. This determination is made at the sole discretion of the construction team in conjunction with the customer. 

At this point, one more full code iteration will occur for the construction team to stabilize the code base and refactor out inefficiencies and correct mistakes made in the implementation of patterns and algorithms. The functionality at the end of this iteration is not expected to change much from the prior iteration. This is placed specifically for the construction team to improve quality, fill out remaining (boundary and exception) test case code, and make the system more resistent to events that could cause failure.

The delivery of the first drop in the stabilization phase will occur to the client systems at this point, and user acceptance testing will commence. The users understand that the system may not be stable at this point, and the users will have been involved in seeing code releases before this point. However, this phase will allow for minor (cosmetic) changes to the functionality as well as the repair of defects. 

All testing is expected to be conducted on the client's site, and will involve client resources in individual contributor roles. It must occur on hardware and software that is a good model for the equipment and configuration that will be used in a production environment. The stabilization phase will include the following specific deliverables:

1. All configuration management will be handed over to the maintainence team (client IT resources) who will be provided documentation and training on how the system works, how to fix expected errors, and how to troubleshoot issues.
2. All user documentation will be delivered to the client and at least one training session will be conducted (for train-the-trainer exercises) or more (for train the user exercises). 
3. Performance, Stability, and Scalability testing results will be delivered to the client showing that the system will operate within the parameters defined during the analysis phase.

The system will be considered complete after the users accept the functionality as being "correctly interpreted from the user cases as delivered". Note that any expensive changes that are first raised during user acceptance testing will be considered a change request if the client organization could reasonably have raised the same issue during one of the demonstrations held during the construction phase. 

Conclusion

The text of an RFP is wildly important to how the entire project will be run, since it drives financial terms, measurement criteria, and even decisions on the part of experienced consultant teams on whether to even bother bidding on the contract.  If the description of the terms of software development are carefully described to encourage agile software development processes, then the client company and the vendor will both benefit from a more reasonable and more orderly process, where information is collected early and presented often in a valuable manner.