Motley says: "Management suffers from the "Crystal Ball Syndrome" when it comes to project planning"

Summary

Motley:  Management wants clear predictability of all the features we will be delivering in a release. I don't have a crystal ball, and agile removes the need for one.

Maven: Balance long term planning with short term iterations. Agile can play along in a waterfall organization. Use Wideband Delphi to help with predictability of the long term and plan and commit to the high priority features that you have high confidence in delivering.

______________________________

[Context: Motley's team is trying to be agile, but also forced to fit within the model of the overall organization, which isn't agile]

Motley: Ahhhh, the trials and tribulations of trying to be agile in a waterfall world. My frustration factor is above the red line today.

Maven: What do you mean?

Motley: We've been discussing agile development for months now. You know, short iterations, shippable quality at the end of an iteration, frequent retrospection, and all that stuff. So far, it has been working out great on our team, but our company as a whole is more waterfall-oriented. Our team is agile, but Marcus' team is not, and the overall release schedule is waterfall in nature. The management team wants a concrete prediction - no scratch that - a crystal clear picture  of what we will be delivering a year down the road.

Maven: Ah, yes. The "crystal ball syndrome". Management wants to know exactly what features are going to ship with the product. They expect detailed estimates that they want the development team to hold themselves to. This scenario is fairly commonplace.

Motley: After practicing agile for a few months, this kind of thinking doesn't seem realistic.

Maven: Perhaps, but keep in mind that Cynthesis also is a business that has to set expectations with marketing, packaging, customers and dependencies. Our partners, for example, are shipping our software on new hardware they are developing and need to have an expectation as to when our piece is going to ship so that schedules can align.

Motley: I guess, but it still doesn't help me predict the future!

Maven: True. We have to balance long term planning (multiple months) with short term planning (a 2 week iteration). Relying exclusively on long term planning is destined to fail as plans always change. Relying exclusively on short term planning does not work for our business.

Motley: It's a no-win situation. I quit.

Maven: I know you are kidding. Trying to be agile in a waterfall world is definitely a challenge, but one that is surmountable.

Motley: The long term plan seems to be a safety blanket for the project. The comfort is not grounded in reality because that long term plan is going to change and we cannot possibly predict all the flaws in the project up front.

Maven: That's where agile comes in. Your long term plan takes the form of a prioritized product backlog (when using Scrum). The keyword here is prioritized. As the environment around you changes, priorities may change, as we discussed previously. There are certain features that you can say with high confidence that the team is going to ship. Communicate these to the project management team. Be conservative. Avoid providing a full plan from the start of the release to the end of the release. And if anyone ever asks you for a Gantt chart depicting the entire release, run for  your life. That thing will be out of date before it hits the printer.

Motley: There's still a problem. Even with communicating the inclusion of the top priority items in the release, I still don't have that crystal ball that tells me that I can even deliver those features. We're back to square one.

Maven: Remember we talked about Wideband Delphi (WBD) estimation? WBD is perfect for this scenario. Using a collaborative estimation technique early in the project when there are lots of unknowns gives you a little bit of confidence in predicting how long the top priority features are going to take. We have been very successful using it in the past.

Motley: You still end up with estimates. We still cannot predict the project long term.

Maven: C'mon, Mot. Be realistic here. Of course you cannot fully predict the entire release. Estimation is a fact of life for a development team. There's no getting away from that. What we want to do is improve our confidence for the top priority items and adjust as we go for the lower priority items.

Motley:  Oooo, getting a little touchy there, Mave! All that is great, but again, I still have to live in a waterfall world as that's the holistic model for the organization. I have to produce full plans. I have to participate in lifeboat drills where we are forced make hard feature cuts. The organization itself plays schedule chicken rather regularly.

Maven: Let's deal with those things in turn. Regarding full plans, you are going to have to do a high level plan for the release so that you can, for example, predict the number of people needed to deliver the feature set. Plan at a very high level, don't do full investigation of the features, and make sure your prioritization is right for the customer. Communicate this high level plan out to the release manager. Make sure your iterations align with the waterfall-oriented milestones. If the release manager is uncomfortable because there are no details, tell her the details will come on a regular basis as you do your iterations.

Motley: So what you are saying is that we do just enough high level planning to set some core expectations, but we don't succumb to the waterfall model and do an extremely detailed plan complete with Microsoft Project Gantt chart, and instead communicate details as we iterate. Sounds reasonable.

Maven: Precisely. As for lifeboat drills, whereas Marcus' team is going to cut down on features, your team is going to treat it differently. You are going to review priorities of features and ensure your product backlog is stack-ranked accordingly. You can communicate which features you are still highly confident in delivering, and where the cut line is on the product backlog. Also communicate that things may change as you go and you will adjust just-in-time.

Motley: Aren't we still setting expectations about exactly what we will deliver?

Maven: Only for features at the top of the list. The focus is on reviewing priorities.

Motley: That will be a hard one to sell.

Maven: You also used an interesting phrase: "schedule chicken". Can you clarify what you mean by that so we are on the same page?

Motley: The way I look at it, it's like chasing a chicken. You are never going to catch it and always be behind. The management team uses phrases like "creative scheduling". That's just code for fooling yourself that you will meet your dates when it is highly likely you won't.

Maven: Ok. Another variant of schedule chicken is this: I am in a car travelling at top speed towards another car. That second car is the schedule. The contest revolves around who flinches first. I try to force the schedule to flinch first, but it never works out that way and it gets the best of me.

Motley: Nice. I hadn't heard that one before.

Maven: Massaging dates without much rationale is a recipe for disaster. Remember the main project management variables:

• Time
• Features
• People/Resources
• Quality

We never play with quality - remove that one. Releases are often date-driven, so time is static. People and resources are fixed early on in the schedule. Remember, adding people to a late project will only make it later. That means the logical lever to pull is features.

Motley: We buffer the project with scope and deliver more or less depending on the health of the project.

Maven: Exactly. One more thing: a key to making this happen is ensuring you have shippable quality at the end of every iteration. That helps you stay clean and potentially ship at any point in time (this isn't completely realistic on most projects, but is a goal to strive for). By doing that you have more predictability of your long term plan.

Motley: Got it. I also have to make sure I act as the filter for the team and communicate with management in terms that they understand even though we have our own model. I refuse to use Microsoft Project, though, so if that has to happen, it's up to the project manager.

Maven: A necessary evil of being a lead is dealing with a bit of politics. You'll do fine.

Motley: You're having a good day, Mave. All sorts of good advice. Glad to see you've been taking your medication once again.

______________________________

Maven's Pointer:  Motley does not have many good things to say about Microsoft Project, but it can be a useful tool when used in the right way. There is nothing wrong with keeping a very high level schedule in Project. List off features at the highest level and milestones that the team as a whole must meet. Individual developers should not be listed on that schedule. Keep the details for your sprint backlogs, if you are practicing Scrum.

Maven's Resources: 

• The Mythical Man Month: Essays on Software Engineering, 2nd Edition, by Frederick P. Brooks, Addison-Wesley Processional, ISBN: 0201835959, August 1995.