Agile Makes Problems Visible

Update: this blog is no longer active. For new posts and RSS subscriptions, please go to

It’s often been said that agile project management techniques aren’t a silver bullet that will magically solve all of your problems.  In fact, their biggest strength lies is not in solving problems, per se, but rather in exposing your buried problems and making them visible so you can deal with them.  For a team switching to Agile it can feel like things are getting worse, not better, but that’s often because they’re being made aware of pre-existing problems they didn’t know they had.  It may not seem like it but this is actually a good thing.

Here’s a real-world example.  My current team of six people was recently assembled to build tools for an internal customer.  I and one other member have a decent amount of experience with agile approaches but the rest of the team does not.  I suggested that everyone read the free ebook Scrum and XP from the Trenches and think about whether they wanted to try that approach.  Everyone readily agreed that our situation seems tailor-made for Scrum so we decided to go with it.

Our team manager did everything right to prepare for our first sprint planning meeting a few weeks ago.  He made sure we had a clearly identified product owner, a product backlog of user stories, and he even printed out each story on a separate sheet of paper and taped them up all over a long whiteboard so we could discuss each one and write notes around them and on them as we made our plans.  I was quite impressed.

We started the meeting and after getting through the preliminaries we turned our attention to the first user story on the backlog.  The idea was that we were going to talk through about 20 stories to get a feel for the overall strategic direction of the project but within a few minutes we went off the rails.  The conversation between the team and our product owner went something like this (not exact quotes, obviously):

“So the first story says the users want to enter information about avails into the system.  That information includes these six fields . . .”

“Wait, why those six fields?  That doesn’t make any sense.  I thought three of them were part of rights, not avails.”

“No, that’s part of avails.  An avail means [these things].”

“Dude, that’s not what avail is.  You’re mixing it up with rights, and maybe pricing rules, too.”

“What?!?!?  Then what the hell is an avail?”

We spent most of our three-hour meeting getting bogged down in the definition of terms and concepts, not getting anywhere near the original goal of walking through all of the stories.  At some point during a painful, aggravated silence our team manager gave me a look that said, “Ok, now what, Mr. Agile Genius?  You said this was supposed to work.  We were supposed to finish this meeting with a  sprint plan and that ain’t happening.”

I raised my hand and said to the room, “Ok, point of clarification here.  Right now it feels like this process isn’t working, right?  We’ve been going around in circles for two hours because we’re not all using the same language.  There are several deep misunderstandings that are preventing us from making progress.  However, the sprint planning process is doing exactly what it’s supposed to be doing.  It made us get together in the same room, talk to each other, and discover that we still have some groundwork to do before we can can even have this planning conversation.  The Scrum process didn’t cause this problem, just just exposed the problem.  Not having a shared understanding of the domain concepts is a problem no matter what process you use.  It’s far better that we discover this problem right here, right now, rather than write a 50-page spec and do three months of dev work around broken concepts before we realize we’re going in the wrong direction.”

Fortunately we were able to work through that particular problem and come to agreement on a shared vocabulary by the end of the meeting.  We’ve had a couple more rough spots crop up since then but in each case we’ve identified the source of the problem, solved the problem, and moved on.  Progress has not been as quick or as smooth as we imagined it would be but that’s not because Scrum isn’t working for us; it’s because we had a lot of problems to work through and those problems were being made visible quickly so we could identify them and solve them.

Software development is hard work.  I doubt we’ll ever find anything that’ll change that.  There are no silver bullets.  But some processes are better than others for pointing you in the right direction and letting you know what problems you need to solve.

Comments (4)

  1. seattlec++ says:

    success belongs to agile, and failure belongs to traditional process

    It's one of the Big Lies of agile development. As you pointed out, any process that required you to meet would have exposed this lack of common understanding. Doesn't matter whether it's writing a spec or reading user stories (which are just another way to write a spec, and somebody did it before the first meeting). Sheesh.

    Iterative development has its good points, if you use the feedback. Test-driven development has its good points, if you write real tests. Pair programming — hmm, probably too wasteful to be useful except for training noobs. Scrum, ok if it's a choice between scrum and nothing-at-all.

    Agile isn't a magic bullet. It's scarcely a weapon at all, unless you know how to shoot it. And if you do, it's a very small, very technical difference to traditional methods. Don't be oversold by agile consultants and agile True Believers. It's just a tool, and it's only as good as the team that wields it.

  2. SaintGimp says:

    @seattlec++, Yes, a formal agile process like Scrum doesn't have an exclusive lock on face-to-face communication.  You could do a straight-up Waterfall process and still quickly discover the particular problem we were having (misunderstanding about core concepts in the domain) if everyone made a point of communicating with each other and listening/reading closely to what everyone else was saying.  But in an environment where people aren't forced to have face-to-face conversations that communication often breaks down.

    Maybe it's just me, but I've seen all too many projects where a PM or arcitect writes a huge, verbose spec, sends it around for review and comments, and people don't really read and comprehend it.  They just kind of scan through it and go, "Yup, that's a spec, all right".  People's brains just turn off when faced with 50 pages of blah, blah, blah.  It's not until a whole bunch of code has already been written that they finally realize that there are huge misunderstandings about the basic concepts.  I agree, the solution to that problem isn't "do Scrum as described by Schwaber", the solution is to come to a real shared understanding.  But without a framework that forces you to have face-to-face conversations, human nature dictates that we probably won't get a shared understanding until disaster has already struck.

    The practical value of Scrum (relative to this problem area) is that a) you get the dev team and the customer in the same room talking to each other, b) you have the "spec" broken down into bite-sized, approachable pieces that can be intuitively understood and discussed, and c) you focus very tightly on the work that's going to be done in the near future and not so much on work that's coming much later.  Sure, you can make those things happen in any process if you're both talented and lucky, but my experience has been that they're much more likely to happen in a process that's designed to make them happen.

    I hope I haven't communicated in this blog that "success belongs to agile, and failure belongs to traditional process".  It's obviously not that black and white.  You can succeed with traditional processes and you can fail with agile.  But, I have to tell you my career in software development has been long enough to see my fair share of both successes and failures and I believe that agile techniques will shift the odds in favor of success.  Do they make for a guarenteed success?  No.  Do they make a statistical difference?  Yes.

  3. Ken says:

    After working in a couple of Agile environments my take is that Agile is a newer, faster way to mismanage projects.

    What is really needed for a succesful project is: clearly defined goals and specifications (the requirements), managers and workers that understand the goals and specifications, workers that are technically skilled enough to turn specifications into a working solution, and a realistic amount of  time to produce a workable solution.  Robert Heinlein said "they didn't want it right, they wanted it by Wednesday".

    Usually the problem isn't about managing the workers, sure there are some people that pump out pages of code to look busy, or there are people that surf the web all day –they can be managed, the problem is to clearly define when the solution is completed. What I mean by that is that no solution is ever good when managment allows the project to run out of control by becoming a moving target, or when feature after feature is added at the last minute. Clearly defined milestones and clearly defined releases need to be managed in order to have a sucessful outcome. No amount of "process" such as Agile can fix a project that has gone out of control at the fundamental level.

  4. SaintGimp says:

    @Ken, thanks for your feedback but I guess I have a different perspective on it than you do, perhaps because of the types of projects I've worked on in the past.  Different industries have different types of software projects with different success metrics and probably influences how various people respond to the agile value proposition.

    You talked about creating clearly defined goals and specifications and clearly defining milestones and releases.  In some circumstances that works fine; it's feasible to get everyone to agree up front on "Yes, this is what we want," and success is defined by whether the development team delivers what was originally agreed upon.  If that's the kind of environment in which you work, then that's great.

    However, as I've written before, for many (most?) kinds of environment that's simply not possible.  We'd like for it to be possible but it's not, much like how I'd love to have an accurate weather forecast today for the evening of December 20th so I could know whether or not to plan on observing the lunar eclipse that's going to occur then.  We don't know how to forecast for specific weather details that far in advance so I'm just going to have to wait until a couple of days prior to the event before I'll know for sure.

    In the same way, for most software development projects it's simply not possible to correctly define all of the requirements up front.  Oh, sure, you can write down a bunch of stuff and treat it like a contract but at the end of the day, for most software projects, success isn't defined by whether or not you delivered what was originally agreed upon.  Success is defined by whether your software solves a real-world problem, or delivers business value, or sells well in the marketplace.  It matters not a hoot whether you deliver what was agreed upon; what matters is that the software does what the customer wants it to do at the time that they receive it (which may be very different than what they thought they wanted when you started).

    So if for most software projects you don't know the ultimate goal when you start, the only thing you can do discover and refine the goal as you go along.  That implies that you're going to be dealing with lots of change.  Agile processes are designed to embrace that change and to create a structured framework that allows change to occur without allowing change to devolve into chaos.  Agile processes make the costs of each change very explicit so at that any step along the way you can tell your customers, "Oh, you realized that you actually want something slightly different than what we originally discussed?  Ok, no problem, we can make that change, but here's the quantifiable impact it'll have on the other features and on the release schedule.  You decide whether it's worth it or not.  The code is always in a shippable state so we can ship whenever you decide that it does what you need (or we run out of time/money) – just say the word."

    A few background posts on this topic:…/formalism-vs-hermeneutics.aspx…/software-development-is-like-weather-forecasting.aspx…/irreducible-complexity-in-software-development.aspx