Agile development and the Software Researcher

A while back I came across this post about agile.

I originally was going to write a comment about that post, but I think the commenters have done a good job at that. I will point out that the environment described in the post is a lot like many of the internet boom startups, few of which are around right now.

But in writing some thoughts down, I realized that I was really writing about why I found agile to be interesting. So that’s what this post is about.

I’ve worked on a ton of projects, some short, some long, some small teams, some big teams. But fundamentally, writing software is an exercise in:

  • Making progress in the face of imperfect information


  • Managing change

These shouldn’t be a surprise to anybody. When we start coding, we always have imperfect information. We don’t know if our design will really meet the customer’s needs, we don’t know how long it’s going to take us to write, we don’t know if our architecture is feasible to build, we don’t know if it will perform well enough,  etc. And things will change along the way. Market conditions will change. Customers will modify their priorities. New technology will become available. People will join or leave the team.

There’s a name for trying do something in that sort of environment, and it’s not “engineering”.

It’s research.

Applied research, to be specific.

And doing research requires a different approach than engineering.

Way back in 1961, President Kennedy made his famous “moon speech“.  At that time, the only US astronaut was Alan Shepard, who had flown a 15 minute sub-orbital flight. The ability to send people to the moon seemed out of reach. But the leaders at NASA came up with a plan – they decided on three different programs: Mercury, Gemini, and Apollo. Each program had a theme and a set of goals, and each flight had a set of smaller goals.

Sometimes things worked, sometimes they didn’t. After each flight, they looked at what had happened, decided how to adapt, and went on to the next flight. And over the span of 8 years, they accomplished their goal. All by following the “design a little, build a little, fly a little” approach.

After Apollo, NASA decided to build the shuttle. They did a lot of design up front, made a lot of promises (both schedule and capabilities) and then started building. They ran into lots of difficulties with the main engines (ultimately solved), and the thermal protection system (not really solved). Ultimately, they finished the vehicle, but it took years longer than they expected and ultimately didn’t do what they had designed it to do or what they needed it to do. And it has very high maintenance costs.

The analogy to software development projects should be obvious. Shuttle is like most big software projects – lots of planning up front, lots of promises, followed by delays as problems are addressed, and then the ultimate release of something that doesn’t measure up to the original idea. We convince ourselves that “this time it will be different”, despite the fact that the same thing has happened every time we take this approach.

Incremental development accepts the realities of the environment, and takes a different approach. You know that you don’t know enough to finish the project and you know that things are going to change, so you bite off a manageable portion – say a month’s worth of work – do enough design, write some code, and then see how well it works.

And after that month, you modify your plan based on what you learned.

Design a little
Code a little
Fly a little

That’s what agile means to me.

Comments (10)

  1. Jay Bazuzi says:

    Right on, brother.

    Yegge seems to think that Agile is for the imbiciles.  He’s not alone.  I’ve seen a lot of people look at pairing & TDD as ways to make up for having weak devs.

    What I see is a way to make great developers even better.

  2. Adam says:

    Jay, I missed where he said anything about TDD and pair programming. He said "design a little, code a little, fly a little".

    I agree with you Scott. Being agile is about the fundamentals: doing small units at a time so you’re never too far off from your goal (plus supporting change without as much overhead).

    I think a Methodology usually gets in most people’s way when thinking about how to "become" agile.

  3. Adam says:

    Oh and my point about the pair programming and TDD is that there you’re getting well beyond Scott’s point about what Agile is and starting to talk Methodology.

    It’s the Methodology that people think is dumb. TDD and XP can work great but people get too fanatical about it at times and fanaticism is what turn people off to an idea.

  4. Srikar says:

    I have experienced wut you have described in your blogticle (Am  I coining a new word?) neways..Every project has some constraints, they can be time based or budget based and I would say both incremental and iterative combined has a solid effect on the outcome. It primarily depends on the person who is leading the development effort. He can either sit there and complain that he cannot move forward without information or give time to business and do an incremental work which might help the clients to realize what exactly is that they want.

  5. Mike Dimmick says:

    I’m tempted to nominate you for Joel Spolsky’s next collection of ‘Best Software Writing’.

  6. ericgu says:


    I’m tempted to thank you for your kind thoughts.

    But seriously, I appreciate the thought.


  7. Mark W says:

    I agree in general.  Software is best when baby steps are taken with an overall framework and goal in mind.

    However, looking at the construction schedule and the constant criticism around the 2010 olympics here locally, it seems that the real engineering trade has the same issues as the software trade has: when you take on large projects that is unique in nature things go out of hand.  The good thing about software is the refactoring cost is low.  The bad thing about construction is that you can’t build a wall or a room and say "let’s refactor this".

    I find this <a href=’‘>comic</a> very appropriate as well.  

    Mark in Vancouver

  8. Srikar says:

    I would like to add couple more  lines to your theory and make it cyclic

    Design a little

    Code a little

    Fly a little

    Fall a little

    Design a little

  9. Randolpho says:

    That was a profoundly accurate description of what building software is really like. From now on, I’m going to call myself a Software Researcher, rather than a Software Engineer.

    Kinda rings better, too, doesn’t it? Makes you think of white shirt/black tie, thick rimmed glasses, and a pipe.

    Wait, can’t forget the slide-rule…

  10. ala arnott says:

    i have been involved with many mnethodologies in my time.

    i started programming before there were any methodologies. back then, we had to make up out own procedures. we used METHODS. i still do.

    a methodology is just an attempt to enable those without sufficient knowledge of method to manage a project. in this, it is doomed. it is also, of course , means of making lots of money for the opriginator of the -ology.

    so much so that as soon as i hear an -ology i look around for the person who is trying to make money out of me.

    forget -ologies. study and use methods. keep it simple (stupid) and forgo the "get rich quick" schemes. you can be agile without being "Agile -ology"