I am going to tell you something that will disturb you. You might laugh, but it will be a cold uncertain laugh that will haunt you as you read on, because somewhere deep down you'll know it to be true. You might brush it off, get on with your day, yet sometime later, a week or a year, it will seep back in and unsettle you to the core. From that moment on you will be changed. You will think different, act different and will fundamentally be different. So take a moment to prepare yourself now, breath deeply, clear your mind and open up to the possibility that building software is hard.
It really is. It's harder then you've ever expected or even experienced. For most people on the planet it is actually impossible, for most of the rest is next to impossible. Only a rare few are even capable of taking on the challenge and most of those are simply in denial. As a species, humans excel at denial. It is built into how our brains work. For the most part we are not physically capable of acknowledging even a simple fact if that fact in some way threatens us. Instead, we adopt elaborate delusions that protect our egos. When it happens to a single person we tend to think of them as crazy, eccentric or quaint. When it happens to a large group we think of them as radicals. When it happens to most of us, we think of it as normal.
Evidence to the contrary enrages us. We react by constructing even more elaborate delusions to surround and trap these pieces of heresy. It happens with our religions, our politics, and our social lives. It even happens with our work. Our careers threaten us more than anything, so naturally we invent our best defenses. Having trouble with your latest review? Your manager is a jerk. Having trouble with all your reviews? There is a worldwide conspiracy against people like yourself. Your product has no appreciable market share? The other company is a monopolist. Can't get a handle around building software? You are using the wrong methodology.
In fact, software development methodologies are one of the biggest delusions of all. They help keep us sane by protecting us from the truth that building software is hard. They delude us into thinking that by following a preset script we will end up successful, that even the biggest software challenges can be overcome by engaging in a widely varying set of patterns and practices, that somehow if we all just planned better with a more precise schedule or gave ourselves the freedom to think big and react quickly the actual problems of finding the right solutions and building the software will take care of themselves. Yet, its just not true. On some level we know its just not true. Every project always ends the same, with some piece of something produced and everyone bemoaning how horrible the process was or how intolerable or inflexible. Instead of realizing that it is just hard and we are not really good at doing it, we distract ourselves by diving deep into naval contemplation, trying to devise improvements for the process for the next go around. However, we might as well have gone out and gotten new haircuts for all the good it does us.
The truth is that most of us can't really build software well, no matter the system. A few among us, however, can with apparent ease accomplish just about any development task, to any degree or complexity, in spite of whichever methodology is in use. We deny this too because it upsets the belief that its the methodology, not ourselves, that is really at the heart of the matter. We think of these folks as renegades or cowboys. We build up even bigger defenses to minimize the damage and marginalize their impact. We tell them they can't work on the entire project, they must pick some smaller piece befitting a mere mortal. We wall them off into little rooms and let them build, and then we fear what they produce, because it always ends up grander, more complete and more compelling than all the work done by everyone else combined. Often their work surpasses even the most ambitious dream. Yet we react to it by squelching it using backroom politics, voting it off the island.
The truth is, that if we could only harness the power of these exceptionally brilliant few in a meaningful way we would be ahead of the game. Instead of devising systems to distract ourselves into believing we all can do it, why not acknowledge the fact that really only these few know what the hell is going on and build a system to basically keep them functioning at prime efficiency? Instead of textbooks lauding the latest practices to keep our armies of developers in sync and on track, we should be pull the men out of the silos and teach them how to be a support team for the guy or gal in the center.
In reality, the really successful product teams already function this way, the ones that build the cutting edge technology that wows the customers and is adopted with great fanfare. Yet, they are often functioning this way by accident. It can only happen in the wild when one of these genius few turns out to also be good at promoting his or her ideas to the degree that management gets fooled into following along, and before even a team can be built around it the core 80% of the product is already designed, developed and ready for beta testing.
What we need is a system that will foster this behavior. The hard part is identifying the right model for the "rest of us," so we instinctively know what to do. We are so used to expecting equal treatment it will be difficult for most to adopt a different way of working. However, it is not inconceivable that a group of primates like ourselves can find an optimal organization that will produce the desired effect. You might say it is more instinctive that you would otherwise imagine. In fact, it may be so instinctive it would be considered primal.
Take gorillas, for example, another set of primates that most certainly operate in this manner. Every group has a dominant ape. The others are often found grooming the prime ape, keeping him at his best. If we switched to this model, we could easily have one dominant developer and the rest would basically be grooms, reviewing code, fixing bugs, writing design docs, running fxcop and polycheck, getting coffee. In fact, some could literally be cutting hair.
The possibilities are endless.
I hope someday you will come to realize the inevitable truth, stop wasting your time refining pointless process minutiae and accept that our ancestors had it right all along.