Primal Development Methodology


I am going to tell you something that will disturb you. You might laugh, but it will be a cold uncertain laugh that will haunt you as you read on, because somewhere deep down you’ll know it to be true. You might brush it off, get on with your day, yet sometime later, a week or a year, it will seep back in and unsettle you to the core. From that moment on you will be changed. You will think different, act different and will fundamentally be different. So take a moment to prepare yourself now, breath deeply, clear your mind and open up to the possibility that building software is hard.


It really is. It’s harder then you’ve ever expected or even experienced. For most people on the planet it is actually impossible, for most of the rest is next to impossible. Only a rare few are even capable of taking on the challenge and most of those are simply in denial. As a species, humans excel at denial. It is built into how our brains work. For the most part we are not physically capable of acknowledging even a simple fact if that fact in some way threatens us. Instead, we adopt elaborate delusions that protect our egos. When it happens to a single person we tend to think of them as crazy, eccentric or quaint. When it happens to a large group we think of them as radicals. When it happens to most of us, we think of it as normal.


Evidence to the contrary enrages us. We react by constructing even more elaborate delusions to surround and trap these pieces of heresy. It happens with our religions, our politics, and our social lives. It even happens with our work. Our careers threaten us more than anything, so naturally we invent our best defenses. Having trouble with your latest review? Your manager is a jerk. Having trouble with all your reviews? There is a worldwide conspiracy against people like yourself. Your product has no appreciable market share? The other company is a monopolist. Can’t get a handle around building software? You are using the wrong methodology.


In fact, software development methodologies are one of the biggest delusions of all. They help keep us sane by protecting us from the truth that building software is hard. They delude us into thinking that by following a preset script we will end up successful, that even the biggest software challenges can be overcome by engaging in a widely varying set of patterns and practices, that somehow if we all just planned better with a more precise schedule or gave ourselves the freedom to think big and react quickly the actual problems of finding the right solutions and building the software will take care of themselves. Yet, its just not true. On some level we know its just not true. Every project always ends the same, with some piece of something produced and everyone bemoaning how horrible the process was or how intolerable or inflexible.  Instead of realizing that it is just hard and we are not really good at doing it, we distract ourselves by diving deep into naval contemplation, trying to devise improvements for the process for the next go around. However, we might as well have gone out and gotten new haircuts for all the good it does us.


The truth is that most of us can’t really build software well, no matter the system. A few among us, however, can with apparent ease accomplish just about any development task, to any degree or complexity, in spite of whichever methodology is in use. We deny this too because it upsets the belief that its the methodology, not ourselves, that is really at the heart of the matter. We think of these folks as renegades or cowboys. We build up even bigger defenses to minimize the damage and marginalize their impact. We tell them they can’t work on the entire project, they must pick some smaller piece befitting a mere mortal. We wall them off into little rooms and let them build, and then we fear what they produce, because it always ends up grander, more complete and more compelling than all the work done by everyone else combined. Often their work surpasses even the most ambitious dream. Yet we react to it by squelching it using backroom politics, voting it off the island.


The truth is, that if we could only harness the power of these exceptionally brilliant few in a meaningful way we would be ahead of the game. Instead of devising systems to distract ourselves into believing we all can do it, why not acknowledge the fact that really only these few know what the hell is going on and build a system to basically keep them functioning at prime efficiency? Instead of textbooks lauding the latest practices to keep our armies of developers in sync and on track, we should be pull the men out of the silos and teach them how to be a support team for the guy or gal in the center.


In reality, the really successful product teams already function this way, the ones that build the cutting edge technology that wows the customers and is adopted with great fanfare. Yet, they are often functioning this way by accident. It can only happen in the wild when one of these genius few turns out to also be good at promoting his or her ideas to the degree that management gets fooled into following along, and before even a team can be built around it the core 80% of the product is already designed, developed and ready for beta testing.


What we need is a system that will foster this behavior. The hard part is identifying the right model for the “rest of us,” so we instinctively know what to do. We are so used to expecting equal treatment it will be difficult for most to adopt a different way of working. However, it is not inconceivable that a group of primates like ourselves can find an optimal organization that will produce the desired effect. You might say it is more instinctive that you would otherwise imagine. In fact, it may be so instinctive it would be considered primal.


Take gorillas, for example, another set of primates that most certainly operate in this manner. Every group has a dominant ape. The others are often found grooming the prime ape, keeping him at his best. If we switched to this model, we could easily have one dominant developer and the rest would basically be grooms, reviewing code, fixing bugs, writing design docs, running fxcop and polycheck, getting coffee. In fact, some could literally be cutting hair.


The possibilities are endless.


I hope someday you will come to realize the inevitable truth, stop wasting your time refining pointless process minutiae and accept that our ancestors had it right all along.

Comments (35)

  1. I am going to tell you something that will disturb you. You might laugh, but it will be a cold uncertain

  2. I am going to tell you something that will disturb you. You might laugh, but it will be a cold uncertain

  3. The Wayward Commenter says:

    Is this about you and your relationship with LINQ to SQL?

  4. Frans Bouma says:

    Brilliant πŸ™‚

    Btw, I wondered what kind of conflict inside your team triggered this post hehe πŸ˜‰

  5. mattwar says:

    This post is not about any conflict. Yet, it may be more about myself than I care to acknowledge. πŸ™‚

  6. Frans Bouma says:

    Ok πŸ™‚

    I do agree with what you said in the post btw. The key issue of course is: how do you recognize the person or small group of persons who you should put in the center of the rest and you should make a leading person/group within the team: it’s not always the people who LOOK like they’re leading, for example because they just have an outspoken opinion and seem to know everything or can do a lot, while they’re more leveraging work from others than accomplishing stuff themselves.

    The presence of an exceptional individual or group of exceptional individuals within a team is IMHO key for success of a software project, and makes the usage of whatever methodology (agile, waterfall, you name it) irrelevant: the quality of what’s delivered isn’t driven by the method, it’s driven by the quality of the people within the team.

  7. Sahil Malik says:

    Here is a delusion that so many of us create –

    Some honky tonk project manager on a software project: "Lets not worry about technology."

  8. Mike Griffin says:

    Matt and I worked together in another life, I still remember the project we did for the Corel Corporation. We were working on a Microsoft Excel clone. Matt, do you still remember the "sparse matrix", he he, I do, it was then I realized you were one of those guys.

    I have no real methodology, I follow no patterns or practices, I know very little about "theory" even, I’m not embarrased to say it. I don’t read books on stuff as it will just ruin my objectivity. This will sound egotistical for sure but I don’t care. When tasked with a problem I immediately think how the API would look to the person that would consume it, it has to be very simple, straightforward and intuitive, then I set about creating that API and it just works. I take no credit for it really (does this play into your psycho mumbo-jumbo?) as I was simply created that way.

    I can tell you one thing (again plays right into your hands) the folks like you are describing don’t do well career wise in most cases because there are simply very few places that recognize, or rather, or willing to recognize that fact. So, in most cases career wise I have just given up and poor my creativity into side projects.

  9. mattwar says:

    Mike, yes I remember the sparse matrix.  (Not a typical sparse matrix mind you.)  I should have patented it.  Had I known the truth then, I could be relaxing on the beach sipping lemonade now.

    I’ve invented some other bizarre data structures since then; the Yam and its successor the YamYam; a structure I called the ‘Stash’, a even a random hash table.

  10. Mike Griffin says:

    Yam = Yet Another Matrix …. πŸ˜‰

  11. Edward Kmett says:

    This basically sounds like the core philosophy behind the lead-developer-centric organizational structure pushed forth in the Mythical Man Month.

    Rather than trying to figure out what makes the prolific developer tick or burdening him with a command structure that takes away his productive time, harness him and enable him.

  12. Elias Amaral says:

    that always happens on open source software. the developers themselves decide how to organize the team, and often this put the ‘productivity heroes’ in a privileged position. when not, usually the code is forked.

    my own methodology is: let me work alone a bit. i will just code for fun. if i need help i can invite some friends. unfortunately, i usually stop coding when it is not fun anymore, so most projects doesn’t become anything usable.

    (well, this applies for my personal projects. in the moment i only study, i don’t have a job)

  13. Welcome to the thirty-fifth edition of Community Convergence. We have an interesting and controversial

  14. Krish says:

    Hi Matt,

    I agree with you only partly.  πŸ™‚  In my opinion, there are two factors:

    (i) The technical complexity of the solution and/or

    (ii) The technology being used (How new is it, what is the learning curve, etc)

    If you take a typical LOB application that has a SQL database and uses technology like C#/ASP.NET/WinForms, I’d say that most developers (your average Mort/Elvis) should not find the going difficult – assuming of course there is a development process being followed.

    Now on the other hand, when the technical complexity of the solution increases – for e.g. you’re building an IDE that is designed to be pluggable or you are building an implementation that translates a LINQ expression tree to SQL or you’re building a multi-threaded platform, then I’d say the average Mort/Elvis will find the going difficult and you’d rely on an Einstein to drive the technical architecture and implementation.

    Anyways, my point being, for a large percentage of software projects – LOB apps being  my primary example, you don’t always need an Einstein (though they would be handy), and assuming you follow a process/methodology that the team understands and supports, you’re on target.

    Thoughts?

    -krish

  15. mattwar says:

    Sure, you can take a bunch of typical developers and have them slog out the implementation of an LOB application, and you’ll get what you get, some confusing, arcane, input-forms nightmare, babble of buttons.  Or you can focus them all on aiding the prime ape on his journey to producing a masterpiece of technical wizardry that not only solves the LOB application problem, but allows the company an opportunity to rethink its corporate strategy, refocussing instead on the sale of that very same LOB app to the other fortune 500’s, boosting profits for the year by 1000%, cornering the market and becoming the next big software powerhouse to compete against Microsoft.

    It’s your choice.

  16. Jeff Kwak says:

    You are right on many levels.

    What you describe is very close to the concept of "The Surgical Team" described in Brook’s classic.

    The hard part is identifying the silver-back in the group. I’ve personally seen the company following the wrong dominant ape along the same path of destruction as the company followed the last dominant ape.

  17. NamelessCoward says:

    I work in the auto industry and have often pondered why the UAW has so many people who do work that could be automated. It would, in my opinion, be a better use of money to install robots on the shop floor and give every engineer an assistant.

    People worry about jobs going to China or India, but really we should be moving to revolutionize the work place in the West. Let China and India have all the jobs we don’t want. We could use the people from those jobs to make the Engineering and Design industries more efficient and productive.

    We could all have people to do the menial and repetitive tasks that I am faced with every day. I’m sure if Henry Ford were around today he would have made the engineering departments of the auto-industry much more like a production line. One person fixes FxCop, another managers bug reports and my ToDo list, somebody else writes specs etc. Also, I must admit I do like the idea of a floating assistant barber around the place.

  18. BiggerNamelessCoward says:

    Matt, I couldn’t agree more.  I have worked so many places, each with a different methodology…all seeking the holy grail of process and procedure.  The one area I might disagree with you is that I don’t think everyone is in denial, I think some just don’t get it.  I think most (in management) truly believe that if they can just create the right process, the people implementing it will not matter.  Yet everywhere I have worked, the most talented developers can succeed no matter what the methodology while the less talented falter.

    The problem I see that frustrates me the most is the often misinterpretation by management of talent.  I have seen far too many times where teams that create their own havoc by poor design or implementation, are hailed as heros for being able to put out high-profile fires; fires that should have never existed in the first place.  Yet teams where everything works smootly, rarely having fires, go unrecognized.

    One place I worked had budget money for new computers for part of the staff.  There were 3 teams each consisting of 4-7 developers.  My team had about a dozen open bug reports of which most were either political or architectural beyond our control.  Another team had many bug reports, but I don’t know how many.  Significantly more than my team though.  A third team had binders and binders of bug reports.  There was an entire cube whose shelves stored the binders with the bug reports.  Management decided to give the new computers to the third team because they needed faster machines so they could fix all those bugs.  They decided my team didn’t need faster machines because we only had a few bugs.  Talk about being punished for success and rewarded for poor quality.  We griped that they need not worry, we would make sure our next deliverable would qualify us for new machines.  πŸ˜‰  Of course we couldn’t do it.  You just can’t turn that off.

  19. What if the ‘silver back’ leaves? Or he gets pissed and starts throwing ape shit. What organization wants to put all thier eggs in one basket? There are ‘silver backs’ in different languages and technologies. And then a new language or technology comes along and a new ‘silverback’ comes to take the old one’s place.

  20. mattwar says:

    A true silverback is not defined a single language or technology.  A true silverback is master of them all, or master of all that matter anyway, or master of all that is really cool right now, or at least capable of convincing everyone else to go with the stuff he is master of for a variety of reasons that hinge on esoterica or incoherent technical babble.

    Besides, the ‘current’ technology is only that ‘new’ stuff that some other silverback at some other company wants you to use.  A true silverback will not be bullied by rank amateurs and posers from outside.

     

  21. yamyam? says:

    I don’t totally agree. Isn’t software development/programming/engineering getting easier? To easy. I think that is the problem. I mean with the RAD tools, and Java/Java derived languages that came out of the 90’s, everyone can be a coder now. Or at least they think they can code. And they do. And they get paid for it, so they are.

    I’m pretty sure most humans are capable of logical thinking. They just don’t have a passion for computers like we do. Of course focus and attention to detail help alot.

    Why don’t more young people pursue a career with us? Because its a dull boring job sitting at a screen all day writing someone else’s code. Bottom line, its an office job. Its pretty high paying work… but how many really want it? Not high paying enough.

    ohhhh wellll. good ego deflation all around then.  I’m up for a new methodology. oooohhh-oooohhhh-aaahhh-ahhh-aaahhhh!

  22. Lazy Coder says:

    "In fact, software development methodologies are one of the biggest delusions of all. They help keep…

  23. sandman says:

    Thats the reason gorilla  is still a  Ape.   We  humans are so powerful because ,  we use collective knowledge to achieve impossible, and we are so succussful, we are Super Ape. You are one frustrated  developer

  24. Sanat says:

    Well, the author is definitely not happy with this Software industry. I believe he tried his best to improve the things but …..

    But one thing that I must say here is that one can work in two different categories. One is "be your own manager" and other is "follow your manager". Software development is definitely a hectic tasks now a days. But implementing and executing good and feasible processes within the project will be benefecial for the team as well as the software.

    — Sanat Sharma

  25. WhamBam says:

    While it is true that different people have differnet abilities, I don’t agree with your basic theme. If what you say is true, why do all the large and successful products come out of large companies having large teams following processes? Why do you think software success rate has increased and it has become more predictable? In fact why have all the coffe boys to pay them large salaries? Why not have just a one superman company?

  26. Me says:

    > What if the ‘silver back’ leaves?

    You never really answered this question.

    And I have another one for you:

    how will you keep the grooming apes motivated?

  27. wma453 says:

    Software development is a new artform. Never before has man been capable of manipulating information the way that a software developer does.

    The technological development in the last 30 years of information appliances have increased exponentially  their capacity to store and manipulate data. On top of that,  new languages and  methodologies appear over time causing more cycles of experimentation. And this can go on forever.  

    We should be brave soldiers as we peer over the horizon into the unknown. hehe.

  28. TedvG says:

    You won’t find much of these "central wizards" you describe, because most of them program in Smalltalk and that magic clean OO language is not well understood and therefore rejected by the ordinary primates :o)

  29. mattwar says:

    SmallTalk might still be interesting for some outcast apes, but true silverbacks need the throngs of avid followers that come with big projects and mainstream languages.  SmallTalk apes can cry in the lonely night all they want about their prowess, yet if nobody else is listening it is worthless as an ego stroke.  No true primeness lies this way.

  30. W.Meints says:

    You confirm a feeling that I have since the second year of my software engineering study. Now that I’m also working in the wild I see exactly this around me everyday.

    It sounds scary and yet I know it can’t be changed.

    Excellent article πŸ™‚

  31. Dathan says:

    Ha ha — great thoughts Matt.  

    I’ve only found one answer and it’s worked 4 time in 8 years: keep starting companies to develop innovative technologies, hire developers who are either excellent like you, or will follow well, then sell and get out when the gem emerges.  Keep them small enough to be fun and not famous, but large enough to be lucrative (~2 years, $0.5-1M return).  Then start over.  

    To put it another way, your lambda is bent the wrong way.  It’s Yam => YamYam when it should be Yam => { fun(Yam); challenge(Yam); sell(Yam); beach(); YamYam(); }

    Gotta go, YamYamYamYamYam started 2 weeks ago…D

  32. mattwar says:

    Dathan, If you use the YamYam name in your next venture I expect some royalties, or at least a company logo’d t-shirt.

  33. Dathan says:

    Oh, and if at all possible, don’t take my advice until LINQ is finished; YamYamYamYamYam depends on it, and it appears to depend on you.  Thanks in advance… D πŸ˜‰

  34. Dathan says:

    Sure thing (on the T-Shirt) — let me know where to send it… D

  35. Anthony D. Green, MCPD says:

    "Isn’t software development/programming/engineering getting easier? To easy. I think that is the problem. I mean with the RAD tools, and Java/Java derived languages that came out of the 90’s, everyone can be a coder now."

    That’s like saying life is easier now because of technology compared to the middle ages. It’s not, in some ways it’s harder. Despite (or perhaps because of) instant world wide communication, and mass food production we as a people are still stressed and challenged. Everything we humans do it to minimize the mundane to make room for more lucrative or challenging mundane.

    Anyone can code now without worrying about memory management and compile-link-run, etc. This doesn’t make success less elusive, just clears away those things which were necessary but are only minimally related to actually writing software.