Re: Breaking the Cycle of Failed Development Projects

Tom is back in the field again after an honourable stint at patterns & practices, and asks in his post “Breaking the Cycle of Failed Development Projects”,

“....While it is strange going back to the world of consulting, I'm sure my tenure with the patterns & practices team will give me some new perspectives for the role. As an example, the subtle persuasion of people like Peter Provost has made me an agile development convert (although certainly not an expert!). While I believe there is a growing trend towards more agile approaches in many organisations, the p&p team is definitely ahead of the curve - but I'm very interested in applying what I've learned about agile development in my new role.....”

“..... The thing I am trying to understand is why we (meaning both the development teams and the business groups) insist on playing this game by pretending it can all be done. Business, of course, wants the certainty on what they will get when and for how much. Most development and consulting shops will take a project with a relatively detailed (but never final) requirements list and fixed schedule, and happily provide a fixed price quote (probably with a hefty risk premium attached). But with such a poor project success rate, surely everybody should know that the emperor has no clothes? My cynical explanation is that this model makes it very easy for everybody to blame someone else. The business can blame the development teams for failing to meet the agreed deadlines or requirements. The development teams can blame the business that the initial requirements were not detailed or accurate enough. Everyone walks away vindicated that it wasn't their fault that the project failed......”

“......I'd love to hear your thoughts on all of this, especially since I've been living in an alternate reality away from the consulting world for quite a while. Is the general state of software development really as bad as I make out? Is agile development a critical factor in addressing these problems? Have you been successful in introducing agile approaches in your own organisations or to external organisations? Is it possible to win a tender by proposing an agile approach when the customer wasn't expecting it? What do you think it will take to break the current cycle of blame and get everyone to think differently about software development?....”

Read his post to get the full context

First off, welcome back to the real world Tom, the field is certainly in need of your unique experiences,

What you are faced with is business as usual in our world. The game you refer to has been allowed to go on (historically), because it’s one of the few ways any business can get business done as far buying and providing custom developed solutions.

I think you will find that neither side of ‘the deal’ are willing to face real TCO estimates of a software solution (not just the development of it, but also management, testing, maintenance etc – which comprise most of the cost these days). This is why sadly, the cheapest tender on paper usually wins, regardless of potential to deliver, or quality of that delivery. The real TCO of solution development (with today’s one-off practices) is way too high to address sensibly to win a deal, and to be frank, the cost of hand crafted projects today IS way too high. If you were to estimate the real costs (door-to-door) of a one-off developed software solution today, I am sure you’d see: the solution providers would never win the business to build, and the customers would never buy the software project.

Today we live in a market where demand is far greater than supply, and today’s approach (or game) is allowed to survive only because anything delivered is better than nothing built. The outcome is that quality falls away like most other things in a project like this - development is the only guaranteed activity - since without it, there is no project at all. But requirements, scope, testing, quality, planning, architecture, and so on, are very optional depending on budget etc. (which we all know is never abundant).

We summarised observations exactly pertaining to these issues as a pretext for earlier software factories, and derived a vicious project failure cycle (Part I and Part II), where each step in the cycle drives the other to spiral in on project failure. There are some other important forces of course that should be understood that affect why you can rarely deliver under these conditions. See the followup where this is further exploered

[To digress just a little, and just for clarity, I think we need to make a distinction between agile process and agile development practices. You can implement agile, iterative processes without the agile practices (such as XP, TDD etc), and you can potentially implement agile practices without agile processes. It’s important to realise that agile practices exist (or evolved) as techniques to deal effectively with delivering within agile processes. If you were to try agile process with a formal practice, you are likely to end up at the end of an iteration with only half complete feature that doesn’t work. Instead of a fully defined feature that works only partially – but nonetheless works.]

p&p is extremely unique in both its project scopes, success rates and it’s motivations, and you can easily see why. In the real world projects of the type Tom  is faced with now, budgets are often limited or fixed. There are certainly a set of core requirements that cannot be comprised upon, and these projects are usually require large complex custom developments spanning many months to years, including existing assets, technologies and incumbent systems that cannot be replaced. These projects are usually based upon existing systems, processes or business requirements. So a large part of the business and technical requirements are already known to the customer, (just perhaps not to any level of detail useful to product management yet).

These types of projects don’t just say “go build us something that our users will find useful for doing their jobs better. ” and then measure its success on the number of users who show interest in it. The product they want building is largely known from a users/business perspective. Secondly, the resources to architect a design and build are typically challenged. (many aspects: including, skills, technology knowledge, commitment to project, various degrees of process and practice freedom). You will find that most organisations struggle with skilled resources on new technologies, and therefore spread the valuable ones around all projects to limited capacities.

p&p is much more successful because they have much higher than average skilled, dedicated, motivated resources working on the cutting edge of technology, with freedom of practice choice, and freedom of product scope. This is just not a reality for most solution providers today. p&p can afford to juggle product scope, because often the deliverable outcome is not well defined at the start, and has to be discovered and will be evolved, and complete when customers requirements are addressed. In this respect full agile process and agile practices are ideal match for these kind of solutions, and when managed well at the development level, have a high chance of success.

In real world projects of the type Tom is now faced with, you will find that customers just don’t buy investing large sums of money into a projects that might deliver what they want. They want guarantees, the type which custom hand crafted solutions can’t promise.

In these scenarios, more formal processes are favoured by product management because it’s used as a strategy to mitigate risk of under-delivery of the product (as a whole). This is based on historical distrust from evidence that if you leave it to the development team, you end up usually with nothing at the end, and all budget spent. It’s this distrust between management and development team that makes formal so attractive to them. The (wrongly placed) belief here is that if management define the timelines and processes and micro-manage them, then the development teams will fall in line and deliver to them. But little coordination is done to qualify what can actually be done. To be fair to product management, why should they care less how it’s actually done? As long as they can deliver an acceptable result.

But sadly in reality of building custom one-off solutions, it is typically done quite badly at the development process level for other reasons (poor asset reuse, skills, poor technology knowledge, changing requirements, poor architecture, little planning, little testing, constrained budgets, isolated developer efforts etc etc). So product management sees fit eventually to step in to fix this by defining a rigid formal development process (often the only way they know how). They erode delivery accountability from the development team by doing this, apathy sets in the dev team as the work piles up, no one has time for process anymore, change management will be poorly managed, adding extra pressure, decreasing overall quality of the deliverables to just meet arbitrary deadlines, that pay quality a token appreciation. It ends up in the blame game you talk about here.

Agile practices can certainly help in some of these cases, but not across the board. Ideally what you need is the ability to choose the right development process (and practices) for each iteration. And iterations need to be variable in length. Some iterations will require more up front planning when requirements are well known, and other iterations will require ability to react to more variable requirements.

Because of managements required involvement, usually the development are not afforded any control over the practice and approach they chose to meet deliverables for any specific iteration.

In reality, very few development teams, could chop and change between agile and formal to match requirements of a particular product release cycle.

• There should be a clear separation between product management and development management
• Development management should be accountable for delivery, development team members should be individually accountable for their work.
• There should be many more iterations.
• There should be variance in length and resource of iterations, depending on what requirements are known upfront for the next cycle, component, feature set etc.
• The development team should have freedom to chose the best process and practice to deliver that in the iteration.
• Quality should be the highest priority.
• Products have to be defined from standard requirements, and assembled from standard parts.