As I said in Nailing the nominals, the two keys to successful big projects (100K+ LOC) are thinking ahead and defining done. Thinking ahead is about design and planning. Defining done is about setting a quality bar and sticking to it. Yet many big projects go astray even when people think ahead and define done. Why?
Often failure is due to poor executive decisions that place their own agendas above shipping. (Given you hit your quality bar, shipping is better—much better.) However, an even more frequent form of failure comes from engineering teams over thinking and over generalizing—trying to solve world hunger instead of feeding the kids in front of you.
It's so tempting, so intellectually pleasing, and so self-serving to examine the customer problem you're solving and see the bigger problem, the more general problem, the problem you can solve for all time, all peoples, and all nations. Oh spare me, and spare your customers. Cultivating the green fields of broad ideas is not only self-serving, it's a recipe for feature-rich and value-poor products and services your customers will use begrudgingly, despise utterly, and abandon gleefully at the first opportunity. Green fields are full of maggots.
Yet, I can't walk down a hallway without hearing people trying to come up with ideas, algorithms, or class structures that "will work for this problem, and all problems like it." Evil. Evil! Warning! Watch out. This line of thinking is EVIL!
What's so evil about general solutions? After all, your code could be both a floor wax and a dessert topping. There are three primary pitfalls:
§ You rarely work through the full general solution in one ship cycle. The unfinished framework isn't quite right, but you've shipped it. Now you are stuck with an unworkable legacy code base—forever.
§ You introduce a broader test surface and a broader security attack surface. Neither is desirable.
§ You put the problem at the center instead of the customer. When the customer isn't at the center, your code loses its soul. It goes from being astounding to being adequate, from marvelous to mediocre.
You saved me from this fanatic
Before we break down the three primary pitfalls of general solutions, I need to calm down my agile zealot readers. Because agile methods put a premium on working software and customer collaboration, they tend to avoid the over-thinking trap. In particular, test-driven development and emergent design deliberately focus on keeping solutions as simple as possible and laser-targeted at customer requirements.
Because agile methods avoid the pitfalls of general solutions and general thinking, many agile zealots believe all big design up-front is vile. They are wrong. Sure, the regular refactoring and rework needed for emergent design isn't a problem for small code bases. However, when you get above 100,000 lines of code serious rework becomes extraordinarily expensive. That's why customer-focused, up-front architecture is essential for large projects.
The good news is that many agile methods like Scrum, continuous integration, and test-driven development (TDD) work well within a large project with a customer-focused architecture. These great techniques can keep a team locked onto the customer, instead of straying off into the green fields of self-gratifying intellectual exercises in futility.
Who will save your soul?
Okay, back to the three primary pitfalls of general solutions— incomplete, unworkable legacy code, expanded test and attack surface, and losing your soul.
Say you are developing a big product like Halo or SharePoint. In either case, you can start with a general framework or you can start with a customer story.
The green fields approach is to start with a general framework. The framework supports all forms of weapons, landscapes, and content viewers. Putting together a game or a Web site is simply a matter of choosing the desired combination of weapons, landscapes, and content viewers. Bingo! You've got yourself a cruddy game or Web site.
The game or Web site has no soul because you focused on the framework instead of staying with the story. In addition, to test the framework you've got to consider every combination of weapon, landscape, and content viewer—each of which also presents an attack vector for hackers. And, should you be foolish enough to expose the framework as an "extensible interface," all these problems become orders of magnitude more diabolical.
It's not gonna get any easier
What's even worse is when you design and develop Halo II and SharePoint 2.0. Inevitably, there are many considerations you didn't address in the prior release that invalidate significant portions of your framework. Unfortunately, those portions are already built and shipped, so you get all the enjoyment of legacy backward compatibility with little of the benefit. Aren't you glad you were just so very clever to focus on the general problem instead of the customer and the story?
"But frameworks and extensible interfaces are critical to our platform!" scream the clueless cretins. Yes, they are when the customer is a developer and the story involves building upon our platform. However, the customers of Halo and SharePoint aren't primarily developers. They are consumers and enterprise customers. The story line for them involves crushing the Covenant and sharing information. Focus on the customer, not on the framework.
Can I tell you a story?
What does it mean to focus on the customer and the story? It means don't solve the general problem—solve the customer problem. The problem customers are trying to solve.
It means knowing the story line (the scenario) for customers. Who are they? What are they trying to accomplish? How are they used to accomplishing it? How might our software help? What would that look like to the customer?
Remember, many of our customers are developers. It's no different for them. Who are they? What are they trying to accomplish? How are developers used to accomplishing it? How might our software help? What would that look like to the developer using our platform and tools?
Once you are focused on the customer and the story, design, develop, test, and deploy that story for those customers—nothing more, nothing less. Make that story compelling and delightful.
Along the way, certain generalizations will emerge. Avoid them unless they are needed for the story. Only engineer what's needed to realize the story for the customer. Everything more general is wasted effort because by the time you need it the story will be different.
Temptations always come along
So you code up one story (scenario) and start working on the next. Perhaps it's even the next release. Aren't you going to wish you had generalized? No. No, you aren't. If you had generalized you would have done it wrong. You wouldn't have known what you know now.
Say while working on the next story, you realize you need to add a new weapon, landscape, or content viewer. No problem—refactor the design and add what you need now that you know what it is and why you need it. The result is better code, tighter code, and better tested code because the tests can focus on the story line.
Focusing on the customer and their story line isn't difficult conceptually or even in practice, but it does require restraint. It is so tempting to solve the big problem for humanity. Resist that temptation. You can't please everyone, so do your best to at least please your customers. In time, humanity will sing your praises.