Bottom-up SOA is harmful and should be discouraged

It's independence day week in the USA, so it is time for me to declare a little independence. 

There are three schools of thought around "how to build an Enterprise Service Oriented Architecture."  They are:

1. Top down - central group decides everything and the dev teams adopt them.
2. Bottom up - central group provides a directory and dev teams make whatever services they want.  Dev teams go to the directory to find services they can use.
3. Middle out - central group provides key elements of the interface, including numbering schemes, message exchange patterns, standard communication mechanisms, and monitoring infrastructure, and encourages the dev teams to use it to build services that can be shared.

Top down works only in some organizations.  I imagine smaller IT shops (with 300 or fewer total employees) and shops with a well established heirarchy or very strong central leadership can leverage this one.  (I'm conjecturing, of course, having not worked in SOA in that setting.  It's an educated guess).  For everyone else, it is slow and difficult to follow the Top Down path. 

Bottom up is an answer to top down seen as "the wisdom of crowds" or "economics in IT."  The idea being that many services can be developed without much central control, as long as they are in a directory.  Everyone would look in the directory to find the services they need and adopt the best or least expensive, and we'd get SOA adoption.  This one has the advantage of scale.  The larger the organization, the argument goes, the better the case for bottom up. 

Middle out is the newest method developed by folks who prefer a blended approach.  I won't go into a lot of details here.

The point I want to make today: Top down is slow, but functional.  You can end up with a SOA that is good for the enterprise.  Middle out has the same advantages and is a bit quicker. 

Bottom up is actively harmful to the enterprise as a whole and should be discouraged at all costs.

Of course, a lot of the literature in the SOA field says "build the catalog first," effectively supporting the bottom-up approach.  And that is a mistake.

As I've said before, I am not paid to solve a particular system problem.  I'm paid to look out for the needs of the enterprise, to create a "gravity" towards the center to counterbalance the natural tendency of IT to develop only for the individual needs, even when it works against the strategic goals of the enterprise.

Bottom up is a mess.  Let's look at the logical conclusion of bottom-up for a minute and compare what results we get with some very common enterprise strategic goals.

Let's say our goals look like this.  This is a very simple set.

1. Share a common concept of customer, order, sale, shipment, and inventory so that we can get a straight answer at the end of the day about the operational and financial health of the business.
2. Couple loosly so that we can keep the costs of maintenance as low as possible and hopefully, over time, divert some of those expensive maintenance resources over to developing new programs. 
3. Build your applications so that other folks can easily integrate with them so that we can add new capabilities to the IT ecosystem quickly. 

On first blush, SOA supports all three. It is clear that you can get common data, loose coupling, and ready integration from SOA, so a lot of folks have cried "SOA is the answer!" and then stopped.  They didn't pick the path to take, so the debate rages on: top-down, bottom-up, or middle-out.

The problem with bottom up is that you end up with systemic effects, not seen by individual contributors, that work against these same strategies.

There are three ways that bottom-up SOA can go in your organization.  One, it is ignored (no one decides to participate).  Two, it is wildly adopted with some folks contributing and others consuming, and Three, it is adopted only be people who have services they want to advertise, but no one consumes them out of fear of taking a dependency.  (There is a common word for both One and Three: "failure".)  Let's look at Two: some contributors and some adopters.

What are the folks contributing?  Well if we are creating a system, but we are not paid to think about the enterprise, then the system we are creating may, but probably won't, deliver capabilities that the enterprise needs.  The service will be 'local-specific'.  That means that the service will serve the needs of the paying customer (a business leader) but not the enterprise as a whole.

So the services that are in the catalog, most of them anyway, are not really 'enterprise ready.'  That means that when others consume them, they are using identifiers that are not enterprise-ready, or they are using data in a manner that may or may not be usable in the data warehouse. 

Remember that consuming a service is only demanded by a business process.  If it weren't for the need to change processes, we would never need flexibility in the first place.  If we do 'bottom-up SOA' without considering the business processes, that doesn't mean they aren't there.  It means we are ignoring them.  It is in some folks best interest that we don't look.  Consuming a service "automates and hardens" a business process, making it more difficult to change.  What if we are hardening really bad process?  Better, what if, in different parts of the company, we are hardening different processes? 

In the past, I had a run-in with very strong-willed and opinionated leaders, both with IT and the business, who strongly defended their absolute right to "do things their own way."  Sometimes that's healthy, but in the cases where it is not, it can become a shouting match.  Bottom-up SOA avoids the fight because if the service is attached in the "right way" you use it, but otherwise, you write your own.  Fight avoided.  However, when a business leader wants to take a basic, core, fundamental process, and go their own way, that is not a fight you should avoid. 

According to some fairly compelling research and literation in the field of business process automation, the most successful companies have automated their core business processes, which provides less flexibility, but gives them far greater agility when it comes to taking on new challenges.  The core stuff, the stuff they do every day, is routine, standard, and well thought out. It is a stable foundation, allowing the company to stretch in a dozen different ways. 

"Bottom up" is simply a mechanism to allow those strong-willed leaders to create multiple infrastructures in the company "because they want to" without any consideration for creating a common and stable foundation for the business.

So lets look at a bottom-up success story three years later.  A number of infrastructures have emerged.  You have one based on Division A's CRM system.  You have another that feeds Division B's ERP system.  You have a third in Division C to surround their mainframe Operations management system, and a fourth in Division D focused on their home-grown web-order management infrastructure.

You have four legacy systems.  Integrating them will require a slow, painful, expensive infrastructure, in the middle, with huge amounts of semantics and rules needed to translate the data from one legacy infrastructure to another.  That central system will be necessary to support Business Intelligence and Financial Reporting (so you cannot avoid it) and it will limit the ability of the company to respond to the needs of the marketplace.

You got SOA, but at a price. 

Of our three principles above:

1. We have our common understanding of customer, but it is so far away from the systems that create that understanding that it is useless for generating real value. 
2. We have our loose coupling, within each ecosystem, but each one is at war with one another, vieing for resources and investment.  The money we saved by developing the individual infrastructures has moved to the central integration hubs and BI.  No win there. 
3. We've got easy integration, as long as the system that needs to consume the data has the same semantic understanding of the data as the system that produces it.  Otherwise, consumer and producer cannot work together, and the cost of forcing an integration is high.

We will have won the battle, but lost the war.

This is not a scenario that I would wish on anyone (except perhaps my direct competition).  To be honest, I think my competition is smart enough to see this for themselves.

Bottom-up SOA may work if your measurement is "get services up an running" (or it may fail).  If it works, it produces a monster that is not worth producing.  That is not success either.  Measuring SOA success by the number of services, or encouraging the creation of services without any form of central understanding of what the services should contain, is doomed.

Avoid Bottom-up.  It is a pox on your house.