Having something to govern with

Joe McKendrick over at ZDNet posted a very thoughtful blog post the other day, responding to a prior post of mine.  My post said, basically, avoid bottom-up SOA at all costs. 

Joe kinda likes bottom up.  Honestly, after reading his blog, I don't think there is a disagreement there.  In fact, I think we are using the term "bottom-up" in a different manner.  Joe is saying that "bottom-up" equals "start small."  If that is the definition: I'm on board.  Start small is nearly always a better way to introduce a new idea and prove business value.  Joe strongly emphasizes governance, which I also agree with (although the term is widely derided).

So, to set the record straight: we agree that with any new thing, start small.  We also agree that you need to know where you are going before you set out to get there.  Basically, I agree with everythng except the definition of "bottom-up SOA" that Joe is using.  Alas, the plethora of definitions for ANYTHNG in the SOA space is a problem that haunts us all.  Let me also clarify something: there is a third way.  I don't like either top down or bottom up.  (read on)

My definition of bottom-up SOA is based upon the notion of having the developers create whatever services they want, and have other apps adopt them as those developers see fit, based on a somewhat 'laissez faire' democratic approach.   Basically, the idea is to create the mechanism and get out of the way.  Democracy is great for countries and citizens.  I'm with Churchill on this.  According to Winston Churchill, "democracy is the worst form of government except all the others that have been tried."  I think that's what makes 'bottom-up' so appealing to us.  It seems natural.

Top Down, in my definition, is more like the Soviet Five Year Plan with all services decided in advance and a large investment needed to make things happen.  For some situations, that model is quite successful.  (Let's not forget that Russia was basically a vast agrarian kingdom with limited industrial capacity when the communists took over.  The Five-Year-Plans were a key part of dragging the Russian society and economy towards industrialization.  Basically, Stalin set strategic goals and the government worked to make them happen.  Human rights were discarded along the way.  The people suffered.  Some five-year plans worked.  Others did not.  All were expensive and very centrally controlled. That is what top-down thinking brings.) 

I'm much more of a "middle out" guy, believing that neither top-down nor bottom-up will ultimately meet the needs of the enterprise.  An enterprise is not a democracy, after all.  The destination is not decided by "the people" but rather by the "benevolent monarchs" that we refer to as CEO.  Therefore, to meet the needs of the enterprise, a simpler, less messy, more directive approach to modernization of the IT infrastructure may be more appropriate.  In the middle-out world, the catalog is not important.

Having sat in a "governance chair" for over a year, I can tell you that governance is like steering a ship.  You need charts to steer by and a good destination to get to.  Pure bottom-up (the way I was using the term) is either totally uncharted (an no service consumption actually occurs) or is charted from the perspective of the local business need, not the enterprise.  The principles are good, but the destination is in conflict.  Each business creates their own chart, and the charts do not agree.  Each local business leader gets their way, but the enterprise suffers. 

In other words, having a destination is not enough.  You need to have ONE destination. 

This is where governance comes in.  As Joe correctly states, without governance, you can't bring people to a single view of the data and integration.  Governance is the key.  As I've blogged before, there are three kinds of governance, all distinct.  You need all three.

  • There is planning governance.  This process aligns business strategy with the project portfolio.  That includes understanding where investments must be made to simplify, replatform, or refactor existing systems to reduce cost and complexity, as well as understanding how each proposed project helps to deliver on the strategic goals of the enterprise.  You need to build the right apps, and planning governance helps you do that.
     
  • There is design governance.  This process aligns the technical specifications and developed services with the percieved and strategic needs of the enterprise.  If the enterprise needs a good mechanism to share customer data, and you are implementing a major improvement to the CRM system, design governance will insure that the services you deliver are actually useful and usable by the rest of the enterprise.  This is a difficult area, and is wildly distinct from planning governance.  You need to build the apps right, and design governance helps you to do that.
     
  • There is operational governance.  This process allows services to be identified, monitored, and managed as corporate information resources.  That includes the recognition of the security requirements, scaling requirements, service level expectations, auditing and exception management requirements that a SOA will affect in an IT environment.  Operational governance often involves automation and tools. 

Bottom up SOA has no planning governance.  You might build the right app... if you are lucky.  Given my definition of bottom-up, you would be foolish to go this way. 

Top down SOA has all three, but it is heavy handed and expensive and not appropriate for many situations. 

Middle out is a governance first approach, that starts by developing artifacts, processes, and (in many cases) systems at all three levels.   This requires a good bit of work at the center, but it is not work to decide what services must be built, but rather to create a periodic table of the services and patterns that are needed, allowing the individual teams to fill in the gaps with sufficient guidance to make an enterprise service without a lot of interference from the center.  In fact, multiple teams can make the same enterprise service and compete on the basis of quality. 

So where does Joe disagree with me?  My use of the term "bottom up" is simply "not SOA" to Joe.  We both agree: catastrophe awaits.  My use of the term "middle out" is fairly close to what he is calling "bottom up" and we both agree that is the way to go.  It's a simple terminology problem. (My apologies for the confusion.)

Joe also includes, in his definition of bottom-up, the notion of starting small.  I think that's more "good business advice" than "SOA advice", so I don't call it out.  Starting small to prove value is excellent advice.  If the business is not ready to buy in to SOA, then definitely prove the case by starting small. 

Great.  What's next?   There is a "crossing the chasm" problem.  Once you have proven the value, how do you drive adoption?  You have proven value to your risk-taking key leader.  How do you get the rest to sign up? 

With the middle out approach, you are ready.  You have created the charts and they all agree.  Your governance body has a single destination to steer towards.  Perhaps the destination is not 100% clear.  That's OK.  They need to be "directionally correct."  Otherwise, if you had gone purely bottom-up, and you proved your value, you would have buy-in to get started, but no destination to go towards, and that is worse.  Far worse.