SOA/ESB Integration in the Real World

Yesterday I helped at the Australian Architecture Forum as a Scribe for

Dr. Don Ferguson. The event as a whole was excellent. Thanks to Nigel Watson and Charles Sterling for letting me assist. On my personal to-do list is the IASA.

Anyway onto the round-table (more like a press conference) transcript:

DON (introduction):

Customers typically ask “Where is the product I can touch?” when they think of ESBs.

The statement that ESB are purely patterns is probably a restrictive statement. People want an ESB on a single server e.g. developed for workflow. The “Internet Service bus” is emerging e.g. BizTalk Service, Yahoo pipes.

Q: Do you see ESB’s reconciling the problems of interoperability between various platforms?

DON: I like to use the “lost in the forest” analogy - a sequence of bad decisions leaves you literally lost in the forest. ESB is not meant to be a solution for this type of behaviour. WS-I was invented so that the various companies will have to conform. ESBs are more interested in routing (e.g.in inventory management an inventory check should not need to know there are two inventory systems)

Q: Do you have any views on testing SOA deployments? If an organization is decomposed to a set of services we have a testing challenge on regular testing and regression testing due to the number of combinations possible.

DON: The problem is in explosive growth and the dependency on existing testing applied to the service being provided. Often the combination is not apparent until it emerges. It depends also on whether you are doing SOA design from top down or bottom up. The temptation is often to get everything in a repository, but often you cannot rely on people to put everything in the registry. Often this leads to companies introducing scanners to detect and discover services. In terms of testing, you cannot develop SOA systems in the same way traditional systems have been built – scenario first development helps. The reason why it helps is because as the number of services increase the harder it becomes to test the solution as a whole. In a SOA world you cannot determine in advance who is going to request what from whom – you have to assume the service is open to all available users. SOA gives you the ability to log usage and therefore test the various usage scenarios and match them with the projected use. In terms of ESB’s they are part application runtime and part application delivery – it can be used as a constraint system.

Q: In the context of ESB’s some people say the SOAP message is the bus. What is your view?

DON: Saying the soap message is like the bus, is like stating the envelope is the postage system. It’s not the way I would think about it. The metaphor I use instead of the bus is the wire. The wire in this case would be something that observes the SOAP message, not the SOAP message itself. For example, if there is a constraint to log all transaction amounts greater than 10,000 dollars the bus can enforce this but to do this with purely SOAP you would need to go back to each sender. A bus has transformation and message side-effects.

Q: What are your thoughts on the marriage or distinction of data binding and services?

DON: Two services accessing the same data source using the same binding, in my opinion, is a bad idea and will make your software less flexible. The two services will be effectively bound to each other.

On the other hand data services that talk to one database, tend optimize the data access as the service can add value and interpret the data. But people can end up not looking at the big picture and therefore make performance decisions that affect the whole of the solution.

Performance can be problematic because there can be unnecessarily chatty conversations. The architect should think of the service as being geographically separate – it’s more like sending a message to the space shuttle. Therefore we should be granular and conservative about the calls we make.

Q: How does ontology fit into SOA?

DON: The semantic problem will be in the domain of industry bodies. Creating your own is probably a bad idea. Semantics will probably be introduced by problem domains and by the related experts (government bodies etc). Tagging and annotation are also ways that services can be categorized. The bottom up approach to discovery is more intuitive in this case. Semantic modelling of services may be specific to the observer, so I am a bigger fan of tagging and annotation that has growth.

Q: Could you tell you your thoughts or preference for a distributed or centralized ESB?

DON: there is no such thing as a centralized ESB

Q: What is your view on service registries and repositories?

DON: Most ESB’s have these built in and they are necessary for performance management and planning etc. There is sometimes the “Highlander” philosophy of there can be only one service. I feel if this arises there is a master data management problem. Getting rid of repositories doesn’t solve this problem. Retrieve, create, update and delete (in order) is a way of viewing this data problem. Retrieve being the first will enforce a centralized read location. Organise the create, update and delete operation and the problem can be solved across the various data stores.

Q: What ways should services be looked at when the service fails and recovery is needed?

DON: It’s a really big question – today in the services world it’s an agreement on backing out what just happened. This is in principle a BPM issue – the problem is that the solutions that have emerged for long running transaction recovery have been too complicated. The trouble in the services world is that if a service changes then all the services that use or depend on this are also impacted. This requires an approach that pushes the responsibility to the individual services and introduces a lightweight approach to transactions.

Q: Should we design services always bottom up?

DON: I think you need to do both opportunistic design and systematic design. One doesn’t change the need for the other. In lots of ways one will ramp up the need for the other. People tend to work around problems and don’t come up with change management or procedural approaches. There may be a model for opportunistic development that is then handed over to be governed and formalized.

Doctors, for example, apply a template to discover what’s wrong with you. Templates may help us to understand what’s happening, like symptoms of a disease, but it’s not until it is well understood that we will see answers to the problems we are facing in services.