Services and the Business/IT Gap

Recently, a gentleman from Switzerland wrote me an email after attending the “WinFX Tour” presentations in Zurich. He is a business consultant advising corporations on the IT strategy and an IT industry veteran with his first programming work dating as long back as 1962. He was quite interested in the Workflow part of my presentation, but wrote me that he thinks that those abstraction efforts go the wrong way. He sees the fundamental gap between business and IT widening and sees very little hope for the two sides to ever find a way to communicate effectively with each other. In his view, IT isn’t truly interested in the reality of business. He wrote me a very long email with several statements and questions, which I won’t quote – the (very long) reply below should give you enough context: 

Your main concern is, in my words, about the disconnect between the reality of the business vs. the snapshot of a perceived business reality that is translated into a software system. I say “perceived” because the capturing of the actual business reality is done by analysts who are on the fence between being business experts and IT experts and even though they would ideally be geniuses in both worlds to do that translation, they often are coming down on one side of that fence in terms of their core competencies.

The only way to close that gap is to pull people off that fence onto the business side and enable them to capture the reality of the business and the way the business processes flow with tools that fit their needs and don’t demand that they are programmers or even have the sense of abstraction that a software designer or process analyst possesses. Our industry is only starting to understand what is required to achieve this and we are certainly thinking hard about these problems.

You state that the Business/IT gap cannot be bridged. I do not fully agree with that assessment. I think what you are observing is a particular effect of software architecture and implementation as it exists today. You are truly an “industry veteran” you can certainly see much clearer how software has evolved since you got into the trade in 1962 than I can as a relative youngling. However, my (humble) observation is that the fundamental concepts of business software design haven’t changes all that much since then. A business application is a scoped set of siloed functionality built for a set of predefined purposes and whether the user interacts with the system through batch jobs, green screens, web sites or whether the system is made up of 5000 identical fat client applications with identical logic that talk to a central database is merely an implementation detail. The tradition of (interactive) business software is very much that we’ve got a system with some sort of menu screen or other form of selecting the task you want to perform with the system and any number of forms/screens/dialogs with which you can interact with the system. The reason for your observation of IT conveniently neglecting at least 20% of what they are told to do is not only caused by them not understanding the business, but also caused by them being not in control of the monsters they create because the scope grows too big, everything is tightly coupled to everything else, and a lot of functionalities are crammed together in ”multi-purpose” user interfaces that often make changes or adjustments mutually exclusive and hence “impossible”.

The actual business process is often external to the software or if the software has workflow guidance, the workflows are not taking all the “offline” activities into account and the process becomes lossy. The way that most applications are built today is that they present the user with a grab-bag of tools and procedures and leaves it up to them to navigate through it. Even worse, business processes are often changed to fit the constraints of software – and not the other way around. Testimony for this is that the organizational structure of many companies is hardly recognizable once the SAP/PeopleSoft/Oracle/etc. ERP consultants have left the building.

So what’s changing with the SOA/BPM “hype” as you call it? Or I shall better say: What’s the opportunity?

First, we’re working hard to get to a point where we can build interoperable, autonomous pieces of software with well-defined interfaces where we don’t have to spend 80% of our time and money trying to make those pieces communicate with each other. Before the industry consensus around XML, SOAP and the WS-* specifications this most fundamental requirement for breaking up the solution silos simple didn’t exist (despite previous efforts like DCE or CORBA). I am not saying that we have completely arrived at that point, but we’ve got a better foundation than ever to build composable, loosely coupled, distributed systems that can interact irrespective of their implementation specifics.

Second, we are starting to see growing insight on the IT side for the need of a common understanding of the idea of “services”. Any employee, department, division or vendor assumes various roles within an organization and renders a certain portfolio of services towards the organization. The notion of service-oriented architecture speaks about writing software that fits into the organization instead of fitting the organization to the software. Just like an employee assumes roles, software assumes roles as well. From an architecture perspective, people and software are peers collaborating on the same business task. The former are just a lot more flexible than the latter.

Business processes (or bureaucracy), whether formalized or ad-hoc, are driven by the flow of information. If the information flow is not central to the execution of the process you have friction and efficiency suffers. With a paper-bound “offline” process where all information flows in a (paper-) file folder and on forms carried by courier from department to department there’s arguably a better and more complete information flow than in an environment where information is scattered around dozens of different computer systems where the flow of information and the flow of tasks are disconnected and only knitted together by people shuffling mice around on their desks.

The opportunity is right at that point. With the technology we have available and a common notion of “services”, the line between the implementation of deterministic work (done by programs) and non-deterministic work (done by humans) begins to blur. It is fairly easy today to write chat-bots, mail-bots, or speech-enabled services that allow rich computer/human interaction within their respective scope. It is likewise fairly easy to expose application to application services that allow exchanging rich data across system and platform boundaries using Web Services. The concrete form of how a service interacts with a peer depends on who the peer is. If you have an address book lookup service, it may have all of these capabilities at once. With that, services can be integrated into real-life ad-hoc scenarios singly or in combination because they are built to satisfy specific roles and their capabilities are exposed for (re)use in arbitrary contexts.

When you have a service that’s specialized on creating complex sales offers and you are a salesman on the road an sit in a taxi, it’s absolutely fathomable to build a solution that allows you to call in, identify yourself towards a voice service and ask for an baseline offer for 500 units of A and 300 units of B for a specific customer and have the result, with all applicable rebates and possible delivery time frames, including shipping cost and considering the schedule of the container freighter ship from China, pushed onto your hotel fax or by email or SMS/MMS. However, the question of how realistic it is to build that service purely depends on how easy it is to make all the necessary backend systems talk to each other and wire up all the roles into a process that can jointly accomplish the task. And it also depends on how well any necessary human intervention into that process can be integrated into the respective flow. Assuming that the resulting offer would cross a certain threshold in terms of the total order amount, the offer might require approval by a manager – the manager renders a “decision service” towards the process and might so by responding to an email that is sent to him/her by a program and will be evaluated by a program.

Ideally, you could teach a service (through mail, speech or chat) the workflow ad-hoc just as you would tell an administrative assistant a sequence of activities. “Get A and B, do C, let Mike take a look at it and send it to me”. The information can be parsed, mapped to activities, the activities can be wired up to a one-off workflow and the workflow can be launched. The crux is that you need to have those individual capabilities and activities catalogued and available in a fashion that allows composition. And the above example of the approval manager goes to show that people’s roles and capabilities must be part of that same catalogue.

We (Microsoft) are already shipping and will ship even more building blocks not only for creating such services and workflows, but we also have an increasingly complete federated identity and access control infrastructure that allows to realize all that decentralized interaction in a secure fashion. From a purely technical perspective, the above scenario is not utopia. We have every single component in place to let customers build this, voice recognition included. However, mind that I am not saying “very easy”.

“BPM” tools such as the Windows Workflow Foundation or BizTalk Orchestration are all about putting the process into the center. Services are all about creating easy-to-integrate, loosely coupled, business-aligned pieces of functionality that can be used and reused in as many contexts as a role in a business can be used in different contexts. A workflow may just be one step that you just have in your head as you are interacting with the address lookup service or it may be a more formalized workflow with several steps and intertwined people-based and program-based activities. The realization here is that while individual roles in an organization are relatively sticky, the way that the organization acts across those roles and tunes the rules for the roles is very dynamic. BPM tools are precisely about shaping and reshaping the flow and rules quickly and deploying those instantly into the business environment.

The design of the Windows Workflow Foundation also recognizes that business processes, especially those that run for days and weeks rather than seconds or minutes, are never set in stone. If you’ve got a (very) long running master workflow that were, for instance, tracking an insurance policy and applies rebates and handles claims as time progresses and suddenly the client comes around and sues the insurance company, that policy certainly no longer belongs into the same bucket as the other 100000s of policies that are being tracked. So for such unforeseeable circumstances, the foundation makes it possible to jump right in, assess the status and redirect or reshape the respective flow on a case-by-case basis and if the new action that you add into the flow in order to terminate it is merely that you hand off the entire case with all of its status to your legal department’s “dropoff service”.

You write about SO/A and BPM as an “effort to extend the borders of IT”. From my perspective it’s rather the attempt by IT to humbly fit itself into the ever changing dynamic nature of business. On an industry level we have started to realize that we need to get away from the thinking that applications are silos and that nobody should factually care about whether he/she interacts with a “CRM” or “ERP” system or needs to navigate across a dozen of intranet websites to get a job done. The whole notion of “we build a loan application handling program” is indeed misguided. That’s the disconnect.

But how alien is the notion of building a library of services that are not wired up into a thing that you can install and instantly run as a program? We hire people into organizations who have a broad education if which we only tap a fraction at any given point in time, but we can trust that we can instantly tap some other capability as the process changes. Which CEO/CFO/CIO will commit to a project that “educates” a software system to have a broad spectrum of capabilities that may or may not be used in the circumstances of “now” but which may become a pressing necessity as you need to quickly adapt to a change in the business? How strange of an idea is it that you might produce a software package that consists of hundreds of roles and thousands of activities but none of them are connected in any way, because that’s the job of the space that’s intentionally left blank and undefined to host the customized business process? How does the buyer justify the expense? What can the vendor charge? Would you continually service and update a “dormant” software-based capability to the latest policies, laws and regulations so that it can be used whenever the need arises?

All that comes back to a completely different communication breakdown: How does IT explain that sort of perspective to the business stakeholders? The great challenge is not in the bits, it’s in the heads.