Jacob and the data

When Jacob gets going, the best thing to do is step back and watch.  He stands, all five-foot-six of him, at a white board, "speaking" through the myriad of boxes, arrows, and random words scrawled across the not-quite-white surface, as though to add another layer of blue ink to the washed-out background of partially erased thoughts.  Words tumble enthusiastically out of his mouth only to slide and bounce against the shiny surface of ink and dust, because his back is to me as he writes, as though the pictures are his mouth, and the sounds of his voice are the crickets of a summer eve.

"The design is easy.  This problem is solved."  More lines and arrows emerge.  He sells his confidence as a commodity, freely traded, highly valued.  I watch and listen.

"We have seen countless articles that show five systems interacting over a messaging hub or bus.  They all show a diagram that looks like..."  An image appears that looks not unlike a spider with square feet.  At the center, a circle, with lines radiating out to rectangles about the size of a paperback book.  In the center circle, Jacob writes "hub" and heads around the room nod.

"The problem your typical message-based architecture is that it is designed to be reliable when one system goes down," he continues, striking a line through one of the rectangles, "but we haven't really discussed what happens when a new system comes up."

In an single fluid motion, he has slid five feet along the room-length white board.  In this conference room, with six other architects, Jacob is in his element.  Standing at the far left of the white board, against the corner, is Tom.  Tom is tall, slim, and a picture of calm.  He stands with a whiteboard marker in his hand, limp by his side.  He had started Jacob going with a seemingly simple question.

"How," Tom had asked, "do we start a subscription to a new system without interrupting the message flow to the existing ones?" 

Of course, no one but Jacob had really understood the question, so he asked it again, this time using the white board.

"If I start with two systems talking," Tom said to the white board, as he drew a box, a circle, and a box, a few inches apart in a straight line.  He connected them with lines.  System A sends 1000 messages a day.  System B subscribes. 

"Now I want System C to start up."   Another rectangle appears, this time directly below the circle.  He quickly swiped a line from the new rectangle back to the central circle.  "But system C will need a lot of data that A has, so we prepopulate C's database."   This time, a dashed line from one rectangle to another. 

Jacob had been clearly ready to jump up, but you could see he was waiting for Tom to finish the question.  I know he meant it out of respect for Tom, who is brilliant in his own right... but you could see an expression crossing Jacob's face like that of a fourth-grader, sitting in the front row, throwing his hand in the air because he has just recognized a question from the teacher that he knows the answer to.

"But that prepopulation takes time.  It always does.  Now, when you turn on C, it has already missed some of the transactions sent out from A.  The database is out of sync unless we turn off System A... but we can't!  It isn't our system and it's mission critical." Tom finished with a slight bit of agitation.  His diagram had a half-finished look about it.  He had circled the bottom rectangle about eight times.  There was a faint smell of white-board ink

That's when Jacob had taken over.  And now, he was on to his second diagram, the spider picture freshly drawn and a new diagram emerging, this time with a row of squares and long thin rectangles both above and below them.

"Let's look at this from a different angle," Jacob said, this time addressing the group.  Patricia was sitting at the end of the long wooden table, and around the back, able to see the entire board, were Phil, Ram, Meiko, and myself.  Tom remained standing in the corner.

Above the top rectangle, Jacob wrote "bus" and we were back on the same page.  He had simply taken the "spider" diagram and blew it out, so that the message mechanism was on top, with the applications below.  But what was the parallel rectangle down below?

As though to answer my question, Jacob pointed at the bottom rectangle.  "This, my friends, is the data bus.  Unlike the message bus," (pen moves to the top rectangle), " which passes messages from one system to another, this little beastie," (pen back down), "collects data extracts from the applications on regular intervals for BI feeds.  This is where you put SQL Integration Services."

"That still doesn't answer my question." Tom chided.  Jacob waved his hands and turned back to the board.  This time, he drew a cylinder next to the top message bus. 

"When messages are sent from A to B, they are also copied to a data store" he said, indicating his new cylinder.  "Assume the data is extracted from system A at noon."

Jacob continued, pointing at the third rectangle in the middle, that we were supposed to infer was system C.  "Now, when C comes online at 4pm, it subscribes.  However, the subscription request includes a request for all messages that A sent since noon, since that's the timestamp of the data extract."

"An agent picks up the message request, goes to the data store, and sends the messages to C that had already been sent by A.  Like replaying a key log."  Now Jacob was facing the room as he spoke.  His diagram was done, or at least we thought.

"So what's the point of the data bus?"  Patricia's turn.  I think she knew the answer, but she was just as interested as I was to hear Jacob explain it.

"To feed the other half of that data store.  The data store holds basic query data, and maybe even some real-time aggregation data, so that many of the messages coming through the message bus can be answered without actually hitting the source system.  Think of it as a BI base for the SOA architecture."

At this point, Jacob drew a large arrow from the bottom rectangle to the 'data store' cylinder. 

"Hell, I learned something today."  Phil drawled in what was left of his Alabama accent after spending a decade in the Pacific Northwest.  "That's why I like these sessions" he said, as he started to collect his things.

The meeting had been over for quite some time.  It was the end of the day, and six folks had stayed behind to draw diagrams and talk shop. 

I left with Phil's words still bouncing around in my head.  "I learned something today."

And that makes it a good day.