Integration Patterns utilizing the Windows Azure Service Bus

For years we have all utilized Gregor Hohpe’s Enterprise Integration Patterns book (and the web site).  Many of these patterns, if not most, are well known and there are many resources out there that talk about how these patterns can be implemented.  There are also a number of resources that talk about how these patterns can be implemented in BizTalk or Windows Server AppFabric. 

However, there is very little that shows how these patterns can be implemented using the Windows Azure Service Bus.  This two part series will take a patterns first  look at how these different patterns can be implemented. 

The patterns that will be covered in this post will focus on the Messaging Channel groupings.  The patterns in that fall in the Messaging Channel group contain:

Point to Point, Publish/Subscribe, Datatype Channel, Invalid Message Channel, Dead Letter Channel, Guaranteed Delivery, Channel Adapter, Messaging Bridge and Message Bus.  

In the next post we will focus on the patterns that fall in the Message Routing group, which contain:

Content Based Router, Message Filter, Recipient List, Splitter, Aggregator, Resequencer, Composed Message Processor, Scatter-Gather, Process Manager and Message Broker.

There is also the Messaging Systems group of patterns but those are being addressed with the Azure Integration Components (that is currently in CTP).  As those components get closer to release I will cover those patterns as well.

Lets start out by looking at each of the patterns in the Messaging Channel Group including a description of the pattern and then how it can be implemented.

Point-to-Point Channel requires that only one receiver gets the message.  If there are multiple receivers, then only one must still get the message.  This functionality can be done directly through WCF (the Service Bus can also be included by hosting the WCF service).  This pattern can also be accomplished through the use of the Service Bus Queue and controlling what services are listening.

Publish-Subscribe Channel requires that the message or event sent to the channel will be delivered to each and every receiver.  Once the receiver obtains a copy of the message it is removed from the channel.  This can be implemented through the use of Service Topics. 

A Service Bus Topic can have multiple subscriptions with each subscription being able to be individually managed.  The subscriber receive all messages, based on filters, that are sent once the subscriber is created.  If a message satisfies the filter criteria for more than one subscriber the message will be delivered to each subscriber and then onto the receiver.  When a message needs to be delivered to more than one subscriber, the message is not duplicated.  The meta data for the filter/subscriptions is stored and is tracked based on which subscriber/receiver has received the message.  In terms of removing it from the channel, think of the concept of ref counts.  One other thing to keep in mind is that if the Receiver is not online to receive the message, then it is stored until the receiver comes on-line.

Datatype Channel requires that each channel will contain the same type of data.  Thereby when the receiver obtains a message from the channel it will always be the same type.  This can be accomplished with directly with WCF (the Service Bus can also be included by hosting the WCF service).  

In addition, an Azure Queue can also be utilized.  You can create a new queue for each type of message that will flow through the queue. 

Invalid Message Channel requires that a message that is deemed to be invalid be moved to a Invalid Message Channel where they will not be processed by the normal receiver.

The Azure Queue implements the dead letter queue concept and uses the $DeadLetterQueue subqueue to store invalid messages.  Messages are placed in the dead letter subqueue:

When a message expires and deadlettering for expired messages is set to true in a queue or subscription.
When the max delivery count for a message is exceeded on a queue or subscription.
When a filter evaluation exception occurs in a subscription and deadlettering is enabled on filter evaluation exceptions

To get to the dead letter queue you would use:

 string queueDeadletterPath = QueueClient.FormatDeadLetterPath("<your queue name>");

the same behavior is available with Topics.  However, to get to the dead letter queue for a topic you would use:

 string subscriptionDeadletterPath = SubscriptionClient.FormatDeadLetterPath("<Your Topic Name>", "<Subscription Name>");

Dead Letter Channel requires that if the system can’t deliver a message there is a mechanism to move the message to a dead letter channel.  This capability is handled like the Invalid Message Channel above.  However, check out this link from MSDN on the QueuePolicy.PoisonMessageDrop property.  It points to the prerelease documentation but it is interesting to read what is being planned. 

Guaranteed Delivery requires that the system persists the message so that it is not lost if there is a crash or outage.  In using the Service Bus Queues, you are ensured of guaranteed delivery as the queue uses SQL Azure as the backing store.

Channel Adapter requires that there is an interface so that any application can connect and be integrated with other applications.  The Service Bus Queues and Topics both provide a full API that can be utilized by a number of different technologies.

Messaging Bridge provides a connection between different messaging systems.  Essentially to transmit message that was intended for another messaging system.  This can be done through either the Service Bus Queues or Topics with the receiver getting the message and forwarding it to endpoints provided by the other messaging system.  In addition, the receiver can get the message and forward it on to on-premise messaging systems through WCF.

Message Bus provides a means for separate application to work together in a decoupled manner with the ability for the applications to be added or removed with no affect on any other applications.  The ability to be decoupled but still allowing the application to work together, share data and operate in a unified manner is the basis for this pattern.  This pattern can be easily implemented through the Azure Service Bus and WCF endpoints.  We will come back to this pattern again when we spend more time covering the integration components.

In this post I have talked about the Service Bus Queue, but there is still some confusion between the Windows Azure Queue and the Service Bus Queue.  Yes, there are two Queues and they are different and have different uses and purposes.  Everything I have been referring to here is in reference to the Service Bus Queue.  You can find more information on the differences between Azure Queues and Azure Service Bus Queues in this article - Windows Azure Queues and Windows Azure Service Bus Queues - Compared and Contrasted.

Lastly, in talking about Gregor Hohpe and his patterns book, there are a number of diagrams that are utilized that represent the patterns.  These diagrams can be downloaded and used in Visio which you can find at https://www.eaipatterns.com/downloads.html