MSMQ, WCF and IIS: Getting them to play nice (Part 3)
Previously, in MSMQ, WCF and IIS: Getting them to play nice:
- In Part 1, we built a client and IIS-hosted service application and got them communicating over MSMQ using WCF's NetMsmqBinding.
- In Part 2, we deployed the same application across multiple servers, and enabled transport security for MSMQ.
In today's thrilling conclusion, we'll improve the resiliency of the solution by going transactional. Fasten your seat belts!
Going Transactional
Before we get started, let's spend a few minutes discussing the advantages and disadvantages of using transactional message queues. The advantages are all pretty nice:
- Messages will be delivered exactly once, and in order.
- Messages are persisted to disk, so they won't be lost if a server goes down.
- Sending and receiving messages can take place within a transaction. I've found this most useful on the receiving side: if you create a single transaction that encompasses both receiving the message and processing it, and a failure occurs during processing, the entire transaction will be rolled back. This means the message will be returned to the queue, rather than lost.
At this time you're probably thinking, "wow, that all sounds great - why wouldn't anyone want all of those?". The main reason is performance - using transactional message queues is typically many times slower than going with their non-transactional cousins. Also while the prospect of losing messages or getting duplicate messages sounds scary, in reality this would only happen under extremely rare and unfortunate circumstances. So the question here shouldn't really be "do you want the improved reliability that you get from transactional message queues", but rather "can you afford to live without it?".
That said, there are any number of scenarios where transactional message queues are justified - such as storing audit records, processing financial transactions or sending greetings in blog post samples. So let's get started!
Create a Transactional Message Queue
The first thing we need to do is create a shiny new transactional message queue. Even though we already have a non-transactional message queue with the correct name, you can't convert a non-transactional queue to a transactional one. So you'll need to unceremoniously delete the existing queue, and create a new private queue, still called MsmqService/MsmqService.svc. However this time make sure you select the Transactional checkbox.
Now, after all the effort we went through to set the ACLs on the previous queue, make sure you set them correctly on the new queue to avoid more painful permissions problems!
Reconfigure your WCF Bindings
Once again, we'll need to modify the WCF configuration in both the client and service to use a new binding. This time we'll be using the MsmqBindingTransactionalTransportSecurity, which will be defined as follows:
<binding name="MsmqBindingTransactionalTransportSecurity" exactlyOnce="true" receiveErrorHandling="Move">
<security mode="Transport"/>
</binding>
The exactlyOnce="true" attribute is WCF-speak for using a transactional message queue. The receiveErrorHandling attribute is only needed on the service side (although it won't do any harm on the client side). This tells WCF what to do in the event that it discovers a "poison message". Poison messages are an important concept with transactional message queues. As discussed previously, if an error occurs while processing a transactional message, the transaction will be rolled back and the message will be returned back to its queue - ready to be picked up again by the same service. If the error was caused by a temporary glitch, the message may be processed successfully the next time around. However if the problem was due to a malformed message or a persistent problem with the application, the message is going to fail over and over again. WCF and MSMQ 4.0 have joined forces to provide support for poison message detection and handling. If the same message fails a number of times (3, by default), it will be considered "poison". What happens next depends on the value of the receiveErrorHandling attribute. If you set it to "Move" (my favourite choice!), it will be automatically put onto a sub-queue called "poison" where it can be manually dealt with by someone else.
So with our new binding beautifully configured, make sure you modify the endpoint definitions to refer to the new binding configuration name, and you're ready to move forward.
Add Transaction Attributes to your Service Implementation
If we want go get the advantage of executing the message receiving and processing in a single transaction, you'll need to tell .NET to enlist your code in the existing MSMQ transaction. This can be done in a single line of code, by decorating your service implementation methods with [OperationBehavior(TransactionScopeRequired=true)].
So far my sample service has consisted of a single line of code. While simplicity is normally a good thing in samples, it's not going to give me any opportunities to check the transactional behaviour or poison message handling. In order to make the scenario a bit more interesting, I've added some code that will let me easily create a poison message. My service class now looks like this:
public class MsmqService : IMsmqContract
{
[OperationBehavior(TransactionScopeRequired=true)]
public void SendMessage(string message)
{
if (message == "Bad")
{
throw new InvalidOperationException("Bad!");
}
Trace.WriteLine(String.Format("Received message at {0} : {1}", DateTime.Now, message));
}
}
As I'm sure you can tell, whenever I send the message "Bad", my service will fail. This will cause a exception to be thrown, and the transaction will be aborted. As a result the message will be returned back to the message queue, ready to be picked up again. Since the message has not been changed, it will continue to fail twice more, after which WCF will decide the message is poison and move it to the "poison" sub-queue.
Check DTC Configuration
Our epic journey is almost at an end. In fact if you're still playing along at home, you can try running the application with the transactional queues to see if it's working. If it's failing, one possible cause is problems with your Distributed Transaction Coordinator configuration. Here are a few things to try:
- Make sure that the DTC service is installed and running on all servers. If you're running Windows Server 2008, the feature may not be installed by default.
- Check your DTC security configuration. Under Windows Vista, launch comexp.msc, then expand Component Services\Computers\My Computer\Distributed Transaction Controller\Local DTC. Under Windows Server 2008 this is slightly easier to find, in Server Manager. In both cases, right-click on Local DTC, choose Properties and go into the Security tab. The exact choice of options probably depends on your scenario, but a good start is to switch on "Network DTC Access", "Allow Remote Clients", "Allow Inbound", "Allow Outbound" and "No Authentication Required'.
- Make sure that you allow DTC traffic through any firewalls. Again, if you run into problems, a good starting point is to temporarily disable all firewalls so you can find out whether that's the source of your problems.
Conclusion
In the last three posts I've documented pretty well everything I've learned over the past few months about getting MSMQ, WCF and IIS 7 playing nice, both on single machines and across multiple machines. Even though it took quite a while to figure all of this out, I still believe the architecture is both extremely flexible and simple to use - the total amount of code in this solution really is tiny. My only real complaint is that there isn't a lot of help available, either in the tools or on the web, to explain why things don't always work first time or how to go about fixing them. Through this post, I'm hoping my team's experiences will make the path a little smoother for you.
Update: By popular demand (OK, one person asked!), source code for the finished project is attached to this post.