Short history of MSMQ Adapters in BizTalk

  • Article

In the beginning: MSMQ AIC

In the beginning there were no adapters. In BizTalk 2000 and 2002 they were called AICs – Application Integration Component. MSMQ AIC for BizTalk 2002 used MSMQ API from Windows Message Queuing and that was it. Windows Message Queuing was doing the network related job and it also stored intermediately messages in MSMQ queues on the machine before they were read by MSMQ AIC of BizTalk or sent to the world over the network.

It was simple and straightforward, but it had a few problems on its own:

  • Reliability: once the message came to some machine and before the message was picked up by BizTalk, the crash of the machine would destroy the message. Not the best story.
  • Double administration: you had to configure MSMQ queue and separately you had to configure MSMQ AIC for BizTalk. Double storage also meant double trouble. You had to check two places with two different tools in the search of a lost message.
  • Double hop: an incoming message was first stored in MSMQ queue on the disk, and then had to be read and stored in BizTalk database. This did not look like a good idea from the performance point of view.
  • Transactional story. Suppose you send a messages to a partner application. In MSMQ you have eventually check the dead letter queue to see if it was delivered or not. That was  inconvenient.

BizTalk 2004: MSMQ/T

These and few other reasons were considered when making the decision to create MSMQ/T Adapter for BizTalk 2004. MSMQ/T Adapter is a full implementation of MSMQ Server that lives in BizTalk. When BizTalk 2004 starts, it also starts a subprocess that listens to port 1801, the port for MSMQ protocol. MSMQ/T adapter is based on Windows Message Queuing server 2.0 and it is actually done by the same people who implemented Message Queuing in Windows.

Now a lot of things become much nicer:

  • No local storage. Hence no separate administration, no extra cracks where the message can fall.
  • No local storage. Hence no extra hop. Message goes straight into BizTalk database.
  • No local storage. Hence no need to cluster BizTalk machine. If the machine crashed before message is safely in the BizTalk database, no ack is sent back and MSMQ protocol takes care of resending the message.
  • No local storage. Hence no need to check the dead letter queue. You can route acks and nacks straight back to the BizTalk applications. Your orchestration sends a message and waits in a try-catch block. If something goes wrong, you catch an exception and know that the message did not make it through. Of course, it takes a bit of time to get ack/nack over asynchronous protocol, but orchestrations are stateful long-running applications, they can wait.
  • And, irrelevant to the local storage, but a great feature: large message support. MSMQ messages are limited to 4 Mb in size. MSMQ/T is able to break a large, say, 100Mb message, into the 4Mb chunks and reassemble them into a single message on another side. MSMQ/T large message support is fully streamed and hence, technically, there was no limit to the size of a message, although if you used XML pipeline, you could hit a problem with XML parser in .Net 1.1 around 2Gb. We successfully passed >1Gb message in out tests, but for support purposes we’ve put more reasonable lower limit.

MSMQ/T also had a few limitations:

· It was based on 2.0 code and hence it did not had most of the 3.0 features, including message queuing over HTTP and few others. When MSMQ/T work had started, MSMQ 3.0 code base was simply not yet developed.

· It implemented only transactional, exactly-once-in-order (EOIO) delivery. The thinking was more or less along the lines, “If a user does not need transactional, why does he need message queuing?”

· No remote read from MSMQ/T “queues”. The reason was that while BizTalk receive locations played the role of a logical MSMQ queue, there was no physical queues in MSMQ/T. Once the message was received, it was at the same time in the same transaction routed to destinations, hence BizTalk “queues” were by definition always empty.

The advantages of MSMQ/T had their trade offs:

  • Now that BizTalk become MSMQ Server listening on the port 1801, running Windows Message Queuing on the same machine become somewhat difficult. If you had Windows Message Queuing running on the machine, BizTalk would fail to start because of a conflict on the port. And once BizTalk starts, Windows Message Queuing was not available. As we found in Beta programs, many users wanted to run regular MSMQ applications on the same box, so it was a handicap. We addressed that with two solutions:
    • First, we figured out the way to install BizTalk and Windows Message Queuing side by side, see here[1]. That required two IP addresses, two computer names, had a few limitations, but otherwise worked ok.
    • Second, we stopped adding MSMQ/T adapter by default. It was still an integral part of the engine, but BizTalk admin had to explicitly “Add Adapter” in the admin console, for BizTalk starting the process. Hence, until you added MSMQ/T adapter, you were able to run Windows Message Queuing. The other side of the story was that once you added MSMQ/T you could not get rid of it other than by reinstalling BizTalk 2004 – a problem we addressed in BizTalk 2006.
  • MSMQ/T looked slower. When you send a message from MSMQ to MSMQ, it is delivered to the destination machine, dropped into the local queue, and that’s it. The interesting part of the message’s life just starts, but the queue on the sender machine is already empty, and it feels like the message was already delivered. With MSMQ/T the message have to go through BizTalk pipelines and then transactionally stored in BizTalk database along within a batch of other messages. In fact, while being stored in the BizTalk database, it was also transactionally routed to BizTalk applications and send ports interested in this message. It takes longer than just dropping a message into the local storage, hence users occasionally observed messages piling up in their outgoing queues on MSMQ machines.
  • In fact, MSMQ/T was slower even in the end-to-end scenarios. The reason was that combining two or three transactions in one is always more complex and slower than doing these transactions independently. MSMQ/T combined in a single act picking the message from the network, passing it through pipelines, submitting it into the MessageBox including routing it to internal BizTalk destinations all in order all the time. Combined into a single transaction with the network protocol designed for reliability, not low latency, resulted in a substantial performance toll.
  • Because MSMQ/T used transactional messages only, once you send from a single MSMQ machine to a single MSMQ/T queue, you could not scale. That is the same limitation as in regular MSMQ, but regular MSMQ allows non-transactional messages, which are not affected by this problem.
    You still can scale if you have multiple senders or multiple destination queues, because you will have different TCP/IP connections that could be load balanced.
  • Users were confused. “Hey, I’ve send a message to MSMQ/T, I check the MSMQ queue on the machine, and I see no message! In fact, there is no queue either!” Well, sure, there is no Windows MSMQ queue, it’s in BizTalk, and it’s already empty, because your message is already routed. Anyway, that was a minor problem.
  • Granted, the previous problem was also rooted in the fact that BizTalk HAT is not a match fro MSMQ MMC administration console when it comes to message queuing. That made administration and troubleshooting of MSMQ/T complicated.
  • Last, but not the least, the only way to include MSMQ 3.0 functionality was pretty much rewriting MSMQ/T using the new code from Windows, which would be a major effort. By that time MSMQ team was already busy working on Indigo, and BizTalk team was busy on a lot of other things in BizTalk, so maintaining MSMQ/T up to the date became a major problem.

MQRTLarge.dll

As already mentioned, a serious advantage of MSMQ/T was the ability to pass large messages. However, while passing large messages between two BizTalk machines was an attractive feature, it did not looked like a really huge winner. What would be great is passing large messages between BizTalk and MSMQ applications.

The problem was solved by providing MQRTLarge.dll that exposed an API similar to regular MSMQ and allowed regular MSMQ applications to send/receive large messages to communicate with BizTalk.

MQRTLarge.dll was supposed to be dropped on a non-BizTalk machine and used by MSMQ applications, and hence it is located in BizTalk 2004 SDK utilities. Because the API was slightly different to accommodate for large messages, the users had to update the MSMQ application code and rebuild it to take advantage of this feature, but other than that it was really nice. It still is.

The only deficiency of MQRTLarge.dll is that it accumulates the whole message in memory and hence available memory was an effective limit of how large message you can pass with it. We were able to pass messages up to 2Gb in size, but it required a good machine. Also, if two applications would send large messages at the same time, their memory demands will combine, which was another problem. But anyway, to have this problem we talk about really large messages several hundred megabytes in size. Most users, we know of, needed to push 4Mb limit of MSMQ just a little to 5-20 Mb, and so they never noticed this problem.

MSMQ/C Sample

BizTalk 2004 SDK also contained an MSMQ/C adapter sample that used standard Windows Message Queuing just like MSMQ AIC for BizTalk 2002. It was just a sample with limited functionality, it was never tested extensively for performance, stress or load, and it never was supposed to be used in the production environment, although we’ve heard of customers who did that. Again, it was just a sample.

MSMQ Adapter for BizTalk 2004 (see here [2] )

Late in BizTalk 2004 development cycle we realized that customers will also need a traditional MSMQ adapter, and so we started to build it. At this time it was already customary to refer to “the other MSMQ adapter” as MSMQ/C, that’s why this adapter also sometimes informally called MSMQ/C Adapter. I actually did that in this blog before. However, it is not the same thing as the sample. We’ve put roughly a year of development and extensive testing into creating it.

MSMQ Adapter for BizTalk is available as a free download from Microsoft.com Download Center[3].

It has roughly the same advantages and the same problems as MSMQ AIC for BizTalk 2002, but provides richer functionality. In particular, it supports large messages using MQRTLarge.dll. The 2Gb message mentioned above was successfully passed through MQRTLarge.dll when testing MSMQ Adapter. The adapter has the same limitations as MQRTLarge.dll, and to avoid processing of two large messages at the same time, there is a “Serial Processing” flag on MSMQ port properties.

MSMQ Adapter is the recommended way to use MSMQ transport with BizTalk.

Here are some advantages of MSMQ Adapter over MSMQ/T:

  • Standard Windows Message Queuing is used with all features of MSMQ 3.0.
  • Other applications using MSMQ will run on the same machine concurrently with BizTalk.
  • Availability of MSMQ administration and troubleshooting tools.
  • Most operations are essentially local, and hence much faster. The adapter just drops a message to a local queue or gets one from there. For remote read Windows Message Queuing API uses an RPC without all complexities and delays of MSMQ protocol. It still takes time to deliver the message, but it happens asynchronously to BizTalk, while the visible processing time is comparable to a File adapter.
  • The adapter is compact, lightweight and functionally rich.

Things you need to keep in mind when selecting MSMQ Adapter for your project:

  • High availability will require an NT cluster. Even so, you will need BizTalk 2006 to utilize it.
  • Ordered delivery in BizTalk 2004 is not achievable with MSMQ adapter (standard feature in MSMQ/T). It’s available in BizTalk 2006 with some configuration work.
  • If migrating from MSMQT, see the detailed upcoming article on possible caveats.

Going forward – BizTalk 2006

In BizTalk 2006, MSMQ Adapter is the main recommended way to handle MSMQ transport. BizTalk 2006 supports NT cluster, hence there is a solution for high availability. Also, in BizTalk 2006 you can setup MSMQ port for the end-to-end ordered delivery (you need to set correctly MSMQ port, MSMQ queue, and the receiving application/send port).

MSMQ/T adapter is deprecated in BizTalk 2006. It means that it is still there and fully supported, but it’s a warning that it may disappear in the next version of BizTalk beyond BizTalk 2006. So, it may be not a good idea to consider it for new projects, unless you have a really strong reason.

MQRTLarge becomes a part of MSMQ Adapter and hence supported the same way as the adapter itself. Notice, that although you technically can make two MSMQ applications exchange large messages, the only officially supported scenarios are where one or both communicating sides are BizTalk MSMQ or MSMQ/T Adapters.

 

[1] https://blogs.msdn.com/eldarm/archive/2004/10/13/241799.aspx

[2] https://www.microsoft.com/biztalk/evaluation/adapter/adapters/msmq/2004/default.mspx

[3] https://www.microsoft.com/downloads/details.aspx?FamilyID=CBA87D07-7F50-4D7B-A888-388D123F736E\&displaylang=en