Why is there a 4MB limit on MSMQ messages?


Here’s a request I’ve received from Dhaval Patel



I would really like to see some blog entries on how to handle messages greater than 4MB. I have heard some colleagues mention ‘COM Wrappers’ but I don’t like the idea. To begin with, most of us were thinking that starting with Vista, the 4MB limitation would go away… it’s a shame that this didn’t happen! I was really looking forward to putting MSMQ to good use via WF and WCF, but seeing this limitation is still there, it seems folks like me will just need to stick to SOAP for eternity 🙂
Please feel free to enlighten me with the history of why MSMQ was forced to handle only 4 MB messages?”


OK, Dhavel, in this blog we will dip into the history behind this limitation.


The first place to start is the excellent MSMQ FAQ document:



3.5    Why is there a 4-MB message size limitation?


There are two main reasons:



  • The Message Queuing driver (Mqac.sys) uses memory-mapped files for message storage, and the present implementation does not break a large message into several message fragments. The driver maps the files to kernel memory address space when Message Queuing runtime sends or receives a message. The kernel memory address space is limited to 16 MB, shared by all drivers and fonts. This is one of the main reasons for the message size limitation. The 16-MB kernel address space limitation also means that a cluster node can host up to three Message Queuing resources, assuming that Message Queuing is not running on the node computer. This is because each resource can consume its own 4-MB range from the kernel address space.

  • The Message Queuing code that manages sessions is not designed to deliver large messages and handle session acknowledgements (ACKs) in the middle of message delivery. This can cause sessions to be closed while a large message is still being sent, because an internal session ACK is not received.

A bit more information on this 16 MB area can be found in the “Deploying Message Queuing (MSMQ) 3.0 in a Server Cluster” document:



Q. Are there any limitations for multiple MSMQ resources in a server cluster?


A. Yes. Each instance of MSMQ on a server node maps 4 MB of the system view space when handling message activity. This results in a default limit of three active, working instances of MSMQ in a cluster node. In a server cluster with three MSMQ resources, a node could have four concurrent MSMQ services running — the service running on the local node plus the three services associated with the MSMQ resources. In such a scenario, message activity might be limited, resulting in resource failures. To overcome this, you should increase the size of the system view space memory pool on each node of a server cluster that is running three or more MSMQ resources. We recommend that you do this even if your node is running less than three.


So we’re talking about the system view space, the desktop heap for drivers. Although you can increase the size of this space with the SystemViewSize registry value, this will not affect the MSMQ message size in any way. The registry value is documented on the TechNet website:



Group and resource failure problems




  • The Message Queuing resource fails to handle message activity correctly which may result in resource failures


Although there is another registry value that does seem to set the MSMQ message size, you shouldn’t use it. This isn’t one of those “It’s not supported but here it is for you to try anyway” references – I do actually mean “you shouldn’t use it”. The value description for MaxMessageSize does say:


Caution




  • Do not change the value of this entry. Increasing this value might prevent you from sending messages larger than 4 MB. Decreasing this value might block transports between Message Queuing computers.


So, in summary, the 4MB limit is there for good design reasons which are still valid in the 21st century.


In the next blog I’ll go into ways to get around this limitation.


[[Edited 18th November, 2009]]


Here is an important note from the very useful Resource Management in MSMQ Applications article:



Message Size


MSMQ can support messages under 4 MB in size only, this includes Queued Components messages. Any attempt to send a message through the system that is larger than this will raise the insufficient resources error. Be aware that Unicode data takes up twice as much space as non-Unicode data, as two bytes are needed for each character.

Comments (9)

  1. I’m not sure that those sound like “good design reasons” — more like “limitations when we chose an implementation”. I’m definately interested in seeing how to get around it. I’ve run into situations where I had to take a less-than-ideal approach because of the 4MB limit. Most of my messages would have ended up around 1-2MB, but it was possible that sometimes we’d have a jumbo message of arbitrary size.

  2. MSDN Archive says:

    OK, I can see what you mean. I was trying to emphasise that the 4MB limit wasn’t an arbitrary choice made without thought. MSMQ’s use of a device driver to manage the storage files does provide very good performance but the cost is a limited bandwidth. There may be a better way of doing this now, several operating systems down the line, but it would need to be very compelling for the product group to scrap the existing approach. Maybe the 4MB bottleneck will become too painful for customers and so MSMQ is changed but that would be MSMQ 5.0 at the earliest.

  3. Dhaval Patel says:

    Thanks for the explanation, John. Definitely helps!!!

  4. Hi, is the 4 MB limit for all messages stored in a queue, or is it for per single message that can be stored in a queue. If it is for per single message, then what is the total size of the message queue?

  5. MSDN Archive says:

    Hi Gobinda,

    The 4MB limit is per message. The total size of the message header + message body cannot exceed 4MB.

    The capacity of an individual queue is determined by three things:

    1  the queue quota, if set in the queue properties

    2  the computer quota, if set in the message queuing properties

    3  the practical capacity of the machine to store MSMQ messages, which is variable.

    For discussions on machine capacity, check this post:

    http://blogs.msdn.com/johnbreakwell/archive/2008/02/29/what-are-msmq-s-limits-if-i-had-a-farthing-for-every-time.aspx

    Cheers

    John Breakwell (MSFT)

  6. Franklin says:

    Excellent explanation. Has the limitation been overcome ? Does the Biztalk Server support message payloads greater than 4MB ?

    Regards,

    Franklin

  7. MSDN Archive says:

    Hi Franklin,

    No, the limit is unchanged.

    BizTalk used MSMQ/T for payloads over 4MB but I think that adapter is now deprecated.

    Cheers

    John Breakwell (MSFT)

  8. Franklin says:

    Dear John,

     Thanks for the clarity. If the MSMQ(T) has been depricated, what suffices as the Messaging Provider for Biztalk?

  9. MSDN Archive says:

    Hi Franklin,

    Normal MSMQ can still be used by BizTalk.

    There is no replacement that easily handles large MSMQ messages.

    Cheers

    John