Microsoft Message Queue - An Introduction

  • Article

In this series I would like to introduce you to MSMQ history, key concepts, dwell into painpoints, suggest workarounds, and give an insigth into to future.

What is MSMQ?

MSMQ enables Asychronous communicaiton between two different applications. Application A(Sender) sends a message to a queue hosted in MSMQ, Application B(receiver) receives and processes the message from the queue MSMQ is usually used in the following scenarios a. Where there is an impedence mismatch between the sending and receiving applicaiton b. Load-levelling scenarios c. Sending and Receiving applications need to be running at different times d. Sending and Receiving applications need to be running across network or even across the internet.

MSMQ enables you to send messages over a variety of protocols such as TCP, IPX (if you care anymore :-) ) , HTTP, PGM (Pragmatic Multicasting) MSMQ provides out of the box security and routing of messages.

The History

MSMQ was designed and developed by a Microsoft developement team in Haifa, Israel.

MSMQ version 1.0 was shipped with the NT Option pack. In this version MSMQ was integrated with SQL Server. The programming model was easy but the Installation and management was cumbersome.

MSMQ version 2.0 shipped as an optional windows component with Windows 2000. Active Directory was born, MSMQ was quick to grab the opportunity to replace SQl-server dependency with AD.

MSMQ vesrion 3.0 shipped as an optional windows component with WinXP and Windows 2003. Major new features like HTTP protocol support, Multicasting support, Distribution lists, etc were introduced. Security and reliability of the product was greatly enhanced in version 3.0

Key Conepts

MSMQ can be installed in two modes

  • Domain Mode - Integrated with Active Directory For MSMQ to be installed in domain mode the computer on which MSMQ is installed should be joined to a domain. The benefits of AD integratio are the following.
    • You can create and use both public and private queues: Public queues are one whose properties(metadeta) are published in the Active directory. Public queues can be discovered by querying the AD. All publics in an enterpirse can be managed through the AD Management console. This acts as a single point of management Note: The messages in the public queues are not stored in AD, only the properties are stored in AD. The messages are stored on the computer on which the queue is create. Private queues are one whose properties(metadata) and messages(data) are published on the computer on which the queue was created. The name "Private" is rather misleading. You can send and receive message to a private queue from anywhere on the network using the Direct format name addressing scheme. Infact this is the only addressing scheme available for pivate queues.
    • Integrated securit: MSMQ should be installed in domain mode if you require out of the box authentication and encryption of messages. This is because for authentication MSMQ signs the message using the user certificate registered in AD. For encryption the message is signed using the machine certificate registered in AD.
    • Routin: If you need to route messages between different sites then you need MSMQ to be installed in domain mode. All routing configurations are stored in AD.
  • Workgroup Mode - Does not require Active Directory If the computer is joined to a domain MSMQ will be installed in workgroup mode. In this mode you have the capability to create and use only private queues. You don't have the out of the box authentication and encryption features. Howevere MSMQ allows you to attach any standard X.509 certificate to the message, but the receiver application need to authenticate the message itself using crypto APIs. The routing feature is also not available if MSMQ is installed in workgroup mode. If you dont require out of the box security or routing workgroup mode is your friend because it is easy to install and manage and doesn't have any external dependencies (AD).

APIs

You can program MSMQ using 4 APIs

  • C++ API: Coding is a really tedious and cumbersome, but obviously the most performant b.
  • COM API: COM wrapper over the C++ API. Better coding experience. Can use VB or C++ to program.
  • System.Messaging: .NET wrapper over the C++ API (uses PInvoke to call into the C++ APIs). Considerably imporved programming experience over C++ and COM APIs. Great increase in productivity. Performance is equal to or better than the COM APIs.
  • Indigo Queuing programming model: The next generation distributed programming model. Built on top of C++ APIs. Service oriented programming model. Performance is equal to or better than System.Messaging. Best programming experience. Microsoft will make all its investments in Indigo going forward.

Queues

Queues are nothing but First-In-First-Out(FIFO) datastructures.Queues can be created using the MSMQ management tool inside Computer management console MMC or programatically. Queues can be created and used dynamically. However you need to be aware of issues that might be caued by replication delays in the case of public queues. Private queues does not have such issues There is no physical limit on the number of queues that you can create on a given computer

Messages

Queues hold messages. Messages contain several properties and body. Some properties such as priority, correlationID, messageID etc are used to specify the contract of the message. Some properties like Label, body etc can contain the payload or data. The total size of the message including properties and body is limited to 4MB. The body of the message can contain pretty much anyting. You can ship COM objects that supports IPersistsStream interface, .NET Data types that can be serialized through binary or XML serializer. You need to be aware of the fact that message occupy both virtual memory and page pool. A fixed number of page pool bytes are consumed per message. If you have a lot of messages (like 1 million) on the disk you will start putting pressure on the page pool. In normal situaltions where sender and receiver applications are working smoothly you don't end up with a lot of messages in the queue. However in scenarios where there is a huge impedence mismatch between the sender and receiver or the receiver applicatio is down you might end up queuing a lot of messages.

MSMQ supports specifying storage quotas at the machine level and queue level. This ensures that you get warnings in the form of Negative acknowledgements when you hit the quota. This can help you prevent situations you have too many messages in the queue which will put pressure on system memory and page pool.

MSMQ support two types of messages

  • Express: Express messages are not stored on the disk and hence will be lost if the computer restarts/crashes. Express messages as the name suggests are very fast. They are used in scenarios where you dont really care about message loss. For example a stock applications sends stock quotes every second and it doesnt care if one of the quote was lost, it can catch up int the next second. However the application wants the message to reach the destination as fast as possible.
  • Recoverable: Recoverable messages are stored on the disk and hence can withstand computer crashes/restarts. Sending and receiving messages

You can send messages to both local and remote queues. You can send messages using several protocols such as TCP, IPX, HTTP, PGM (Multicasting). Receive is always done over RPC. You can receive messages from both local and remote queues.

Messages can be send and received as a part of a transaction.

MSMQ supports two types of transactions

  • Internal transaction: You can send and receive messages to MSMQ as a part of a transaction
  • MSDTC co-ordinated transaction You can send and receive messages as a part of a transaction where other resource managers such as SQL-Server can participate.

You can send and receive transactional messages to both local and remote queues.

High Availability and Scalability

MSMQ provides high availability by the use of Microsoft Clustering service. MSMQ supports both Active-Active and Active-Passive clusering. MSMQ provides scalability in two scenarios.

Load-balancing

In this scenario you can have a farm of MSMQ servers behind a hardware or software load balancer. Clients can send messages to the MSMQ servers through the load balancer.

If you are using the TCP protocol to send messages you are limited to only non-transactional sends.

If you are using the HTTP protocol you can send messages both transactionaly and non-transactionlly.

Load-levelling

In this scenario the queues are hosted on a central server (clustered for high availability). Client applications send messages to the queues. The reveiver applications are hosted on a farm of MSMQ servers. They receive and process messages from the central queues. Messages in the central queues are levelled across the server farm.

Security

MSMQ supports authentication and encryption of messages. When you send a message you set certain properties of the message to specify whether you want authentication and encryption or not. You can also select the has algorithm and encryption algorithm that msmq uses to sign and encrypt the message.

You can set the "requires authentication" property on the queue thereby resticting the applications to be able to send authenticated messages to the queue.

You can also set the encryption property of the queue to accept only encrypted messages.

Note: Messages are encrypted only on the wire, once they reach the destination queue messages are decrypted and stored as plain text.

-Anandraj