Memory Growth in BizTalk Messaging
By Kartik Paramasivam and Raied Malhas
In this document we list various reasons that might lead BizTalk to get into an Out Of Memory situation and then suggest mitigations/solutions for such conditions.
1. Introduction to BizTalk Hosts:
Before discussing Out of Memory conditions, let us talk about hosts and how adapters are mapped to hosts.
Adapters (receive and send) and orchestrations run under the BizTalk NT Service. An instance of the BizTalk NT Service is called a host instance. There can also be receive adapters (HTTP/SOAP) which can run under in any other process (e.g. IIS) instead of the BizTalk NT Service. Such adapters are called isolated adapters.
Hosts in BizTalk have 1 or more host instances (process). The default BizTalk installation has 2 hosts.
1) Default host
2) Isolated Host.
Note: You can only have 1 host instance per host on a given server.
The following is the mapping between end points (receive or send port) and hosts:
Send Port -> Send Handler -> Send Host
Receive location -> Receive Handler-> Receive Host
Typically you should create separate receive hosts for receive adapters and send hosts for all send adapters. Doing this gives each adapter a separate process to live in. This guarantees that one adapter will not adversely affect another. Also if your BizTalk host/process goes Out of Memory, you will know which components are running in that process.
2. The Obvious Suspects for Out of Memory conditions:
When the process goes Out Of Memory (OOM), you need to 1st find out what components are running in that process. If possible, follow the recommendations listed above for separating out receive/send adapters and orchestrations into different hosts. Once you have done that then you can check if one of the following 3 conditions exists in the process that is going OOM.
1) You are executing transforms/maps on relatively large messages in receive/send port or XLANG. The point here is that XSL transforms load the whole message in memory to transform them.
Solution 1: Decrease the number of messages that your process operates on concurrently (section on Send Host below might give you some idea).
Solution 2: Decrease the size of your XML message that you are trying to transform.
2) You are executing XML receive/send pipeline on a document that contains one or more items such as the following:
- Large attribute values
- Large element values
- Large attribute or element tags.
If one of the above applies to your XML document then that item is fully loaded in memory.
Solution 1: Try to limit the size of the above entities.
Solution 2: If you can’t limit the size, then make sure your process doesn’t concurrently operate on multiple documents. (See sections below on limiting concurrency)
3) You have a custom pipeline component or adapter, which loads the whole document in memory. Most of the components shipped with BizTalk (except transforms) support streaming as opposed to loading the whole document in memory, and hence have a low memory foot print. However, we have observed that custom pipeline components written by customers may or may not support streaming.
3. Out of Memory in Send Host:
The Send Host of BizTalk Adapters can run Out of Memory when it is processing a large number of messages (i.e. when the system is under high stress loads). This can cause the Send Host memory utilization to rise rapidly causing the system to get into an Out Of Memory condition. Usually, the larger the messages being processed the faster the memory utilization of BizTalk’s SendHost process will be.
Calculating the Maximum number of messages that will be loaded in-memory by BizTalk’s Send Host process:
Under high stress loads, the BizTalk send host process will load as many instances of messages into memory as it can until the number of messages exceeds the “HighWatermark”. By default in BizTalk Server 2004, SP1, this value is 200 for messaging. Below you’ll learn how to configure this value, more details on Watermark settings are available in the “BizTalk Server Performance Characteristics” document at: https://www.gotdotnet.com/team/wsservers/). So, if you have a Dual processor server that is processing a large number of messages, the BizTalk host will load a maximum of:
[HighWatermark Value] * NumberOfProcessors
= 200 * 2
= 400 messages in-memory
For a Hyper-threaded server:
If you have a hyper-threaded server, then the number of processors perceived by the BizTalk send host doubles the actual number of processors. So if you had a dual processor BizTalk server and if we assume that the default value for the maximum number of message instances processed at once is 200 (so, ”HighWatermark” value is equal to 200), then the BizTalk process hosting the Send Host will load a maximum of:
[HighWatermark Value]*[NumberOfProcessors for a Hyper-threaded machine]
= 200 * [NumberOfProcessors * 2]
= 200 * [2 * 2]
= 200 * 4
= 800 messages in-memory
So, in a case where you have a dual processor hyper-threaded BizTalk server with 2GigaBytes of RAM that’s under high stress, there is a possibility that SendHost will get into an Out of Memory state. The process in this case will be loading 800 messages (as calculated above) in memory. The following graph shows an example of such a case where the memory usage of the send host process grew very rapidly to 1.5 Giga Bytes in less than 15 minutes:
Graph1: An example of a test that hit an Out of Memory condition in a SendHost containing FILE Adapter
Note: The memory footprints of BizTalk hosts can be monitored using Performance Monitor to view “Private Bytes” counter for the required host (receive, send, etc…) instance under the “Process” object.
Note: On a 32 bit box, the process can only grow to around 1.5 GB Max (2 GB in some rare cases) even if you have more physical memory to spare (i.e. 8 GB RAM for e.g.)
How to avoid Out of Memory in the SendHost:
Fortunately, the Out of Memory condition can be avoided by configuring the setting that controls the maximum number of messages that can be loaded into memory concurrently.
To configure the maximum number of message instances, in BizTalk’s Management Database (default name is: “BizTalkMgmtDB”), open the “adm_ServiceClass” table. You’ll see for “Messaging InProcess” row, by default, the value for LowWatermark is set to 100 and for HighWatermark to 200. The HighWatermark value is what dictates the maximum number of message instances that can be loaded in-memory of the Send Host process. Change the Low and High Watermark values to lower numbers, an example of lowered values are:
LowWatermark = 25
HighWatermark = 50
If those values (25-50) still show OOM conditions, you should decrease those values more until you stop hitting the Out of Memory condition.
The following graph shows an example of the same test case that the graph above shows. In the graph below, the Low and HighWatermark values were changed to 25-50, respectively. Please note how the memory remained constant without growing to get into an Out of Memory:
Graph2: An example of a test that avoided hitting an Out of Memory in a SendHost containing FILE Adapter
Note: Decreasing the Watermark values can decrease Performance. So, only decrease those values if you run into an Out of Memory condition.
4. Out of Memory in Receive Hosts:
It is fairly rare to see a host containing a receive adapter to go out of memory. If that happens it is typically due to one of the cases listed in section 2.
If none of the suggested recommendations in Section 2 can be performed, then reducing the number of messages processed concurrently can help protect the receive host from getting into an out of memory condition.
Note: Remember however that by doing so, we decrease the concurrency and hence the throughput/performance will decrease.
1) Reduce the messaging engine thread pool size: A user can control the number of threads used by the messaging engine to publish messages into the message box. By reducing the number of threads used by the messaging engine we reduce the rate at which the receive adapter will receive messages into the message box. Remember this setting only needs to be done for the host corresponding to the receive handler for the adapter.
By default the messaging engine creates 10 threads/cpu for
publishing messages.
To specify the messaging thread pool size
1. Click Start, and then click Run.
2. In the Run dialog box, in the Open box, type regedit, and then click OK.
3. In Registry Editor, expand HKEY_LOCAL_MACHINE, expand SYSTEM, expand CurrentControlSet, expand Services, right-click BTSSvc* (you would first need to determine which of the BTSSvc* regkeys correspond to the host corresponding to the receive handler), point to New, click DWORD Value, type MessagingThreadPoolSize, and then press ENTER.
4. In Registry Editor, double-click MessagingThreadPoolSize.
In the Edit DWORD Value dialog box, in the Value data box, type a number between 1 and 30, and then click OK.
5. Note for Custom Adapters:
If you have verified all of the above conditions and if you think your custom receive/send adapter is causing the OOM conditions, then you need to look at the source code of your adapter and verify the following:
The following only applies to Adapters written in Managed code:
The adapters get an object of type IBTTransportBatch using the GetBatch() API on the TransportProxy object. Once the adapter is done using the IBTTransportBatch object the adapter needs to do a Marshal.ReleaseComObject( batch) in a loop to release the object.
Basically the .net GarbageCollector does not kick in in time to release the unmanaged memory. Hence failure to call ReleaseComObject() will result in what looks like a memory leak.