OAB full downloads and server side paged pool depletion
Recently I participated in another fun critsit where a customer had massive issues caused by full OAB downloads. The customer had several hundred thousand cached mode Outlook clients trying to download the full OAB all at the same time. This caused a very high network bandwidth consumption issue, and also contributed to server side paged pool depeletion problems.
After troublshooting this further with the customer we saw that each cached mode Outlook client had several open connections to multiple public folder stores. This was a huge impact for the exchange servers. The reason is that each connection from a cached mode Outlook client was increasing the user’s token size. Each client connection to a Windows server requires some paged pool kernel memory. Eventually this lead to paged pool depletion problems on some of the servers. Once the servers end up in this state, it is all down hill from there.
In this blog I list out some of the top offenders that can cause full OAB downloads. I have also linked a MUST read blog in regard to Tokens and Memory Usage.
PLEASE READ THIS BLOG -> Large Security Tokens and Kernel Memory Exhaustion
Here are some of the top offenders that I was talking about:
1. PDN Changes
2. The 1/8 Rule
3. Invalid proxy address’s that are over 64 characters – (fixed with a hot fix) (Signature Mismatch)
PDN Changes
1. If the PDN table changes in the OAB by either a new PDN or removal of a PDN then all Outlook cached mode clients using V2 of V3 OAB will also attempt a full download. The PDN table is the set of all PDNs (Parent Distinguished Names) found in the directory.
2. Manually modifying a legacyExchangeDN in the AD to create a PDN that did not exist before. This most often is done by accident if someone is editing this value and mis-types the value, thus creating a new PDN.
3. With Exchange 5.5 and ADC in place, creating a new container in 5.5 and inserting an object into it, or deleting the last object in a 5.5 container.
4. With Exchange 5.5 and ADC in place, ADC set to replicate the container hierarchy to 5.5, and the administrator creates new mail-enabled objects in a new AD container. The ADC will create the new container in 5.5 and back-replicate the new 5.5 distinguished name as the legacyExchangeDN of the AD object creating a new PDN.
5. Add an administrative group. The first mailbox created on a server in this AG will cause a new PDN to show up in the directory.
6. Deleting the last object with a particular PDN in its legacyExchangeDN or proxy Addresses. Example: A few years after consolidating and deleting a site, the last mailbox formerly in that site is finally deleted. The x500 placeholder is gone and reduces the size of the PDN table.
7. Adding/removing/modifying an X500 proxy address with new PDN. This can be done using ADU&C.
If the X500 address is in the local org, but the organizational units and containers are new or mistyped, a PDN will be added or deleted from the table.
The reason why this is such a problem is because the pdndex.oab file can not be re-indexed after it has been created on the client side. This can only happen on a full download. To stop Outlook clients from doing full downloads is to make sure that both the client and server must be running SP2. If the client is on SP2 it should automatically update to the version 4 providing the client is already using a Unicode profile and that they actually connected to the server where the v4 is. In SP2 we apply the diff files differently and do not rely on the PDN table.
8. Any applications that have the ability to write data to the active directory. Example - Importing proxy addresses or legacyExchangeDN's to the active directory and the import filters are set in correctly.
For more information on PDN changes please refer to this blog: OAB PDN Changes and Exchange Site Consolidations
The 1/8th Rule
When Outlook 2003 has not downloaded a .diff file for several days, and the sum of the .diff files is greater than 1/8th the size of the whole offline Address Book, Outlook 2003 downloads the whole offline Address Book. If there are many Active Directory objects that have a change to an attribute that is in the offline Address Book, that change may cause the .diff file to be greater than 1/8th the size of the whole offline Address Book. Therefore, all Outlook 2003 clients try to download the offline Address Book.
For example, the following changes might cause the .diff file to be greater than 1/8th the size of the whole offline Address Book:
1. You update the phone numbers of a large set of users with a new area code.
2. You add each user's department information to the directory.
3. You add a new address type or a new address book.
Outlook clients fail to apply the OAB diff files due to signature mismatches.
This problem occurs during the OAB Generation process when OABGen.dll reads in objects from Active Directory that contains a proxy address that exceed 64 characters in size. Objects that have proxy addresses that are longer than 64 characters are legitimate objects as far as the OAB Generation process is concerned, however the problem is when the Outlook client downloads the diff file it will encounter a signature mismatch. Once the Outlook client fails to apply the OAB difference file, the Outlook client will be forced to download the full OAB message to correct the issue.
Note: The 64-character limit includes the at sign (@).
Note: By default, the Offline Address Book logging feature is enabled in Outlook 2003.
These events are similar to the following one you will see with Outlook logged enabled:
Event Source : Outlook
Event ID : 27
Event Type : Error
Description : OAB ModDif failed. (Details record in event data).
Event Source : Outlook
Event ID : 27
Event Type : Information
Description : Starting OAB Download.
Event Source : Outlook
Event ID : 27
Event Type : Information
Description : OAB Download Succeeded.
Microsoft has confirmed that this is a problem and there is a hot fix available for this:
Microsoft KB Article for this: 909565 - When you run Outlook 2003 in Cached Exchange Mode, all the Outlook clients in the Exchange Server 2003 organization try to download the offline address book at the same time - https://support.microsoft.com/default.aspx?scid=kb;EN-US;909565
Large Security Tokens and Kernel Memory Exhaustion caused by Outlook cached mode client connections.
There are certain situations where and Exchange 2003 Server can experience a Windows kernel paged pool depletion problem caused by token sizes and the amount of Outlook Cached mode client connections.
Symptoms Experienced:
1. Users experience slow or unresponsive performance.
2. Server Bluescreening
3. You may have to restart the Exchange server to resume typical performance levels.
4. Paged pool allocation failure events that are similar to the following are logged in the System log:
Event ID: 2020
Source: SRV
Type: Error
Description: The server was unable to allocate from the system paged pool because the pool was empty.
CAUSE: This problem occurs when paged pool memory is exhausted because of too much memory being allocated to client access tokens.
Microsoft has confirmed that this is a problem and there is a hot fix available for this:
Microsoft KB Article - 912480 - An Exchange Server 2003 server that hosts many Outlook client sessions may run out of paged pool memory - https://support.microsoft.com/default.aspx?scid=kb;EN-US;912480
PLEASE READ THIS BLOG --> Large Security Tokens and Kernel Memory Exhaustion
How can we try to control the network consumption by using OAB Throttling?
The answer to this is throttling the bandwidth from the server. By default, every client request for a full offline Address Book download is served immediately, and the public store does not limit the number of concurrent full offline Address Book downloads that can occur.
Example 1: If a public store that supports 10,000 users receives 3,000 requests in one hour and the offline Address Book size is 100 megabytes (MB), the server must deliver 300 gigabytes (GB) of data. This traffic could potentially overload a 100-megabits-per-second (Mbps) local area network (LAN) for longer than ten hours. This traffic could potentially overload a gigabit-per-second LAN for longer than one hour.
How do we calculate the largest OAB message size?
1.To find out the biggest message size you can run OABInteg /s:exchsrvname /t:oabfldcheck /v:2 /l against the public folder information store that holds your oab files (not a replica). This will dump out the message and attachment information. The messages that are in the information store contain the attachments and they are in compressed state.
Here is an example of the Oabfldcheck summary:
Scan Completed
+------------------+
Message Class Normal found: 2
Message Class Differential found: 175
Message Class Unknown found: 0
Message Attachments found: 503
Messages found but unable to read the properties: 0
System folders found: 3
Highest sequence number found: 231
Lowest sequence number found: 16
Biggest attachment found: 16700 Bytes
Smallest attachment found: 102 Bytes
Biggest message found: 60234440 Bytes
Smallest message found: 106 Bytes
You can then take the size of the largest message (here it is 60MB) x total users downloading the OAB x request per hour = number of data to be delivered by the server.
To download OABInteg please see Microsoft KB Article ID: Q907792 - Description of the Offline Address Book Integrity (OABInteg) tool
How to enable OAB Throttling
Use one of the following methods to determine the OAB Bandwidth Threshold data value to use:
Method 1 – Less Accurate
1. Set the value preemptively. *Based on your knowledge of the network topology* and how full offline Address Book downloads from multiple servers can add up to overload the LAN, set the thresholds on the individual servers to the largest values that are consistent with not overloading the LAN.
Bandwitdh Example: Type 5000 to configure the server to use 5000 kilobytes per second (KBps) as the bandwidth threshold for offline Address Book download throttling. 5000 KBps is approximately 40,960 kilobits per second (Kbps), or 40.96 megabits per second (Mbps).
Method 2 – Most Accurate
1. Set up performance monitor and watch the extended MSExchangeIS Performance Monitor counter OAB: Full downloads bytes/sec during a company-wide download that overloads the LAN. This can be a short period of time.
2. Make note of the bandwidth used by offline Address Book downloads on each server.
3. Set the threshold values to a percentage of those values. For example, set the threshold values to 60% of those values.
To turn on this feature on for a public store server that is used for offline Address Book distribution, follow these steps.
Warning: If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly.
***Use Registry Editor at your own risk***
1. Click Start, click Run, type regedit in the Open box, and then click OK.
2. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem\
3. On the Edit menu, point to New, and then click DWORD Value.
4. Type OAB Bandwidth Threshold (KBps) for the name of the DWORD, and then press ENTER.
5. Right-click OAB Bandwidth Threshold (KBps), and then click Modify.
6. In the Base area, click Decimal.
7. In the Value data box, type the value that you want to use, and then click OK.
8. Click OK, and then quit Registry Editor.
Things you need to Understand about OAB Throttling
1. To return to the default behavior, delete the registry value.
2. This feature does not decrease the overall number of downloaded bytes by the Outlook clients.
3. When you limit the bandwidth that is used by full offline Address Book downloads, this feature may extend the time elapsed until all clients receive their updated full offline Address Books. Therefore, you should only use this feature if you have very large offline Address Books and you must protect your LANs from overloading. Additionally, you should set the threshold value as high as possible.
4. After you apply this registry setting, when an Outlook client tries to download a full offline Address Book, the public store determines the average full offline Address Book bytes that were downloaded over the previous ten seconds. One of the following two behaviors will occur:
- If the value is less than the OAB Bandwidth Threshold, the client can continue with the full offline Address Book download at full speed.
- If the value is more than the OAB Bandwidth Threshold, the client cannot continue with the full offline Address Book download, the extended MSExchangeIS Performance Monitor counter OAB: Full download attempts blocked is incremented by one, and the Outlook client receives the an error message.
5. Clients that are using Microsoft Office Outlook 2003 SP1 or earlier receive the following error message: 'Microsoft Exchange Server' reported error (0x8004010B) : 'Unknown Error 0x8004010B' -> The Outlook client will try to download the full offline Address Book again, one time every hour, until it succeeds.
Dave