But I Really Need To Open 500 PSTs!

Today we’re revisiting an issue I first encountered in Outlook 2003. We had a customer, a law firm, who kept all the mail associated with a particular case in a PST. Their users would pass these PSTs around, and depending on their case load, a single user might have over 100 PSTs open at the same time. This worked fine when they had Outlook 2000, but when they installed Outlook 2003 they found if they tried to open more than 20 or so PSTs, they started getting errors such as “Outlook failed to add the personal store to the session” and "Out of memory or system resources. Close some windows or programs and try again."

Although we don’t recommend opening so many PSTs at the same time, we investigated to find out what had changed, and whether we could help them out. In order to understand what had changed, it helps to know a bit about how the PST provider, and by proxy, Outlook, manages open PSTs.

There are two mechanisms involved in the PST provider’s management of PST files to consider. The first is the per-PST cache. Any time you’re dealing with random reads/writes to a file, maintaining an in memory cache is critical to performance. The PST provider uses an algorithm based on the file size and other internal details about the PST to determine how large of an in-memory cache to allocate. In Outlook 2002, we greatly increased the size of the caches we use for all types of PST files. Larger blocks of memory mean we consume more of the address space available to the process. In practical terms, this meant we could open about 20 average sized PSTs before we were no longer able to allocate a block of memory large enough to hold the cache for the next file. As a test, I tried to open brand new PSTs and was able to get 85 of them before we ran out of memory.

Based on our investigation, and the needs of this and other customers, we decided to go in and fine tune the cache sizes to allow more PSTs to be opened at once. This involved a tradeoff in performance, so we placed this change behind a registry key. The KB article 907651 documents the hotfix we generated for Outlook 2003, as well as the UseLegacyCacheSize registry key. Outlook 2007 includes the fine tuned cache as well, but the path to the registry key changes slightly. Whereas in Outlook 2003 the key is under “HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\ Outlook\PST”, in Outlook 2007 you’ll set it under “HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\ Outlook\PST”. (Incidentally, this fix was backported to Outlook XP too. There, the reg key will have a 10.0 in it.)

Now, once this registry key is set, if we try and stress test how many PSTs we can open, we’ll see we hit a practical limit of just under 300, but, unlike before, our address space isn’t exhausted. And where before, if your MAPI based process could only open 20 PSTs, you could always run another process and open 20 more, this new upper limit is spread across all MAPI processes on the box. This is due to the second mechanism the PST provider uses, shared memory.

As you might recall from my discussion of PST files in multiple profiles, the PST provider uses shared memory to track and coordinate access to PSTs across multiple processes. Each PST opened incurs a tax on this shared memory, and after enough PSTs are opened, there is no longer space in this block to track any more PSTs. As part of the work we did in introducing the UseLegacyCacheSize, we also increased the size of this shared memory block, which is what will allow you to reach nearly 300 open PSTs across multiple processes.

Now – just because you can open nearly 300 PSTs doesn’t mean you should. You’ll start to see noticeable performance issues when you get over 100 PSTs. In fact, if shouldn’t set UseLegacyCacheSize at all if you don’t need it. The smaller cache has a performance penalty. If you have an application which for some reason needs to open a large number of PSTs, I would consider designing it not to need all of them open at the same time. Only open the PST when you need it and unload it immediately after.