Microsoft Exchange and Search Folders

  • Article

The Exchange search folders tend to be a very complicated subject for a lot of administrators. There are two different ways of filtering messages in MAPI - (Search folders and restrictions).

  • A search folder is like a regular MAPI folder, except that instead of containing real messages, it contains only links to messages in other folders which meet some specified restriction. A search folder is created by calling IMAPIFolder::CreateFolder with a folder type of FOLDER_SEARCH, followed by a call to IMAPIContainer::SetSearchCriteria to specify the search restriction and search domain. The search domain may be an arbitrary set of folders in the same store, and may be recursive, so that messages in any sub-folder of the specified domain folders are also searched.
  • A “restriction” is formed by calling IMAPITable::Restrict on a MAPI Table. The resulting table shows only the items which match the restriction specified in the Restrict call. A restricted contents table can show only messages from a single folder. This usage sometimes causes confusion with the MAPI-defined structure which describes the filter criteria for a search folder, which we also call a restriction.

The only way that the contents of a search-results folder can be modified is through a call to SetSearchCriteria. For more information about the implementation of SetSearchCriteria, see Knowledge Base article 260322: How To Search Folders with the SetSearchCriteria Method.

Search-results folders have the following limits:

  • Messages cannot be moved or copied into search-results folders.
  • Search-results folders cannot contain subfolders.
  • Clients cannot make a search-results folder the subject of a search.

When a client (any application with search capabilities) sets up a restriction on a contents table object (which is known as a VMSG – View Message Object), we will create a hidden search folder which searches the original folder, and will point the VMSG at the search folder. Clients have no knowledge that this has occurred, and will only that some of the messages have disappeared from the table. This is not a fast process because even a restriction on a small table requires that we create a new folder row and a new message folder table in Jet. Creating tables via Jet will cause us to log transactions because we are creating objects. Excessive logging can lead to complications and 623’s (Version Store Out Of Memory). For more information on Version Store please see this blog: "Troubleshooting Version Store issues - JET_errVersionStoreOutOfMemory". In addition to that blog I also reviewed an article that we wrote and we blogged about it on the Exchange Blog site: "Version Store issues revisited - Event ID 623 at Information Store service startup".

Now there are two types of search folders (Temporary and Permanent) that can get created. Search folders are the folders that clients create when they perform a search of a folder and its subfolders. The search folder contains information such as the folders that are being searched, the messages that meet the search conditions, and so on. When you create a hidden search folder this is actually a “permanent” part of the database, it can be reused later when someone performs the same restriction. Search folders are very much like regular folders. Each search folder has a single row in the folders table. Instead of a message folder table, there is a search table with a name like “S-1-4309”. The S denotes that this is a search folder and the following [1-4309] is the FID of the search folder.

Search tables will contain a subset of message folder tables that have columns. Ideally you want to have only a few columns and each row should be just a pointer to the “real” message in another folder.

Search folders can be created by a number of applications like Outlook, or any 3rd party application that issues searches for messages in an information store. The problem starts to manifest itself when a lot of search folders are created and never deleted by the application that creates them.

On temporary or normal folders, there is a property called ptagSearchBackLinks that lists all of the search folders that reference this normal folder. When items in the folder change (when messages are added, modified, or deleted), the folder must notify the search folder through the ptagSearchBackLinks list. When search folders are deleted (from the applications that created them) or when the client closes the 'Find' dialog box, the folders are not necessarily removed from the ptagSearchBackLinks list the same number of times that they were added.

What is a Search Queue and how is it used in Exchange?

When Exchange has a larger search operation it will use the search queue. When a search event is issued against a search folder, we create a task and we enter that task into the databases search queue. The search queue is a Jet table and is a permanent part of the database. Each row in the search queue will represent a particular search event that was previously entered. For example “search folder ‘Inbox’ with respect to a search folder called ‘Dave’s Search Folder’, and create links for any messages which match”.

When a search folder is created, or other operations occur which cause a search to be modified, the appropriate search events are added to the search.  Once the transaction has been committed, Exchange will then call a function to start the search task operation. This will process the search events listed in the databases search queue. This is done so if the information store is shut down or crashes before the search is complete, the search event will still be in the search queue.  When Exchange starts up, it will also call a function to start searching for any search tasks events in the search queue.

When a restriction is started, the same search events are entered in the search queue as for a regular search folder.  Rather than starting a separate background task to actually perform the search, restrictions process the search events immediately on the RPC thread before returning.

Having a ton of search folders will impact an exchange servers performance. When messages are created, modified, or deleted, all of the search folders linked will be updated immediately, within the same transaction as the original message operation. Since all search folders are updated in one big transaction, all search folders need to be locked before the transaction begins. When the exchange server starts to process batches of transactions it will begin to use up the stores Version Store cache to process these transactions.

There is a way that this can be controlled to a small extent and you can age out the views. For more information please see this article: How to Temporarily Delete Existing Search Folders.

I have also written a tool that will allow you to take an isinteg.pri file and search through it to look for users that have a very high amount of search folders.

Download Link: https://code.msdn.microsoft.com/Release/ProjectReleases.aspx?ProjectName=FindSearchFolders&ReleaseId=1456

Here is what the output looks like if you are search for a mailbox that has a high item count and high search folders. The action plan for finding the mailbox with the high search folder count is listed at the end of this output.

02-17-2010 - Find Search Folders Output - 02-17-2010-11.02.09.txt created. Program Logging - Enabled.
***************************************************************************************
NOTE: There is no support for this tool and will be used as is!
Purpose: Scan an isinteg.pri file and find mailboxes with a large number of search folder and backlinks
02-17-2010 - Trace time: 02-17-2010-11.02.09
***************************************************************************************
02-17-2010 - Program Initialization - Start
02-17-2010 - Create Program Directories - Start
02-17-2010 - Directory already exists: C:\Documents and Settings\dgoldman.NORTHAMERICA\Application Data\FindSearchFolders
02-17-2010 - Directory already exists: C:\Documents and Settings\dgoldman.NORTHAMERICA\Application Data\FindSearchFolders\Logs\
02-17-2010 - Directory already exists: C:\Documents and Settings\dgoldman.NORTHAMERICA\Application Data\FindSearchFolders\Backup\
02-17-2010 - Create Program Directories - Complete
02-17-2010 - Program Initialization - Complete
02-17-2010 - Getting current application run path: D:\SourceProjects\E12SP3\sources\tools\store\src\FindSearchFolders\bin\Debug
02-17-2010 - Starting Isinteg parse - Start
02-17-2010 - Initializing Isinteg parser  - Complete
02-17-2010  -  Search size set to: 75
02-17-2010  -  Isinteg is from: Server name: IsIntegStore
02-17-2010  -  Isinteg was run against database: Database name: SG1\DB1
02-17-2010  -  Processing Mailbox Table
02-17-2010 - ==========================================================
02-17-2010 - Processing Indexes
02-17-2010 - ==========================================================
02-17-2010 - Display Name = Not Found - MBX Table dump only
             Parent FID=0006-0000003039AC
             Root FID=0006-0000007D45A9
             Restriction=
             Name=Inbox
             Folder FID=0006-00000030084C      
             Folder Type=1
             Search Folder Count = 501
             Backlink Folder Count = 501

.. Truncated for readability ...

02-17-2010 - ==========================================================
02-17-2010 - Processing Mailbox Table
02-17-2010 - ==========================================================
02-17-2010 - Mailbox - Goldman, Dave - RootFID=0006-0000007D45A9
02-17-2010 - Mailbox - User, Test - RootFID=0006-0000006A73B0
02-17-2010 - ==========================================================
02-17-2010 - Message Counter Table
02-17-2010 - ==========================================================
02-17-2010 - Parent FID=0001-000000000021
                   Root FID=0006-0000007D45A9
                   Folder FID=0001-0000001F54B2
                   Msg Count=9074
02-17-2010 - ==========================================================
02-17-2010 - Total number of mailboxes with over 75 search folders: 3
02-17-2010 - Highest number of search folders found in a single mailbox: 501
02-17-2010 - Parent Folder: Parent FID=0006-0000007D45AA
02-17-2010 - Highest offender is: Root FID=0006-0000007D45A9
02-17-2010 - Total folders that do not contain search folders: 0
02-17-2010 - Total mailboxes found on this information store: 247
02-17-2010 - ==========================================================
02-17-2010 - These are the possible folder types
02-17-2010 - ==========================================================
02-17-2010 - Folder_Root = Folder Type = 0
02-17-2010 - Normal_Folder = Folder Type = 1
02-17-2010 - Search_Folder = Folder Type = 2
02-17-2010 - MDB_Attachment_Folder = Folder Type = 4
02-17-2010 - Repl_Folder = Folder Type = 5
02-17-2010 - Categorization_Folder = Folder Type = 7 / (These are Categorized Views)
02-17-2010 - ==========================================================
02-17-2010 - Total Stats
02-17-2010 - ==========================================================
02-17-2010 - Total type Root Folders found: 247
02-17-2010 - Total type Normal folders found: 8261
02-17-2010 - Total type Search Folders found: 1
02-17-2010 - Total type MDB Attachment Folders found: 0
02-17-2010 - Total type Repl Folder found: 0
02-17-2010 - Total type Categorization (Views) Folders found: 1312
02-17-2010 - ==========================================================
02-17-2010 - NOTE: Folders with a type of 7 are cached views in the information store. For more information please see: https://technet.microsoft.com/en-us/library/bb124802(EXCHG.65).aspx To expire cached views from the information store, please see: https://technet.microsoft.com/en-us/library/bb124802(EXCHG.65).aspx

02-17-2010  -  Who has the highest search folder count in an isinteg dump
02-17-2010  -  ==========================================================
02-17-2010  -  Isinteg parse - completed
02-17-2010  -  Closing all file streams and log files

How to find which mailbox corrosponsed to the following object.

1. Look at each object and look for the  Root FID=0006-0000007D45A9

11-10-2009 - Display Name = Not Found - MBX Table dump only
Parent FID=0006-0000007D45AA
Root FID=0006-0000007D45A9
Restriction=
Name=
Folder FID=0006-0000007D0F83
Folder Type=1
Search Folder Count = 501
Backlink Folder Count = 501 

2. Go to the top of the log and search down and the mailbox will be found in the mailbox table. This is your offender.

Dave