Outlook 2007 Beta Documentation - Notification Based Indexing Support

Article
05/10/2006

[This is now documented here: https://msdn.microsoft.com/en-us/library/bb821096.aspx, https://msdn.microsoft.com/en-us/library/bb821097.aspx, https://msdn.microsoft.com/en-us/library/bb821099.aspx, https://msdn.microsoft.com/en-us/library/bb821100.aspx]

This is preliminary documentation for Outlook 2007 Beta 2. It does not apply to earlier versions of Outlook.

Notification Based Indexing Support
A store provider supports Notification Based Indexing if it doesn’t require incremental or full crawls and indexes all the items through notifications. Sometimes these providers are referred to as “pusher” stores, since such a store provider will push URLs to the indexer by calling appropriate indexer APIs. These URLs are built based on algorithms documented below. Each URL corresponds to one message, folder or attachment. The indexer will pass the URLs to the MAPI Protocol Handler (MAPI PH) which will parse it open to determine which message/folder/attachment it corresponds to. MAPI PH then opens that object through MAPI and gets the properties to be indexed. The text is returned to the indexer so that it can word-break it and save the terms in the catalog. Some properties like bodies or attachments may require the use of the filter host which is a different process. The filter host will basically use IFilters to crack the documents and get the text so that the indexer can then word-break the text.

New Store Support Mask Flag
This flag is set by pusher stores.

Definition

 #define STORE_PUSHER_OK ((ULONG) 0x00800000)

If the store provider sets this flag the MAPI PH won’t crawl the store and the store is responsible to push any changes through notifications to the indexer to have messages indexed.

Usage
This flag can be retrieved by getting the property PR_STORE_SUPPORT_MASK from the store.

New Properties for the MAPI PH

 #define PR_PROVIDER_ITEMID         PROP_TAG(PT_BINARY, 0x0EA3)
#define PR_PROVIDER_PARENT_ITEMID  PROP_TAG(PT_BINARY, 0x0EA4)

These properties are used to identify an item or a folder by the store provider. They are retrieved when the store provider gets the search results from the search engine and are used to identify which items matched the query. Store providers can provide provider specific values for these properties but their values shouldn’t change between sessions otherwise they won’t know how to map the items when getting search results.

MAPI URLs
Every time a folder, message or an attachment is to be indexed a unique Unicode URL has to be generated and sent to the indexer. This URL will later be used to identify which object to be indexed in the MAPI PH. Store providers that want to implement notification based indexing are responsible for generating these URLs.

MAPI URLs have the following format:

 Mapi://SID/StoreDisplayName ($HashNumber)/StoreType/FolderNameA/…/FolderNameN/[EntryIDEncoded[?at=AttachIDEncoded:FileName]]

Parameters

SID
Current user’s SID

HashNumber
DWORD in hex calculated based on the store entry id and in some case the file path. This value will be stored in the registry and will be used later to identify the store in the MAPI PH. This number has to be calculated in a way that minimizes collisions between other stores. See below for the algorithm Outlook uses to calculate the hash.

StoreType
Number that identifies the type of the store that contains the object to be indexed.
Here are the possible values for pusher stores:
0 – Norma store (non public or delegate stores. This includes default stores, PSTs, etc.)
1 – Delegate store (used for delegate items cached locally)

For stores which have been crawled by the indexer, this value will always be ‘X’.

FolderNameA/…/FolderNameN
The path from the root of the IPM_SUBTREE to the folder or message. For instance, a message in the “Family” folder under “Inbox” will have Inbox/Family for this parameter.

EntryIDEncoded
MAPI Entry ID for the item encoded as a Unicode string. See below for how it gets encoded. Note that when viewed as text, this encoded entry ID will appear as random Hangul characters or boxes depending on available fonts.

AttachIDEncoded
Attachment ID encoded as a Unicode string.

FileName
Attachment file name as it appears in the message.

Here are some examples of MAPI URLs for a folder, message and attachment respectively:

 mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($be19928f)/2/Office
mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($484efb89)/0/Calendar/곯가가가걍걝걌곌겷걢곒갑겛개가검걟곔걙곾걤곂갠가
mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($484efb89)/0/Inbox/곯가가가걍걝걌곌겷걢곒갑겛개가검걟곔걙곾간곷갦가/at=겅걋각가:somefile.txt

Algorithm to calculate the store hash

 DWORD ComputeStoreHash(ULONG cbStoreEID, LPENTRYID pbStoreEID, LPCWSTR pwzFileName)
{
  DWORD   dwHash = 0;
 ULONG   cdw = 0;
    DWORD*  pdw = NULL;
 ULONG   cb  = 0;
    BYTE*   pb  = NULL;
 ULONG   i = 0;

  // Get the Store Entry ID
   // pbStoreEID is a pointer to the Entry ID
  // cbStoreEID is the size in bytes of the Entry ID
  pdw = (DWORD*)pbStoreEID;
   cdw = cbStoreEID / sizeof(DWORD);

   for (i = 0; i < cdw; i++)
    {
       dwHash = (dwHash << 5) + dwHash + *pdw++;
 }

   pb  = (BYTE *)pdw;
  cb  = cbStoreEID % sizeof(DWORD);

   for (i = 0; i < cb; i++)
 {
       dwHash = (dwHash << 5) + dwHash + *pb++;
  }

   // You may want to also include the store file name in the hash calculation
 // Get store FileName
   // pwzFileName is a NULL terminated string with the path and filename of the store
  if (pwzFileName)
    {
       while (*pwzFileName)
        {
           dwHash = (dwHash << 5) + dwHash + *pwzFileName++;
     }
   }
   // dwHash now contains the hash to be used. It should be written in hex when building the URL.
  return dwHash;
}// ComputeStoreHash

Algorithm to encode the Entry ID and the attachment ID
The goal of this algorithm is to generate a compact representation of the Entry ID or attachment ID.

 const WORD kwBaseOffset = 0xAC00;  // Hangul char range (AC00-D7AF)
LPWSTR EncodeID(ULONG cbEID, LPENTRYID rgbID)
{
 ULONG   i = 0;
  LPWSTR  pwzDst = NULL;
  LPBYTE  pbSrc = NULL;
   LPWSTR  pwzIDEncoded = NULL;

    // rgbID is the item Entry ID or the attachment ID
  // cbID is the size in bytes of rgbID

   // Allocate memory for pwzIDEncoded
 pwzIDEncoded = new WCHAR[cbEID+1];
  if (!pwzIDEncoded) return NULL;

 for (   i = 0, pbSrc = (LPBYTE)rgbID, pwzDst = pwzIDEncoded;
        i < cbEID;
       i++, pbSrc++, pwzDst++)
 {
       *pwzDst = (WCHAR) (*pbSrc + kwBaseOffset);
  }

   // Ensure NULL terminated
   *pwzDst = L'\0';

    // pwzIDEncoded now contains the entry ID encoded.
  return pwzIDEncoded;
}// EncodeID

New Notification to Notify the MAPI PH of the Process Pushing MAPI URLs to be Indexed
A new MAPI notification type has been introduced to facilitate shutdown scenarios for pusher stores. These stores will have to persist what has to be pushed since it may not be able to index everything before a shutdown occurs. When a store provider is a pusher it should send the following notification so that the MAPI PH knows which process it should watch for a given store. This way if the process is shut down or crashes the MAPI PH will immediately detect that and close (stop indexing) the store it opened.

 #define fnevIndexing      ((ULONG) 0x00010000)

/* Indexing notifications (used for FTE related communications)     */
/* Shares EXTENDED_NOTIFICATION to pass structures below,     */
/* but NOTIFICATION type will be fnevIndexing             */

// Stores that are pusher enabled (PR_SUPPORT_MASK contains STORE_PUSHER_OK)
// are required to send notifications regarding the process that is pushing.
#define INDEXING_SEARCH_OWNER     ((ULONG) 0x00000001)

typedef struct _INDEX_SEARCH_PUSHER_PROCESS
{
    DWORD       dwPID;          /* PID for process pushing */
} INDEX_SEARCH_PUSHER_PROCESS;

BLOB structure associated with each URL when indexing
When pushing URLs to be indexed pusher stores should also create a blob with some information that will be used by the MAPI PH. This blob will be associated with each URL and will be sent when the URL is pushed to the indexer.
The format of the blob is the following:

 DWORD  dwVersion
DWORD  dwFlags
ULONG  cbProfileName
WCHAR  wszProfileName   
ULONG  cbProviderItemID
WCHAR  wszProviderItemID

Note that these values have to be written to the blob in the order shown above. The provider item ID should only be sent for folders to prevent opening extra folders to get this information.

Parameters

dwVersion
This is the version of the data being sent. Currently this value is 1.

dwFlags
Reserved for future use. Currently this should be 0.

cbProfileName
Size of the profile name in bytes. This information will be useful for the MAPI PH to know which profile to use when indexing the item.

wszProfileName
Null terminated Unicode string with the profile name.

cbProviderItemID
Size of the Provider Item ID in bytes

wszProviderItemID
Null terminated Unicode string with the provider item ID that uniquely identifies the item in the store.

Named Properties
The following named properties are indexed by the MAPI PH. These properties are documented as read-only. They should not be used to create or modify items.
These GUIDs represent the name spaces of the named properties

 const GUID PSETID_Appointment   = {0x00062002, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PSETID_Task          = {0x00062003, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PSETID_Address       = {0x00062004, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PSETID_Common        = {0x00062008, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PSETID_Log           = {0x0006200A, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PS_PUBLIC_STRINGS    = {0x00020329, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PS_INTERNET_HEADERS  = {0x00020386, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};

MNID_ID properties:

 // In PSETID_Address
#define dispidWorkAddressStreet 0x8045
#define dispidWorkAddressCity 0x8046
#define dispidWorkAddressState 0x8047
#define dispidWorkAddressPostalCode 0x8048
#define dispidWorkAddressCountry 0x8049
#define dispidInstMsg 0x8062
#define dispidEmailDisplayName 0x8080
#define dispidEmailOriginalDisplayName 0x8084

// In PSETID_Task
#define dispidTaskStartDate 0x8104
#define dispidTaskDueDate 0x8105
#define dispidTaskActualEffort 0x8110
#define dispidTaskEstimatedEffort 0x8111
#define dispidTaskFRecur 0x8126

// In PSETID_Appointment
#define dispidLocation 0x8208
#define dispidApptStartWhole 0x820D
#define dispidApptEndWhole 0x820E
#define dispidApptDuration 0x8213
#define dispidRecurring 0x8223
#define dispidAllAttendeesString 0x8238
#define dispidToAttendeesString 0x823B
#define dispidCCAttendeesString 0x823C

// In PSETID_Common
#define dispidReminderSet 0x8503
#define dispidSmartNoAttach 0x8514
#define dispidCommonStart 0x8516
#define dispidCommonEnd 0x8517
#define dispidRequest 0x8530
#define dispidCompanies 0x8539
#define dispidReminderNextTime 0x8560

// In PSETID_Log (also known as Journal)
#define dispidLogType 0x8700
#define dispidLogStart 0x8706
#define dispidLogDuration 0x8707
#define dispidLogEnd 0x8708

MNID_STRING properties

 // In PS_PUBLIC_STRINGS 
"Keywords" 

// In PS_INTERNET_HEADERS
"return-path"

[4:12 - fixed some truncated prop vals]

[12:10 - 12/20/06 - fixed one-off bug per Robert's comment]

Outlook 2007 Beta Documentation - Notification Based Indexing Support

Additional resources