Outlook 2007 Beta Documentation – Notification Based Indexing Support


[This is now documented here: http://msdn.microsoft.com/en-us/library/bb821096.aspx, http://msdn.microsoft.com/en-us/library/bb821097.aspx, http://msdn.microsoft.com/en-us/library/bb821099.aspx, http://msdn.microsoft.com/en-us/library/bb821100.aspx]

This is preliminary documentation for Outlook 2007 Beta 2. It does not apply to earlier versions of Outlook.

Notification Based Indexing Support
A store provider supports Notification Based Indexing if it doesn’t require incremental or full crawls and indexes all the items through notifications. Sometimes these providers are referred to as “pusher” stores, since such a store provider will push URLs to the indexer by calling appropriate indexer APIs. These URLs are built based on algorithms documented below. Each URL corresponds to one message, folder or attachment. The indexer will pass the URLs to the MAPI Protocol Handler (MAPI PH) which will parse it open to determine which message/folder/attachment it corresponds to. MAPI PH then opens that object through MAPI and gets the properties to be indexed. The text is returned to the indexer so that it can word-break it and save the terms in the catalog. Some properties like bodies or attachments may require the use of the filter host which is a different process. The filter host will basically use IFilters to crack the documents and get the text so that the indexer can then word-break the text.

New Store Support Mask Flag
This flag is set by pusher stores.

Definition

#define STORE_PUSHER_OK ((ULONG) 0x00800000)

If the store provider sets this flag the MAPI PH won’t crawl the store and the store is responsible to push any changes through notifications to the indexer to have messages indexed.

 

Usage
This flag can be retrieved by getting the property PR_STORE_SUPPORT_MASK from the store.

New Properties for the MAPI PH

#define PR_PROVIDER_ITEMID         PROP_TAG(PT_BINARY, 0x0EA3)
#define PR_PROVIDER_PARENT_ITEMID  PROP_TAG(PT_BINARY, 0x0EA4)

These properties are used to identify an item or a folder by the store provider. They are retrieved when the store provider gets the search results from the search engine and are used to identify which items matched the query. Store providers can provide provider specific values for these properties but their values shouldn’t change between sessions otherwise they won’t know how to map the items when getting search results.

 

MAPI URLs
Every time a folder, message or an attachment is to be indexed a unique Unicode URL has to be generated and sent to the indexer. This URL will later be used to identify which object to be indexed in the MAPI PH. Store providers that want to implement notification based indexing are responsible for generating these URLs.

MAPI URLs have the following format:

Mapi://SID/StoreDisplayName ($HashNumber)/StoreType/FolderNameA/…/FolderNameN/[EntryIDEncoded[?at=AttachIDEncoded:FileName]]

 

Parameters

SID
Current user’s SID

HashNumber
DWORD in hex calculated based on the store entry id and in some case the file path. This value will be stored in the registry and will be used later to identify the store in the MAPI PH. This number has to be calculated in a way that minimizes collisions between other stores. See below for the algorithm Outlook uses to calculate the hash.

StoreType
Number that identifies the type of the store that contains the object to be indexed.
Here are the possible values for pusher stores:
0 – Norma store (non public or delegate stores. This includes default stores, PSTs, etc.)
1 – Delegate store (used for delegate items cached locally)

For stores which have been crawled by the indexer, this value will always be ‘X’.

FolderNameA/…/FolderNameN
The path from the root of the IPM_SUBTREE to the folder or message. For instance, a message in the “Family” folder under “Inbox” will have Inbox/Family for this parameter.

EntryIDEncoded
MAPI Entry ID for the item encoded as a Unicode string. See below for how it gets encoded. Note that when viewed as text, this encoded entry ID will appear as random Hangul characters or boxes depending on available fonts.

AttachIDEncoded
Attachment ID encoded as a Unicode string.

FileName
Attachment file name as it appears in the message.

Here are some examples of MAPI URLs for a folder, message and attachment respectively:

mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($be19928f)/2/Office
mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($484efb89)/0/Calendar/곯가가가걍걝걌곌겷걢곒갑겛개가검걟곔걙곾걤곂갠가
mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($484efb89)/0/Inbox/곯가가가걍걝걌곌겷걢곒갑겛개가검걟곔걙곾간곷갦가/at=겅걋각가:somefile.txt

 

Algorithm to calculate the store hash

DWORD ComputeStoreHash(ULONG cbStoreEID, LPENTRYID pbStoreEID, LPCWSTR pwzFileName)
{
	DWORD	dwHash = 0;
	ULONG	cdw = 0;
	DWORD*	pdw = NULL;
	ULONG	cb 	= 0;
	BYTE*	pb 	= NULL;
	ULONG	i = 0;

	// Get the Store Entry ID
	// pbStoreEID is a pointer to the Entry ID
	// cbStoreEID is the size in bytes of the Entry ID
	pdw = (DWORD*)pbStoreEID;
	cdw = cbStoreEID / sizeof(DWORD);

	for (i = 0; i < cdw; i++)
	{
		dwHash = (dwHash << 5) + dwHash + *pdw++;
	}

	pb	= (BYTE *)pdw;
	cb	= cbStoreEID % sizeof(DWORD);

	for (i = 0; i < cb; i++)
	{
		dwHash = (dwHash << 5) + dwHash + *pb++;
	}

	// You may want to also include the store file name in the hash calculation
	// Get store FileName
	// pwzFileName is a NULL terminated string with the path and filename of the store
	if (pwzFileName)
	{
		while (*pwzFileName)
		{
			dwHash = (dwHash << 5) + dwHash + *pwzFileName++;
		}
	}
	// dwHash now contains the hash to be used. It should be written in hex when building the URL.
	return dwHash;
}// ComputeStoreHash

 

Algorithm to encode the Entry ID and the attachment ID
The goal of this algorithm is to generate a compact representation of the Entry ID or attachment ID.

const WORD kwBaseOffset = 0xAC00;  // Hangul char range (AC00-D7AF)
LPWSTR EncodeID(ULONG cbEID, LPENTRYID rgbID)
{
	ULONG	i = 0;
	LPWSTR	pwzDst = NULL;
	LPBYTE	pbSrc = NULL;
	LPWSTR	pwzIDEncoded = NULL;

	// rgbID is the item Entry ID or the attachment ID
	// cbID is the size in bytes of rgbID

	// Allocate memory for pwzIDEncoded
	pwzIDEncoded = new WCHAR[cbEID+1];
	if (!pwzIDEncoded) return NULL;

	for (	i = 0, pbSrc = (LPBYTE)rgbID, pwzDst = pwzIDEncoded;
		i < cbEID;
		i++, pbSrc++, pwzDst++)
	{
		*pwzDst = (WCHAR) (*pbSrc + kwBaseOffset);
	}

	// Ensure NULL terminated
	*pwzDst = L'\0';

	// pwzIDEncoded now contains the entry ID encoded.
	return pwzIDEncoded;
}// EncodeID

 

New Notification to Notify the MAPI PH of the Process Pushing MAPI URLs to be Indexed
A new MAPI notification type has been introduced to facilitate shutdown scenarios for pusher stores. These stores will have to persist what has to be pushed since it may not be able to index everything before a shutdown occurs. When a store provider is a pusher it should send the following notification so that the MAPI PH knows which process it should watch for a given store. This way if the process is shut down or crashes the MAPI PH will immediately detect that and close (stop indexing) the store it opened.

#define fnevIndexing		((ULONG) 0x00010000)

/* Indexing notifications (used for FTE related communications)		*/
/* Shares EXTENDED_NOTIFICATION to pass structures below,		*/
/* but NOTIFICATION type will be fnevIndexing				*/

// Stores that are pusher enabled (PR_SUPPORT_MASK contains STORE_PUSHER_OK)
// are required to send notifications regarding the process that is pushing.
#define INDEXING_SEARCH_OWNER		((ULONG) 0x00000001)

typedef struct _INDEX_SEARCH_PUSHER_PROCESS
{
	DWORD		dwPID;			/* PID for process pushing */
} INDEX_SEARCH_PUSHER_PROCESS;

 

BLOB structure associated with each URL when indexing
When pushing URLs to be indexed pusher stores should also create a blob with some information that will be used by the MAPI PH. This blob will be associated with each URL and will be sent when the URL is pushed to the indexer.
The format of the blob is the following:

DWORD  dwVersion
DWORD  dwFlags
ULONG  cbProfileName
WCHAR  wszProfileName	
ULONG  cbProviderItemID
WCHAR  wszProviderItemID

Note that these values have to be written to the blob in the order shown above. The provider item ID should only be sent for folders to prevent opening extra folders to get this information.

 

Parameters

dwVersion
This is the version of the data being sent. Currently this value is 1.

dwFlags
Reserved for future use. Currently this should be 0.

cbProfileName
Size of the profile name in bytes. This information will be useful for the MAPI PH to know which profile to use when indexing the item.

wszProfileName
Null terminated Unicode string with the profile name.

cbProviderItemID
Size of the Provider Item ID in bytes

wszProviderItemID
Null terminated Unicode string with the provider item ID that uniquely identifies the item in the store.

Named Properties
The following named properties are indexed by the MAPI PH. These properties are documented as read-only. They should not be used to create or modify items.
These GUIDs represent the name spaces of the named properties

const GUID PSETID_Appointment   = {0x00062002, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PSETID_Task          = {0x00062003, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PSETID_Address       = {0x00062004, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PSETID_Common        = {0x00062008, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PSETID_Log           = {0x0006200A, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PS_PUBLIC_STRINGS    = {0x00020329, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
const GUID PS_INTERNET_HEADERS  = {0x00020386, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};

MNID_ID properties:

// In PSETID_Address
#define dispidWorkAddressStreet 0x8045
#define dispidWorkAddressCity 0x8046
#define dispidWorkAddressState 0x8047
#define dispidWorkAddressPostalCode 0x8048
#define dispidWorkAddressCountry 0x8049
#define dispidInstMsg 0x8062
#define dispidEmailDisplayName 0x8080
#define dispidEmailOriginalDisplayName 0x8084

// In PSETID_Task
#define dispidTaskStartDate 0x8104
#define dispidTaskDueDate 0x8105
#define dispidTaskActualEffort 0x8110
#define dispidTaskEstimatedEffort 0x8111
#define dispidTaskFRecur 0x8126

// In PSETID_Appointment
#define dispidLocation 0x8208
#define dispidApptStartWhole 0x820D
#define dispidApptEndWhole 0x820E
#define dispidApptDuration 0x8213
#define dispidRecurring 0x8223
#define dispidAllAttendeesString 0x8238
#define dispidToAttendeesString 0x823B
#define dispidCCAttendeesString 0x823C

// In PSETID_Common
#define dispidReminderSet 0x8503
#define dispidSmartNoAttach 0x8514
#define dispidCommonStart 0x8516
#define dispidCommonEnd 0x8517
#define dispidRequest 0x8530
#define dispidCompanies 0x8539
#define dispidReminderNextTime 0x8560

// In PSETID_Log (also known as Journal)
#define dispidLogType 0x8700
#define dispidLogStart 0x8706
#define dispidLogDuration 0x8707
#define dispidLogEnd 0x8708

MNID_STRING properties

// In PS_PUBLIC_STRINGS 
"Keywords" 

// In PS_INTERNET_HEADERS
"return-path"

[4:12 – fixed some truncated prop vals]

[12:10 – 12/20/06 – fixed one-off bug per Robert’s comment]

Comments (12)

  1. wzhao2000 says:

    Hi, Steve,

    I have some questions about fnevIndexing and PR_PROVIDER_ITEMID. Could you shed some light on them? Thanks!

    For the new notification type fnevIndexing. I remember there is an MS KB saying MAPI notification may be dropped under heavy load, is that the same case for this new notification?

    We need to store a reference of an item to an external place to open it later (a.k.a., across MAPI sessions). We are using long term PR_ENTRYID. But the issue is that PR_ENTRYID could change if the item is moved. PR_PROVIDER_ITEMID mentioned here sounds perfect for us. Is there any way to use PR_PROVIDER_ITEMID to open the item, or map to PR_ENTRYID without searching the store?

    Also, is PR_PROVIDER_ITEMID set by Outlook or Exchange? I’m using Outlook 2007 against Exchange 2003 and seems I can see it. Not sure if it’s available in earlier Outlook or Exchange versions?

    Thanks

    George

  2. Stephen Griffin says:

    The concern about dropped notifications relates mainly to Exchange servers under load. In a cached mode profile the notifications come from the local PST provider, so this shouldn’t be a problem.

    PR_PROVIDER_ITEMID is new to Outlook 2007. The PST provider will expose it. I don’t think Exchange will have any concept of this property. I don’t know of any way to map PR_PROVIDER_ITEMID to PR_ENTRYID. Looks like you’ll need to search.

  3. wzhao2000 says:

    Steve, Thanks for the info.

    Another question, the hashing routine mentioned above, does it have anything to do with the Windows Desktop Search (WDS) mailbox hash? For example, in WDS, the path to identify a mailbox could be: (the d3f5 parts seems to be an WORD instead of DWORD though)

    mapi://LocalHost/default/Mailbox – Wang, George ($d3f5)/

  4. Stephen Griffin says:

    Most likely they’re related – I’m not "up" on WDS as of yet though to positively confirm.

  5. Robert says:

    Hi, Stephen,

    your EncodeID function above has a memory bug.

    After you finished the for loop pwzDst will point *behind* your allocated area. Adding the terminator will overwrite memory not belonging to the array.

    The solution is easy: Just allocate one more item on the array.

    Nonetheless, your site is quite informational.

  6. (Now that Outlook 2007 is available I’m reposting some of the articles from the Outlook 2007 Beta Documentation

  7. (Now that Outlook 2007 is available I’m reposting some of the articles from the Outlook 2007 Beta Documentation

  8. Stein Gran says:

    Hi Stephen,

    I’m creating a solution where I have to generate MAPI URLs to elements in MS Exchange, and I’m thinking of using the methods here to generate the MAPI URLs. But regarding your code here to generate the store hash, what is the filename when the store is Exchange?

    Thanks in advance.

  9. Stephen Griffin says:

    Note that the hash algorithm allows for a NULL file name. Give that a try and let me know how it works out.

  10. Wayne Nelson says:

    Is there a way for my application to listen for the Notification Based Indexing events?

    The app I am writing wants to know when emails have been received, deleted, moved, copied etc.

    Those notification events tell me what I need to know.

  11. Stephen Griffin says:

    You don’t want indexing events. You just want to log on to the store and register for regular MAPI notifications. See the MAPI docs, or check out the MFCMAPI source – http://codeplex.com/mfcmapi.