The Nickname Cache

[This is now documented here: https://msdn.microsoft.com/en-us/library/ff625288.aspx ]

One of the long standing requests for Outlook development is for documentation around the format for the nickname cache. A couple months ago, development asked if I’d be interested in hosting a preview of documentation for the NK2 file. Of course, I said yes. I didn’t write this doc, but I did assist in the tech review. Down the road, you can expect to see a version of this documentation show up in the MSDN. I’ll link back to it here when it does.

This documentation applies only to Outlook 2003 and 2007 (despite mostly saying only 2007 throughout the article). There are differences in the format used by Outlook 2010 and we do expect to document them, most likely when this information is incorporated into the MSDN. For now, if you use this to mess with Outlook 2010’s nickname cache, you’re on your own.

Finally, I’ve uploaded a PDF of this document here, which includes the parsing for a sample NK2 file.

As always, let me know if you find any problems with this documentation.

[Edited 5/25/2010 to incorporate minor updates based on user feedback]

Enjoy!

Steve

Outlook 2003/2007 NK2 File Format and Developer Guidelines

This post explains how Microsoft® Office Outlook® 2007 interacts with the nickname cache file, also known as the “.nk2 file.” The .nk2 file is where Outlook 2007 persists the autocomplete list, which is the list of names that displays in the To, Cc, and Bcc edit boxes while composing an e-mail. This post also discusses the binary format of the file and the recommended ways for interacting with the .nk2 file.

This blog post should contain sufficient information to support reading and modifying the .nk2 file.

You can use any programming language to write your application because there are no dependencies on the Outlook object model or MAPI APIs.

Outlook 2007 Interaction with the .nk2 file

When a user logs on to Outlook 2007, Outlook reads the user’s MAPI Profile. Later, when the autocomplete list is shown, the .nk2 file is loaded. The .nk2 file has the same name as the profile that was used to log on, typically outlook. nk2, in the default scenario.

This file can be found here:

%APPDATA%MicrosoftOutlook

Outlook 2007 interacts with the .nk2 file in two ways:

1. Loading the .nk2 file.

2. And, later, saving the .nk2 file.

Loading the .nk2 file

Outlook 2007 loads the .nk2 file when any item with addressing functionality gets initialized. For example, e-mail addresses are used in a new mail, a mail reply, a contact item, a meeting request, etc. To load, Outlook 2007 reads all of the contents of the file as a binary stream into a structure in memory.

For autocomplete operations, Outlook interacts exclusively with this in-memory structure for the duration of the outlook.exe process lifetime. Outlook 2007 does not interact with the .nk2 file in any additional ways until it saves the in-memory structure back to disk on shutdown.

Saving the.nk2 file

Outlook 2007 saves the .nk2 file on shutdown if the autocomplete list has changed in any way.

Here are the ways that the autocomplete list gets changed:

· A new nickname entry is added through resolving a name, picking a recipient from the address book dialog, or sending mail to a recipient that was not already in the list.

· An entry is modified by sending mail to an existing recipient in the list.

· An entry is removed by the user through the UI.

· Other minor scenarios not relevant to this blog post

Outlook 2007 does not save the .nk2 file on shutdown if the autocomplete list has not changed. The save involves writing the in-memory structure to the .nk2 file as a binary stream.

If the size of the .nk2 file contents shrink during an Outlook session (for example, the user deletes some entries), Outlook saves the new contents to the file, but the file itself will not shrink in size.

Recommendations

· Never partially modify the .nk2 file. The supported interaction is to 1) read the entire file into memory, 2) modify the memory structure, and 3) write out the entire file when the modifications are finished.

· We recommend locking the file from modification by other processes while you’re reading it and writing it using standard Windows file locking APIs (e.g. LockFile in C/C++ and FileStream.Lock in C#).

· Don’t interact with the .nk2 file while Outlook is running. If Outlook is running while you modify the file, Outlook will likely overwrite your changes when it shuts down.

· Do not write properties of type PT_MV_UNICODE and PR_MV_STRING8 into an .nk2 file to be consumed by Outlook 2003. These properties are only understood by Outlook 2007.

· Do not write properties of types that are not mentioned in this document.

NK2 File Format

In addition to knowing how Outlook interacts with the .nk2 file, you must also understand the binary file format of the .nk2 file.

The .nk2 file is a set of recipient property rows that are saved as a binary stream along with some bookkeeping metadata that is used only by Outlook 2007 and Outlook 2003.

The metadata is relevant to Outlook’s interactions with the .nk2 file sothird parties must preserve what is in each metadata block when saving a modified .nk2 file. In other words, third parties should modify only the row-set portion of the binary format and preserve what was already in the metadata blocks of the file.

When creating a new .nk2 file from scratch, use the metadata values from the binary example to populate the metadata in the new file.

File Visualization

The high-level layout of the file looks like this:

Metadata (12 bytes)

Number of rows n (4 bytes)

Number of properties p in row one (4 bytes)

Property 1’s property tag (4 bytes)

Property 1’s reserved data (4 bytes)

Property 1’s value union (8 bytes)

Property 1’s value data (0 or variable bytes)

… (property 2 through property P-1)

Property p’s property tag (4 bytes)

Property p’s reserved data (4 bytes)

Property p’s value union (8 bytes)

Property p’s value data (0 or variable bytes)

Number of properties q in row two (4 bytes)

… (row two’s properties)

… (row 3 through row n-1)

Number of properties r in row n (4 bytes)

… (row n’s properties)

Metadata (12 bytes)

High-level Layout

Broadly speaking, this is the layout of the .nk2 file:

Value Data

Number of Bytes

Metadata

12

Row-set

Variable

Metadata

12

Row-set Layout

The row-set layout is as follows:

Value Data

Number of Bytes

Number of rows

4

Rows

Variable

The number of rows identifies how many rows will come in the next part of the binary stream sequence.

Row Layout

Each row is of the following format:

Value Data

Number of Bytes

Number of properties

4

Properties

Variable

The number of properties identifies how many properties will come in the next part of the binary stream sequence.

Property Layout

Each property is of the following format:

Value Data

Number of Bytes

Property Tag

4

Reserved Data

4

Property Value Union

8

Value Data

0 or variable (depending on the prop tag)

Interpreting the Property Value

The Property Value Union and the Value Data are to be interpreted based on the property tag in the first 4 bytes of the property block. This property tag is in the same format as a MAPI property tag. Bits 0 through 15 of the property tag are the property’s type. Bits 16 through 31 are the property’s identifier. The property type determines how the rest of the property should be read.

Static Values

Some properties have no Value Data and only have data in the union. The following property types (which come from the Property Tag) should interpret the 8-byte Property Union data as follows:

Prop Type

Property Union Interpretation

PT_I2

short int

PT_LONG

long

PT_R4

float

PT_DOUBLE

double

PT_BOOLEAN

short int

PT_SYSTIME

FILETIME

PT_I8

LARGE_INTEGER

Dynamic Values

Other properties have data in a Value Data block after the first 16 bytes that contain the Property Tag, the Reserved Data, and the Property Value Union. Unlike static values, the data stored in the 8 byte Property Value union is irrelevant on reading. When writing, make sure to fill these 8 bytes with something, but the content of the 8 bytes doesn’t matter. In dynamic values, the property tag’s type determines how to interpret the Value Data.

PT_STRING8

Value Data

Number of Bytes

Number of bytes n

4

Bytes to be interpreted as an ANSI string (includes NULL terminator)

n

PT_UNICODE

Value Data

Number of Bytes

Number of bytes n

4

Bytes to be interpreted as a UNICODE string (includes NULL terminator)

n

PT_CLSID

Value Data

Number of Bytes

Bytes to be interpreted as a GUID

16

PT_BINARY

Value Data

Number of Bytes

Number of bytes n

4

Bytes to be interpreted as a byte array

n

PT_MV_BINARY

Value Data

Number of Bytes

Number of binary arrays X

4

A run of bytes containing X binary arrays. Each array should be interpreted like the PT_BINARY byte run described above.

Variable

PT_MV_STRING8 (Outlook 2007)

Value Data

Number of Bytes

Number of ANSI strings X

4

A run of bytes containing X ANSI strings. Each string should be interpreted like the PT_STRING8 byte run described above.

Variable

PT_MV_UNICODE (Outlook 2007)

Value Data

Number of Bytes

Number of UNICODE strings X

4

A run of bytes containing X UNICODE strings. Each string should be interpreted like the PT_UNICODE byte run described above.

Variable

Significant properties

As mentioned above, the binary blocks that represent properties have property tags that correspond to properties on address book recipients. For properties that aren’t listed here, you can look up the property description at https://msdn.microsoft.com/en-us/library/cc433490(EXCHG.80).aspx.

The properties below are the minimum set of properties necessary for a row to be valid. Therefore, new rows added to the .nk2 file must be populated with the properties below.

Property Name

Property Tag

Description (see MSDN for more details)

PR_NICK_NAME_W (not transmitted on recipients, specific to .nk2 file only)

0x6001001F

This property must be first in each recipient row. Functionally serves as a key identifier for the recipient row.

PR_ENTRYID

0x0FFF0102

The address book entry identifier for the recipient.

PR_DISPLAY_NAME_W

0x3001001F

The recipient’s display name.

PR_EMAIL_ADDRESS_W

0x3003001F

The recipient’s e-mail address (e.g. johndoe@contoso.com or /o=Contoso/OU=Foo/cn=Recipients/cn=johndoe).

PR_ADDRTYPE_W

0x3002001F

The recipient’s address type (e.g. SMTP or EX).

PR_SEARCH_KEY

0x300B0102

The recipient’s MAPI search key.

PR_SMTP_ADDRESS_W

0x39FE001F

The recipient’s SMTP address.

PR_OBJECT_TYPE 0x0FFE0003 Represents the type of this recipient. Values can either be MAPI_MAILUSER or MAPI_DISTLIST.
PR_DISPLAY_TYPE 0x39000003 Similar to PR_OBJECT_TYPE, but it’s used by Outlook’s UI to determine how to display the recipient (for example, bolding a distribution list). Common values for this are DT_DISTLIST and DT_MAILUSER.
PR_NEW_NICK_NAME (not transmitted on recipients, specific to .nk2 file only) 0x6002000B Specifies whether this row was just added to the nickname cache or not. If you are creating a new row, this should be set to true.

PR_DROPDOWN_DISPLAY_NAME_W (not transmitted on recipients, specific to .nk2 file only)

0x6003001F

The display string that shows up in the autocomplete list.

PR_NICK_NAME_WEIGHT (not transmitted on recipients, specific to .nk2 file only)

0x60040003

The weight of this autocomplete entry. The weight is used to determine in what order autocomplete entries show up when matching the autocomplete list. Entries with higher weight will show before entries with lower weight. The entire autocomplete list is sorted by this property. The weight periodically decreases over time and increases when the user sends an e-mail to this recipient. See the description below for more information about this property.

PR_NICK_NAME_WEIGHT

The set of rows in the .nk2 file is sorted in descending order by the PR_NICK_NAME_WEIGHT property and the .nk2 file should always preserve this sorted characteristic. Therefore, any changes to a row’s weight should also ensure that the row’s position retains the sorted order of the entire set of rows. Any additions to the row-set should be inserted to the proper position to maintain the sorted order.

The minimum value of this weight is 0x1 and the maximum value is LONG_MAX. Any other values for the weight are considered invalid.

When Outlook 2007 sends a mail to or resolves a recipient, it will increase that recipient’s weight by 0x2000.