Roadmap for Outlook Personal Folders (.pst) Documentation


By Paul Lorimer, Group Manager, Microsoft Office Interoperability

[UPDATE: 05/24/2010, Two open source projects to facilitate interoperability with Outlook .pst data files]
[UPDATE: 02/20/2010,
New Office Documentation Now Publicly Available

Data portability has become an increasing need for our customers and partners as more information is stored and shared in digital formats. One scenario that has come up recently is how to further improve platform-independent access to email, calendar, contacts, and other data generated by Microsoft Outlook.

On desktops, this data is stored in Outlook Personal Folders, in a format called a .pst file. Developers can already access the data stored in the .pst file, using Messaging API (MAPI) and the Outlook Object Model—a rich set of connections to all of the data stored by Outlook and Exchange Server—but only if Outlook is installed on the desktop.

In order to facilitate interoperability and enable customers and vendors to access the data in .pst files on a variety of platforms, we will be releasing documentation for the .pst file format. This will allow developers to read, create, and interoperate with the data in .pst files in server and client scenarios using the programming language and platform of their choice. The technical documentation will detail how the data is stored, along with guidance for accessing that data from other software applications. It also will highlight the structure of the .pst file, provide details like how to navigate the folder hierarchy, and explain how to access the individual data objects and properties.

This documentation is still in its early stages and work is ongoing. We are engaging directly with industry experts and interested customers to gather feedback on the quality of the technical documentation to ensure that it is clear and useful. When it is complete, it will be released under our Open Specification Promise, which will allow anyone to implement the .pst file format on any platform and in any tool, without concerns about patents, and without the need to contact Microsoft in any way.

Designing our high volume products to enable such data portability is a key commitment under our Interoperability Principles, which we announced in early 2008. We support this commitment through our product features, documented formats, and implementation of standards. The move to open up the portability of data in .pst files is another step in putting these principles in action.

Over the past year, Microsoft Office has taken several steps toward increased openness and document interoperability. We’re proud of the work we’ve done around document interoperability, offering customers a choice of file formats and embracing a comprehensive approach that includes transparency into our engineering methods, collaboration with industry stakeholders, and shared stewardship of industry standards.

We’re excited about the possibilities created for our customers and partners by this kind of effort, and we look forward to continued collaboration with the industry in our pursuit of improved interoperability with Microsoft Office. Stay tuned.

Paul Lorimer, Group Manager, Microsoft Office Interoperability.

Related Items:

Comments (44)

  1. OneNote format? says:

    Please make OneNote 2007 format open as well.

  2. Leith Bade says:

    Are there any plans to also document the .MSG file format? This is another useful format to have for importing into document management systems.

  3. Alex Ott says:

    Hi Paul

    Is it possible to get early access to .pst format documentation? I’m working with many of MS formats, and we’re planning to implement support for them all in our products.

    P.S. My company has signed agreement with MS about support for different office formats

  4. jccim says:

    @Leitg Bade: the .MSG File Format Specification is actually already available at http://msdn.microsoft.com/en-us/library/cc463912.aspx  

  5. maricela morales says:

    how can I help. I can’t wait to having alternatives in a central manner. My productivity could greatly improve with alternatives to traditional email. a next generation integrated email…maybe called eknowledge is desperatedly needed. EMAIL HAS TO GET SMART. IT IS HOLDING US BACK…I am tired of all the notifiations from social network sites requiring horrendous time for organization and filtering…AY CARAMBA! Help@

  6. Jeff McKay says:

    I’m an "interested customer" (and Microsoft Partner) – can I sign up to get an early look at the documentation?  Is there a way for me to be notified when it is available?

  7. Jonathan F. says:

    Please share this information with the Entourage group! To put it mildly, I was flabbergasted to discover that Entourage 2008 could NOT import contents directly from an Outlook 2003 PST file. I know that people get all hot and bothered discussing the merits of Mac OS vs. Windows, but for me, they’re both just tools to run software applications. When Microsoft designs apps for the respective OS’s, I expect that those apps should work nicely together. Instead, upon opening up Entourage, I felt beholden to some juvenile argument that was held between the Windows Office team and the Mac Office team. The above mentioned trajectory sounds much better, so please make sure it happens!

  8. The PST file is one of the biggest problems with Outlook. Why would anyone else want to implement this on other platforms?

  9. Darren Baker says:

    Brilliant! Any timeframe on this?

  10. @tj says:

    What about the .OST file format too?

    Is that documented somewhere as yet or will it be covered in the .PST format documentation set?

  11. Helen says:

    Interesting.

    Any comments on whether the format is documented in the context of MAPI calls (requiring the implementation of underlying COM as well as MAPI APIs for use), or simply as a byte stream oriented structure?  The latter would be preferable from a multi-platform development standpoint.

  12. robertb says:

    Why not rather work on a format which removes the limitations inherent in the .pst format ( 2gb file size, 65535 items ) such as storing in a local MS SQL DB, or heaven forbid an open format such as sqlite?

  13. mike says:

    This is good news and it will open the way for people to create innovative software components and applications which access PST files.

    Mike

  14. Fredre says:

    "This documentation is still in its early stages and work is ongoing."

    Can you give a better time line for release like mid 2010 or 2011 or never ?

    That would help

  15. decatec says:

    Interop is good, but please first:

    -give us .NET Developers and MS partners easy .NET APIs rather than specs

    -put formats in XML (e.g MSG, EML) like you did for the Windows Event Log since Vista

    – Outlook (PST) storage could be extended/replaced by SQL Server (Filestream) or Sharepoint for better centralization/backup/retention/ .. you name it

  16. Dave says:

    Please release as soon as possible, ie beta documentation. Incomplete documentation is better than no documentation.

  17. Jaspreet says:

    Any timeline for the release of this documentation ?

    Is there any way to join the beta testing  ?

  18. Steve says:

    What is the timeframe for releasing said documentation?

  19. jccim says:

    @Helen:

    The specification will document the on-disk structure of .pst files. Developers will be able to develop multi-platform solutions that do not require MAPI or other runtime components provided by Outlook.

    @tj:

    The .pst and .ost files are created using the same underlying database engine, but there are some important differences in the file formats.

    It is important to note that an .ost file is designed to be a local cache of data stored on Exchange servers, and the authoritative user data can be accessed from the Exchange servers through the documented Exchange protocols.

    We will not be covering the .ost file format as we meet with industry experts, but people are welcome to provide us with feedback related to documenting the .ost file format.

    @Darren Baker:

    Thanks!  We plan to release the .pst specification to the general public in the first half of 2010.

  20. That’s a good idea to open the specification to the public, so we can develop 3rd party tools more easily.

    We are looking forward to the release.

    Best regards

    Wolfgang Zerzawy

  21. David says:

    PST file format API already exist. Check PST .NET:

    http://www.independentsoft.de

    Regards,

    David

  22. Volker Dilg-Gruschinski says:

    As the underlying database engine is both the same for pst and ost files, will that documentation allow 3rd party tools to access and read ost files as well ?

    It’s not the question how the usage is designed.

    The general question is:

    Would this documentation allow a hacker to read my confidential data from an ost file ?

    Volker

  23. Is there any plans to also release a specification for winmail.dat stuff?

  24. Hai says:

    Does this also benefit MAPI service providers that use the wrapped PST store?

    Is this solely for the purpose of letting non-Outlook process to read/write information in a PST file?

    How should the potential new applications handle scenarios of concurrent access to the same PST file from both Outlook process and the 3rd party application process? Or this is mainly for off-line manipulation to the PST file when Outlook is not running?

  25. jccim says:

    @OneNote question:

    A draft of the OneNote file format specification has already been posted: http://msdn.microsoft.com/en-us/library/dd924743.aspx. It covers OneNote 2010.

  26. jccim says:

    Thanks everybody for your participation and interest. Here are a few more answers/comments from the team :)

    @Robertb

    The 2GB file size limit has been removed with the release of Outlook 2003. Please refer to this article http://support.microsoft.com/kb/830336 for more information. We are committed to investing in our storage technology to meet the increasing demands of our users.

    @Hai

    The .pst file format specification will cover wrapped .pst stores as wrapped pst stores are just a type of .pst files. It is important to note that MAPI and the Outlook Object Model provide a runtime service to serialize requests from multiple processes that are trying the access the same .pst file. If you want to implement a solution that directly accesses a .pst file without going through the runtime service provided by MAPI, you will need to make sure no other processes are using the same file.

    @Volker Dilg-Gruschinski

    In theory software developers can use information provided in the .pst file format specification to access ost files as well. Regarding the security concern, we don’t recommend users to rely on the rudimentary encoding in our file formats to protect confidential data. We recommend users to take advantage of advanced encryption features provided by the operation system or 3rd party tools to secure confidential data.

    @David

    We welcome developers to take advantage of the .pst file format specification to create tools that interoperate with Outlook. As of today we have not released a draft or final version of the .pst file format specification to the public.

    @David

    winmail.dat contains data encoded in TNEF format, which is documented in http://msdn.microsoft.com/en-us/library/cc425498.aspx

  27. walo says:

    you have to open all formats or adopt odf.

  28. Roger says:

    The format was hacked long ago. It’s about time MS recognized it.

  29. jccim says:

    The best way to participate is to send an email message to pstinfo@microsoft.com .

    Daniel Ko from the Outlook development team has just posted this information:

    http://blogs.msdn.com/outlook/archive/2009/10/29/outlook-pst-file-format-and-interoperability.aspx

  30. GS says:

      I am eager to see the documentation and how it can be used better than the MAPI?

  31. Pim van der Eijk says:

    I fear the documentation for this format will 5000 pages and just show that this is yet another very complex, binary format. A format so complex that no-one will be able to implement it correctly and robustly, except someone who was been working in the Office Division of Microsoft for the past ten years, and knows two dozen people who have been there too. Or am I too pessimistic?

    It would be better for Office to move to some clean new format, like an relational data store format or a (ZIP’ed)  XML format so that people can use SQL or XPath to manipulate it.  And preferably standardize such a format in an open standards body. Is anything like that on the agenda?  

    I regret the day I started to use Outlook, as my email of the past 15 years (> 10 GB) is locked in a proprietary, undocumented file format. It’s amazing to me that PST has never been in the spotlight as much other Office file formats. Those formats, unlike PST, have been supported by other products (like OpenOffice) for years, so opening up PST should IMO have been done years before and with higher priority that developing Office Open XML.

    But in any case, document PST is a good step!

    Pim

  32. Thank you for providing the pst format.  We have instances where we must extract the contents of pst files for litigation purposes.  There have been some pst files that were damaged and the pst file fixing software provided with MSOutlook could not fix them.  In this case, we need to be able to extract as much information as possible from the files.  The format will allow us to interactively try to navigate to accessible parts of the file.  (In our case, the files being worked with are off-line and not being referenced by outlook, so access conflicts should not be a problem.)  Thanks again.

  33. It will be interesting to see the final documentation.

  34. Russell Mangel says:

    To: Pim van der Eijk

    I would read all 5000 pages, nine times if necessary.

    Consider the following:

    The PST file format has been with us since the Microsoft Exchange Client was released with Exchange 4.0. since at least 1996. Every version of Outlook uses the PST file format.

    I would be willing to bet there are far more PST files backing Outlook installations, than Outlook to Exchange Server (which uses Exchange Database). This makes the PST file *wildly* popular.

    I have been waiting for this documentation for many years.

    Kudos to Microsoft.

    Russell Mangel

    Las Vegas, NV

  35. Angelo says:

    This is a great strategic move.  I was considering how I need to long term archive email and the fact that pst is Not publicly documented, and that there are really no 3rd party tools, really made me question it as an option.

    I may end up stayingwith pst if there was signifcant 3rd party development and integration.

    Can’t wait.  

  36. Eric says:

    Hu, Angelo,

    We use a 3rd party tool for few months now, that is already able to backup and syncronize PST files incrementally, even if Outlook is open. EdgeSafe PST2PST Backup. http://www.datamills.com. Very fast and also able to retain email history.

  37. Willem Grooters says:

    I’m interested in the format for another reason: I’m using different mail-clients on different systems – of which Windows Outlook is just one. The problem I face is taht I need to exchange information from one environnment to the other – and each system has it’s more or less own format for storing mail, contacts and so on. This makes any of these products a closed to other systems. For most of these systems, the format is described so it’s easy to write a program to store the information centrally – in Outlook’s .PST files, for instance. But I do NOT want to be restricted to .NET; moving .PST files to other systems for proicessing ought to be an option.

  38. It will be interesting to see the final documentation.

  39. Some Guy says:

    I thought a roadmap usually included some form of timeframe?

  40. John M says:

    Hi,

    I have also did reverse engineering on PST file and ready with a recovery utility for free.

    If anyone want help email me jm2251@gmail.com

  41. I am a Microsoft Outlook user. I use outlook 2007, I usally get an error in outlook. I read your post, it was really nice and I got lot of knowledge. Do you suggest me to use Outlook 2010, Is that better then 2007 and does it has inbuild pst repair tool. Please suggest.

  42. "I thought a roadmap usually included some form of time fram"

    Not necessarily

  43. John Smith says:

    Perfect Data Solutions is gives you wonderful solution for extract OST file to PST. PDS Outlook OST PST converter program can convert OST file to PST file instantly. Use this link and make OST to PST file exactly.

    http://www.transferosttopst.net

    http://www.ost-to-pst.org/