Why are custom properties created on Windows 2000 lost when I view the file from newer versions of Windows?


In Windows 2000, Explorer let you add properties like Summary and Author to nearly any file type. But when you view the files from a machine running Windows XP or later, those properties are lost. Where did they go?

Most file types do not have extensibility points for adding metadata. For example, every byte of a plain text files is devoted to text data; there is nowhere to put metadata like Author or Summary. In Windows 2000, the shell chose to store this extra information in NTFS alternate data streams (or more accurately, the shell chose to use the STGFMT_FILE storage format, which is implemented in terms of NTFS alternate data streams.) Storing the information in alternate data streams attaches the data to the file without affecting the file contents.

This was a clever idea, taking advantage of NTFS’s ability to attach arbitrary data to a file, but it also had a serious problem: Alternate streams are not preserved by simple and common operations like sending the file by email, copying the file to a (FAT-formatted) USB thumb drive, uploading or downloading the file from a Web site, or burning the file to a CD. Basically, once the file leaves the comfortable confines of your local hard drive, there’s a good chance that the metadata will be destroyed.

To avoid this problem, Windows XP switched to storing the metadata in the file contents itself. Doing this, however, requires support from the file format. Each file type can have registered for it a property handler which describes how to read and write properties for a file. (Windows itself comes with a few such handlers, such as for JPEG images and MP3 files, with more recent versions of Windows supporting more properties.) If no such property handler is registered, the shell will use structured storage, provided the file format is compatible with structured storage.

The data you added in Windows 2000 are still there. It’s just that newer versions of Windows don’t bother looking for them. (If you were sufficiently resourceful, you could write a program which opens the file in STGFMT_FILE mode, reads the properties, then reopens the file via the shell namespace and writes the properties back out.)

For lots of programming goodness about the shell property system, check out Ben Karas’s blog, which I have been liberally linking to.

Comments (25)
  1. Mike Caron says:

    Queue people going "But, Windows should have done both: Stored in the file if possible, but fall back to the ADS if not!" in 3… 2… 1…

    (This would be a bad idea, incidentally, since Windows 2000 would see /some/ of the data (the ones that went to the alternate data stream), but not others.)

  2. asdbsd says:

    @Mike Caron: >since Windows 2000 would see /some/ of the data (the ones that went to the alternate data stream), but not others

    And that is worse how? Option 1: 2k-XP fail; XP-2k fail. Option 2: 2k-XP success; XP-2k partial success. Yeah I guess features take time, but it would have been better this way, in theory.

    ["I took a file and added a Subject property on my Windows XP machine. I then edited the Subject on my Windows 2000 machine. When I view the Subject on my Windows 2000 machine it has the new subject, but when I view it on my Windows XP machine, it has the old subject. Mass confusion." -Raymond]
  3. roman says:

    So where does Windows XP keeps metadata for plain text files?

  4. henke37 says:

    @roman In the top secret file called "metadata4PlainFiles.dat" located in your Windoors profile.

  5. Joshua says:

    Oh the joys of backwards compatibility.

  6. Danny Moules says:

    "But, Windows should have done both: Stored in the file if possible, but fall back to the ADS if not!"

    I read that as AIDS. And in the context the sentence still made perfect sense. Oh dear.

  7. Anonymous Coward says:

    Wow, did Windows actually abandon a feature some obscure application could have been depending on?

    [The API didn't change. Just the undocumented backing store. Applications which used the property system to access properties continued to work just fine. -Raymond]
  8. blah says:

    Agree with Anonymous Coward. MS sure loves to speak from both sides of its backend regarding compatibility. Crappy Application X must continue to provide the same user experience no matter what! But let's gut A, B, and C from the official shell.

  9. Joshua Ganes says:

    File metadata sounds like such a nice feature in theory. In practice, however, it is simply not practical. Many protocols and software products are simply not designed to keep this data intact when moving files from place to place.

  10. ADS=Worst feature ever says:

    "This was a clever idea…". No it was not.

    1. What if you ran w2k and edited file metadata on a FAT partition?

    2. How did you think future os/programs would solve all problems associated with alternate streams? Right, you didn't think.

    3. You forgot to mention that ADS usage was removed from xp explorer because it opened a gigantic security hole.

  11. Brian G. says:

    I wish people who hate Windows so much would give up on it and move on to the obviously superior alternatives.  Why try to fix something so fundamentally broken?  All this time you spend commenting on blogs could instead be used convincing your superiors at work that you need to migrate OSes.

  12. Joshua says:

    [The API didn't change. Just the undocumented backing store. Applications which used the property system to access properties continued to work just fine. -Raymond]

    Gak! Document your file formats and your wire protocols. It is singularly inappropriate to provide an API for accessing something but the data store is undocumented.

    Rest assured, the backing store will be reverse engineered and you will be stuck with supporting it anyway.

    [It's already documented. Finding it is left as an exercise. (Hint: Go to the Wikipedia entry for "COM Structured Storage.") -Raymond]
  13. BC_Programmer says:

    Gak! Document your file formats and your wire protocols. It is singularly inappropriate to provide an API for accessing something but the data store is undocumented.

    Why should they document them? Why is it inappropriate?

    Or are you speaking as a freetard?

  14. Troll says:

    Property handlers were introduced in Windows Vista, not XP. You have to install WDS 4 on XP to get property handlers and still Explorer doesn't really benefit from them. Just replace "XP" with Vista and the article won't be full of inaccuracies. Also, the approach Vista/7 take is better (storing metadata within the file) but Microsoft does a very poor job of supporting others' formats. They should ship property handlers for as many formats as possible. Because no one writes property handlers for some formats which actually support storing XMP or other metadata tags, the user is left without the ability to tag any of his non-text based formats in Vista/7. Hopefully, Windows 8 will correct this abomination. It's been 5 years since Vista arrived but there are only a few property handlers for some obscure formats like FLAC. We need property handlers *AT LEAST* for AVI (store them in RIFF chunks), MP4, MKV/MKA, MOV, PNG, PDF, GIF, JPEG 2000. Out of the box, Windows currently only supports JPG, TIFF, MP3, HDP/WDP, WMA and WMV.

  15. Stefan Kanthak says:

    | Property handlers were introduced in Windows Vista, not XP.

    That's simply wrong!

    PropertyHandlers were introduced in XP: PhotoMetaDataHandler.Dll implements one for image formats (TIFF, GIF, PNG, JPEG, JFIF, ICO, RLE, BMP, WDP) using CLSID {A38B883C-1682-497E-97B0-0A3A9E801682}, and ZipFldr.Dll implements another for the "CompressedFolders" using CLSID {888DCA60-FC0A-11CF-8F0F-00C04FD7D062}

  16. Troll says:

    @Stefan Kanthak, The property system was introduced in Vista (blogs.msdn.com/…/729153.aspx). PhotoMetadataHandler.dll is a backported component of Windows Photo Gallery/Windows Imaging Component to XPSP3. You will see even its file version is 6.0.6001.22185 (vistasp1_ldr.080521-1506).

  17. Alex Grigoriev says:

    "You have to install WDS 4 on XP to get property handlers"

    And immediately uninstall that performance hog…

    Oh, and updating AVI file properties may not be feasible. Those might be located in the middle of the file. Also, the standard AVI properties don't map easily to those Windows uses.

  18. Cheong says:

    When I heard about ADS doesn't work on other filesystems, I always think that would it be a good idea that: When Windows has to copy a file that have ADS attached to other places which doesn't have the support, Windows can copy the ADS as hidden file in names like filename.stream1.ADS if the target supports long filename. Sure the file view can become obscure to some users if "view hidden file" is enabled, it could help in some case… possibly even in email (which would require email client support to make attachment's ID follows certain rule for easy discovery)

  19. Cheong says:

    When I heard about ADS doesn't work on other filesystems, I always think that would it be a good idea that: When Windows has to copy a file that have ADS attached to other places which doesn't have the support, Windows can copy the ADS as hidden file in names like filename.stream1.ADS if the target supports long filename. Sure the file view can become obscure to some users if "view hidden file" is enabled, it could help in some case… possibly even in email (which would require email client support to make attachment's ID follows certain rule for easy discovery)

  20. Stefan Kanthak says:

    | PhotoMetadataHandler.dll is a backported component of Windows Photo Gallery/Windows Imaging Component to XPSP3.

    Not quite right: Windows Imaging Component is/was available for XPSP2, see http://www.microsoft.com/…/details.aspx

  21. 10 HGR 2 says:

    The Property System APIs may have been introduced in Vista, but the underlying functionality was present in XP.  Well before the release of Vista, I remember right-clicking on a JPEG, adding a caption, then trying to figure out how it was stored.  Naturally, I did what Raymond Chen hates and wrote my own EXIF header parser, which displayed and edited the tags created by XP.

    I'm a Mac OS user these days, but credit where it's due: Preview.app will display a load of information about exposure times, DPI and suchlike, but won't show – let alone edit – any of the standard JPEG caption tags.  iPhoto is even worse, storing captions in its own files where no other application can see them.  So, Mac OS / iLife still haven't caught up with XP when it comes to JPEG captions.  They're ahead on PDF management, but I don't think that's Microsoft's fault.  Given the legal trouble you had with PDF export from Office 2007, I assume making a Windows native PDF viewer would be treated with hostility.

    On the JPEG side, it would have helped if other photo albums had made use of these tags (and the Property System APIs, when they became available), instead of kludging up proprietary caption storage.  When the Microsoft Research time machine comes back from the shop, someone needs to send copies of this blog post to every digital camera manufacturer in 2002.  It's doubly annoying when the music player world did this properly: Windows Media Player, iTunes and a plethora of other applications use the same MP3 header tags, to everyone's benefit.

  22. Troll says:

    @Stefan Kanthak, wrong again I'm afraid. WIC is available for XPSP2 as a backported downloadable component but built-in only beginning with XPSP3.

    @10 HGR 2, Yes somewhat. Microsoft called them "metadata handlers". The ability to edit metadata of some file types is there in XP such as JPEG (only EXIF metadata). Office also installs its metadata handler for formats which are not OLE compound documents (DOCX, XLSX). But that's about it. WMP may have it this ability via Advanced Tag Editor but not the shell. Other document formats require property system APIs introduced with Vista and later and it can store metadata in XMP format.

    Also XP still has the Summary tab on NTFS volumes for any file type whereas the article says it doesn't give access to that metadata. The Summary tab was removed in Vista with no consideration for backward compatibility with existing tagged files using ADS, especially when developers aren't writing ANY property handlers at all for Vista/7. Another issue is MS have made the property system very difficult to debug.

  23. Stefan Kanthak says:

    Don't be afraid: <support.microsoft.com/…/en-us> and <support.microsoft.com/…/en-us> say "WIC is also available for Windows XP SP2" resp. "WIC is available as a download for Windows XP"!

    JFTR: the version of the original WIC DLLs for Windows XP and 2003 is 6.0.6000.16385, their timestamp 2006-11-01.

    You can stop trolling now!

  24. Troll says:

    @Stefan Kanthak, huh? Trouble comprehending English? You can stop countertrolling now.

  25. John Belli says:

    This doesn't explain where the extra stuff in the VERSIONINFO resource has gone, like SpecialBuild, etc.

    [I don't believe I ever claimed that this article explained that. There are many things this article doesn't explain. -Raymond]

Comments are closed.