Why are hidden files with a leading tilde treated as super-hidden?


Open a command prompt and perform the following operations:

C:> cd /d %USERPROFILE%\Desktop
C:\Users\Bob\Desktop> echo 12345 > ~test.txt
C:\Users\Bob\Desktop> attrib +h ~test.txt

This creates a hidden file called ~test.txt on the desktop. Configure Explorer to show hidden files. Observe that the ~test.txt file does not appear.

But wait, there's more, if you're running Windows 7 (but not Windows 8 or higher): Configure Explorer to show both hidden files and protected system files. The ~test.txt file will now appear, and it will be dimmed because it is hidden. Use Ctrl+C and Ctrl+V to create a copy of the file. Observe that the copy has both the hidden and system attributes, even though the original did not have the system attribute.

A customer discovered this behavior and wanted to know whether it was a bug or a feature (or a buggy feature).

There are multiple things going on here, so let's take them separately.

First, why doesn't ~test.txt appear on the desktop even though Explorer is configured to show hidden files?

This behavior dates back to Windows Vista. If there is a hidden file whose name begins with a tilde, then Explorer treats it as if the system and hidden attributes are both set, causing the file to be treated as super-hidden. That's why you have to disable "Hide protected operating system files" in order to see them.

Why does this rule exist?

In practice, hidden files that begin with a tilde are temporary files, usually to represent auto-saved contents, or as part of a write-rename-delete save operation. These files are not intended to be user-manipulated, so Explorer treats them as super-hidden so that the user won't be tempted to rename or delete them and mess up the operation of the program that created them.

Second, why does copying these artificially-super-hidden files cause the copy to become super-hidden for real?

This is a case where Explorer faked itself out.

The code that creates item IDs for files reads the file attributes and records them for future reference. It is this code that checks for the leading tilde and if found internally sets the FILE_ATTRIBUTE_SYSTEM flag on the item it created. This is what causes hidden files beginning with a tilde to be treated as super-hidden.

The problem is that this code ends up doing too good a job of fooling the rest of the shell. There is no flag anywhere that says, "Psst, by the way, the system attribute you see on this item? Yeah, it's a total fabrication. The real file doesn't have that attribute."

When it comes time to copy the file, the shell looks at the item ID and says, "Well, it says here that the original has the system attribute, so I'll set the system attribute on the copy." The shell copy engine doesn't know that the attribute is a lie.

This problem was fixed in Windows 8 as a side-effect of a re-write of the way the shell copy engine copies files. The shell now uses the Copy­File2 function to copy files, relying on the kernel function to do the heavy lifting, and using the callback function to monitor progress and possibly cancel the operation. The kernel function doesn't know about these mysterious shell item IDs. All it knows how to do is copy files, and it obtains the attributes directly from the source file, which as we recall is marked hidden but not system.

Bonus chatter: The "heavy lifting" alluded to above can be quite substantial. In addition to copying the file contents, it also copies the alternate data streams and file attributes, and can also take advantage of things like copy offload.

Comments (15)
  1. skSdnW says:

    So Vista/7 has a custom copy engine that just uses CreateFile+ReadFile+WriteFile? Or is it using CopyFileEx and then additionally applies the attributes from the pidl?

  2. Lars says:

    I am not a Windows user anymore (mostly), but I remember from back when I was that there would be lots of files with leading tildes sprinkled around in temp directories. Stale temp files, not ones in use. It seems counterproductive to prevent the user from deleting files that may well be stale and unneeded.

    1. cheong00 says:

      These files were supposed to be deleted with “disk cleanup” tool.

  3. DWalker07 says:

    Right, those often-stale tilde temp files, in temp directories, can be years old.

    We need an attribute that says “delete this file on reboot”. Not for use by “Microsoft Office auto-recover in case of crash”, but for use by programs that write scratch info to files during processing which never needs to be recovered.

    MS Office auto-recover files can PROBABLY be deleted if they are a year old….

    1. So the system should trawl all directories on all volumes on startup, looking for files with that bit set?

      ISTM that many such cases could be fixed by judicious use of FILE_FLAG_DELETE_ON_CLOSE.

      1. “use of FILE_FLAG_DELETE_ON_CLOSE.”

        For which, of course, the first Google result is an Old New Thing post .

      2. Joshua says:

        Requires admin to use. :( Should be able to do it if only owns the file but Windows has a hard time working that way.

        1. Since when does D_O_C need admin rights? It needs FILE_SHARE_DELETE on all concurrently open handles, whether created before or after. (Why is that anyway? The file is deleted as soon as there are no open handles to it anymore, i.e. as soon as all processes have declared not to care about the file anymore. As long as anyone cares, the file stays around. So why share-delete mode?)

      3. Dave Bacher says:

        Imagine an API — CreateVolatileFile.

        I mount a VHD at %USERPROFILE%\AppData\Volatile

        At login, I quick format the VHD. Doing it that way lets you swap it to a RAM drive or dedicated SSD device, if you want to, for speed. Quick formatting a VHD should take on the order of milliseconds. You could just use a directory and drop the tree, that’s reasonably fast too, it just doesn’t give you as nice of optimization options.

    2. GL says:

      One can run cleanmgr on a scheduled basis to remove old files in the temp folder.

      What is interesting is that Explorer has got rid of this lie completely. If I create “~1” and set it as Hidden, its Properties window has the “Hidden” checkbox enabled. If I remove Hidden, and add System and Hidden at the same time, its Properties window has the “Hidden” checkbox disabled — indicating it indeed has the System attribute set. Of course Explorer still displays a hidden “~1” as a “System file” type file. Double clicking it will bring up the “You may not want to open it” dialog. There is one more interesting bug here: the “You may not want to open it” dialog says the file is “.sys” file while it is not.

  4. Patrick Hughes says:

    Interesting you mention alternate data streams. Those have been treated like red headed step children for a long time and I’m finally getting ready to abandon their use.

    1. Don’t abandon alternate data streams yet. Most developers haven’t even been told they exist, lol. Let me give you an example of a file manager that has first-class support for streams.

      https://www.kickstarter.com/projects/129274931/asynchronicity-shell

      (this is 1/2 answer, 1/2 shameless but highly relevant plug. I wanted to show and discuss with you privately Mr. Chen but you’re impossible to contact!)

    2. ErikF says:

      Alternate data streams really shouldn’t be relied on, considering that as soon as you copy a file to a USB stick or network location practically all the metadata tend to disappear. On a local system I can think of useful reasons for Windows (and maybe some system utilities) to use ADS, but I would never use it for normal programs as I can’t trust that it will be available!

  5. 640k says:

    The offload copy engine has been available for lanmanger for ages, at least it worked in the 90-ies. Have you been reinventing the wheel?

  6. kme says:

    Since the files are treated as super-hidden under the assumption that they’re short-lived temporary files, maybe the heuristic should be “hidden, starts with a tilde, and was last modified within the last day”.

    That way stale/orphaned ones will show up.

Comments are closed.

Skip to main content