You just have to accept that the file system can change


A customer who is writing some sort of code library wants to know how they should implement a function that determines whether a file exists. The usual way of doing this is by calling GetFileAttributes, but what they've found is that sometimes GetFileAttributes will report that a file exists, but when they get around to accessing the file, they get the error ERROR_DELETE_PENDING.

The lesser question is what ERROR_DELETE_PENDING means. It means that somebody opened the file with FILE_SHARE_DELETE sharing, meaning that they don't mind if somebody deletes the file while they have it open. If the file is indeed deleted, then it goes into "delete pending" mode, at which point the file deletion physically occurs when the last handle is closed. But while it's in the "delete pending" state, you can't do much with it. The file is in limbo.

You just have to be prepared for this sort of thing to happen. In a pre-emptively multi-tasking operating system, the file system can change at any time. If you want to prevent something from changing in the file system, you have to open a handle that denies whatever operation you want to prevent from happening. (For example, you can prevent a file from being deleted by opening it and not specifying FILE_SHARE_DELETE in your sharing mode.)

The customer wanted to know how their "Does the file exist?" library function should behave. Should it try to open the file to see if it is in delete-pending state? If so, what should the function return? Should it say that the file exists? That it doesn't exist? Should they have their function return one of three values (Exists, Doesn't Exist, and Is In Funky Delete State) instead of a boolean?

The answer is that any work you do to try to protect users from this weird state is not going to solve the problem because the file system can change at any time. If a program calls "Does the file exist?" and the file does exist, you will return true, and then during the execution of your return statement, your thread gets pre-empted and somebody else comes in and puts the file into the delete-pending state. Now what? Your library didn't protect the program from anything. It can still get the delete-pending error.

Trying to do something to avoid the delete-pending state doesn't accomplish anything since the file can get into that state after you returned to the caller saying "It's all clear." In one of my messages, I wrote that it's like fixing a race condition by writing

// check several times to try to avoid race condition where
// g_fReady is set before g_Value is set
if (g_fReady && g_fReady && g_fReady && g_fReady && g_fReady &&
    g_fReady && g_fReady && g_fReady && g_fReady && g_fReady &&
    g_fReady && g_fReady && g_fReady) { return g_Value; }

The compiler folks saw this message and got a good chuckle out of it. One of them facetiously suggested that they add code to the compiler to detect this coding style and not optimize it away.

Comments (31)
  1. Jon says:

    "Should they have their function return one of three values (Exists, Doesn’t Exist, and Is In Funky Delete State) instead of a boolean?"

    Everybody knows its:

    enum Bool

    {

       True,

       False,

       FileNotFound

    };

    Source: http://worsethanfailure.com/Articles/What_Is_Truth_0x3f_.aspx

    I guess that is slightly better than how UNIX lets you delete files in use. Those disappear from the directory listing but still exist on the disk until the last person closes it. So you can have a 10 GB file that isn’t part of any directory but it is taking up space. Although that has the advantage of allowing me to recreate it immediately while the old file is still open.

  2. Keithius says:

    Heh. Very amusing. (I can understand why the compiler folks got a chuckle out of that one.)

  3. Bryan says:

    I got a good laugh out of this one.

    Another one I say ‘hear hear!’.

    People always want to take control over the whole system instead of just admitting that there are, sometimes, some situations for which they just can’t account.

  4. Greg D says:

    Almost every time I’ve tried to write something and caught myself running into some bizarro-world corner case, I’ve realized (sooner or later) that the WTF wasn’t in the corner case.  Rather, it was in the new software’s underlying assumptions.

    It can sometimes be a real challenge to take a deep breath, step back, re-evaluate assumptions, and correct them after I’ve been committed to a course of action.  My boss’ boss really doesn’t like it.  Heh.

  5. Neil says:

    If you read worsethanfailure.com more carefully you’ll find that the solution to the compiler team optimising the condition is of course simply to nest your ifs i.e.

    if (g_fReady) { if (g_fReady) { if (g_fReady) { return g_Value; } } }

  6. Dan says:

    No, clearly the solution is:

    while (g_fReady) {

     Sleep(0);

    }

    // Aha!  Caught you, you sneak!

  7. Dave says:

    Of course the compiler guys laughed at you, that code will never work! You have to declare g_fReady as volatile. :-)

  8. Ben Cooke says:

    Presumably the succinct answer is: Don’t check to see if the file exists before you open it, just open it and see if you succeed!

    (Of course, if you’ve asked a user to select a file and that file goes away between the user selecting it and you opening it, then there’s little you can do aside from telling the user "I couldn’t open the file. Sorry.". However, list a bunch of files onscreen for selection is slightly different to just checking if a file exists, and has a lot more places where the filesystem can change out from under you and cause confusion.)

  9. Steve Loughran says:

    If the g_fReady was marked as volatile then yes, the if() shouldnt be optimised, and it shouldnt be pulled in to a register. That’s part of the C++ specs, I recall vaguely.

    otherwise, they could put it a register, flatten the loop, or even re-order the accesses. So the second g_fReady check could take place before the first…  

  10. JM says:

    Of course, you could stick all the qualifiers you’d want on that variable and it still wouldn’t do anything to prevent the race condition. That’s what makes the compiler joke so sweet.

    When you’re trying to solve the wrong problem, it doesn’t matter how intricate your solution is.

  11. DrkMatter says:

    "otherwise, they could put it a register, flatten the loop, or even re-order the accesses."

    The C++ standard guarantees the order of evaluation of successive "&&" so that if the first one evaluates to false, it never evaluates the second one.

  12. Don’t check to see if the file exists before you open it, just open it and see if you succeed!

    And if you are to display a list, simply open all files first (deny sharing). After the user selects the file, close others :)

    Seriously, I bumped into a case where GetFileAttributes said a file did not exist, and it was wrong, opening the file would not fail (because the file was there). I traced it to invalid file date/time.

    I say ‘Just open it’, there’s no need for a DoesFileExist function in a library if files are to be accessed…

  13. Gene says:

    When you’re trying to solve the wrong problem, it doesn’t matter how intricate your solution is.

    Man… sounds like something I’d see on a despair.com poster…

  14. Yuhong Bao says:

    And that is exactly why IsBad*Ptr is almost useless. You are better off using SEH to catch exceptions that result from the memory access instead.

  15. pete says:

    This kind of thing usually happens when somebody is using the file system as something other than a file system. They really should be using a named semaphore or something, but instead they’re using the existence of a file. It would be less sad if portability were in the picture, but based on RC’s description, it’s not.

  16. pete says:

    Meanwhile, the real reason I came down here to fill in this form is to tell a story about compiler optimization. I once wrote code something like this:

       Thing * GetThing (void);

       bool gotThing (!!GetThing ( ));

    gotThing would occasionally be false because the compiler "optimized away" the !! and assigned the least significant bits of the pointer to the bool. The compiler’s tech support department assured me this was by design. Happily, I persuaded them to run their answer past an actual compiler engineer, who promptly flogged them with a wet noodle.

  17. tcliu says:

    Would transactional NTFS enable one to do something like what this person is trying? That is, if we began a transaction at the start, checked whether the file existed and found that yes it did, we would be sure that it would keep existing until the xa was committed or aborted?

  18. foxyshadis says:

    tcliu, that’s much too low-level, when all you really need is to open a file to make the kernel be the gatekeeper, not the filesystem driver – if someone else has incompatible flags it’ll let you know right away that it can’t enforce your preference. Once you’ve opened it, other programs that try to mess with it will be unable to do so, the only thing you have to watch out for is your own threads in that case.

  19. Aaargh! says:

    “I guess that is slightly better than how UNIX lets you delete files in use.”

    The Unix way has one really big advantage, you can do updates while the system is running. Replacing libc on a running system is no problem, just reload all daemons after the update and you’re off, no reboots needed.

    And of course, in Unix you don’t delete the file as such, you unlink the file. The directory entry and the actual file are not one entity, there could be multiple directory entries pointing to the same file.

    Not being able to delete an open file is one of the things that really annoys me about Win32. Especially since Win32 severely lacks proper tools by default. On Unix a simple lsof | grep <file> will tell you the process that has a file open, on windows standard tools like these are missing, requiring you to get some nonstandard tool from the web somewhere.

    [Windows can replace files that are in use, it just chooses not to because doing so creates other problems. -Raymond]
  20. Aaargh!: The standard tool on Windows is called <a href="http://www.microsoft.com/technet/sysinternals/utilities/processexplorer.mspx">Process Explorer</a>. As many other useful admin/poweruser tools, it is not shipped with any version of Windows I know of. But it does exist and has existed for quite some time.

  21. Aaargh! says:

    "As many other useful admin/poweruser tools, it is not shipped with any version of Windows I know of. But it does exist and has existed for quite some time."

    I know, I use that regularly.

    But processexplorer is a GUI tool, so you can’t script it, and since it’s not installed by default you can’t assume it’s on the system.

    These kinds of tools should be standard if Microsoft ever wants Windows to be seen as a serious OS instead of just a toy.

    Why doesn’t MS just supply bash and the standard Unix tools with Windows ? It won’t cost them a penny and prevent a lot of annoyance from users who expect these tools to be available on a machine. Even Apple includes them with their OS.

  22. Ian says:

    No, clearly the solution is:

    >

    while (g_fReady) {

      Sleep(0);

    }

    I know this was a joke, but just in case anyone thinks this is a good idea, note that Sleep(0) won’t let a lower-priority thread run. Apparently, SwitchToThread() is the correct function to call if you want to give up the rest of your timeslice. But this won’t necessarily work on a hyperthreading CPU, where you should use YieldProcessor() instead.

    At least, that’s what I heard.

  23. Goran says:

    Aaaargh! :

    "But processexplorer is a GUI tool, so you can’t script it"

    Well… The latter does not follow the former.

    If you need something done from the command line, you need command-line switches. If you need some output from it, Automation is the key (PE would need to support it, though). I am saying that because redirecting and parsing stdout is crap compared to programmatic access.

  24. Aaargh! says:

    "I am saying that because redirecting and parsing stdout is crap compared to programmatic access."

    Only because windows lacks proper tooling. Usually it’s way easier and quicker to just create a small shellscript. Plus, sysadmins can edit shellscripts, sysadmins can not directly edit compiled binaries.

  25. adam says:

    I actually ran across code like that once.  I couldn’t figure out what the hell they were trying to do and ended up asking the author.  Three months later I was out of that place…

  26. SirPavlova says:

    Process Explorer is a recent Microsoft acquisition as part of Sysinternals, not a standard Windows tool. I rather expected it to become a standard tool with Vista, but strangely enough it didn’t.

    Besides, if the tool doesn’t ship with the system, it’s not really standard. The whole idea of a standard tool is that it’s there when you need it. If you have to get it from microsoft.com, you might as well be downloading any third party program (a group which basically includes Process Explorer – it hasn’t changed much since they were bought).

  27. Evan says:

    There is a paper, A methodology for implementing highly concurrent data objects, from TOPLAS 1993 that includes the following to try to stop a race condition:

    while ((*Q)–>responses[P].toggle != new_toggle

       || (*Q)->responses[P].toggle != new_toggle)

    { … }

    I haven’t proven that it doesn’t help, but boy am I suspicious at it…

  28. movl says:

    But wait a second, doesn’t windows already provide ways to prevent race conditions? I think they were mutexes and events. If one such object has the …_SYNCRONIZE or so access right then the thread can wait for it. The real problem would then be a deadlock. You might be able to catch that by using a “controller” thread, which is required if you do your own (userland) threads implementation.

    Just thinking out loud here.

    [In the future, please complete your thoughts before posting them. -Raymond]
  29. movl says:

    Only to be removed for going too far off-topic? No thanks :p

  30. You have to accept that the registry will change.

Comments are closed.