File Handle Hell


Raymond Chen has an interesting post about a program that tried to open a file, got back an error code, and used the error code as a file handle. In this particular scenario everything worked, but completely by accident.

Reading it, I remembered back to my own “bug from hell” related to file handle.

The scenario: I was working on NuMega’s BoundsChecker in the very early days of Win32. BoundsChecker forces its DLL into the target process very early in the target process lifetime. The BoundsChecker DLL then writes various messages to its output file.

One particular program (SourceSafe) had some old dead DOS code that immediately closed the three standard handles (stdin, stdout, stderr) to free up as many handles as possible.

Imagine what happens in this scenario:

  • The BoundsChecker DLL loads, and opens an output file. The file handle it gets back is 1.
  • SourceSafe blindly closes stdin, stdout, & stderr (0,1,2, as I recall). Thus, it closes BoundsChecker file handle. BoundsChecker has no knowledge of this.
  • SourceSafe opens its open file handle, and gets back 1.

Imagine the havoc this creates. Both BoundsChecker and SourceSafe are writing to the same file handle (1). They’re writing completely different data. The upshot in this case was that the SourceSafe database got completely corrupted.

Man, did I take a lot of flack for that one!


Comments (8)

  1. Steve Eck says:

    We had an interesting scenario with file handles at my work this summer.

    On 64-bit Windows 2003, new child processes started by 32-bit Windows services would eventually stop inheriting open handles from their parents. They could open files, and get back 0/1/2/etc as file handles, but they weren’t inheriting handles from their parents anymore (the file handle was returned as -1).

    This wasn’t something that we could force to happen, but once it started occurring it wouldn’t go away until a reboot.

    Eventually someone at MS tracked it down to a bug and provided a hot-fix, but it took a long while to get there.

    http://soeck.blogspot.com/2004/06/bad-file-descriptor-for.html

  2. Steve Eck says:

    We had an interesting scenario with file handles at my work this summer.
    <br>
    <br>On 64-bit Windows 2003, new child processes started by 32-bit Windows services would eventually stop inheriting open handles from their parents. They could open files, and get back 0/1/2/etc as file handles, but they weren’t inheriting handles from their parents anymore (the file handle was returned as -1).
    <br>
    <br>This wasn’t something that we could force to happen, but once it started occurring it wouldn’t go away until a reboot.
    <br>
    <br>Eventually someone at MS tracked it down to a bug and provided a hot-fix, but it took a long while to get there.
    <br>
    <br><a target="_new" href="http://soeck.blogspot.com/2004/06/bad-file-descriptor-for.html">http://soeck.blogspot.com/2004/06/bad-file-descriptor-for.html</a&gt;
    <br>

  3. Dave Seidel says:

    Ah, yes, the good old days. But you forgot to mention that after you did it, you went and did it again a few minutes later, which made it even worse! I remember spending a fair amount of the next two weeks after that trying to restore the SourceSafe data. 🙂

    You should know that even today, we tell the tender young things that we hire out of college: "Don’t run BoundsChecker on the version control system!"

  4. Jim Austin says:

    I was just reminiscing with one of my co-workers and told him that story just a few hours before I saw your post.

    Ah, the good old days, we certainly had fun back then.

  5. Carol Tyler says:

    Maybe you can tell your readers about "Matt’s Huge Irritating Banner" next…

  6. John Maver says:

    Yes, tell us that one. I haven’t heard that one yet.

  7. Jim Austin says:

    What about the phone sex incident?

  8. Tony says:

    NOW I UNDERSTAND! (Sorry for shouting)

    We had the same problem with VSS and the Novell Netware Client. Some Netware error information landed in some weirdly named aafaaaaa.b file and damaged the VSS repository…

Skip to main content