File Handle Hell

Raymond Chen has an interesting post about a program that tried to open a file, got back an error code, and used the error code as a file handle. In this particular scenario everything worked, but completely by accident.

Reading it, I remembered back to my own "bug from hell" related to file handle.

The scenario: I was working on NuMega's BoundsChecker in the very early days of Win32. BoundsChecker forces its DLL into the target process very early in the target process lifetime. The BoundsChecker DLL then writes various messages to its output file.

One particular program (SourceSafe) had some old dead DOS code that immediately closed the three standard handles (stdin, stdout, stderr) to free up as many handles as possible.

Imagine what happens in this scenario:

  • The BoundsChecker DLL loads, and opens an output file. The file handle it gets back is 1.
  • SourceSafe blindly closes stdin, stdout, & stderr (0,1,2, as I recall). Thus, it closes BoundsChecker file handle. BoundsChecker has no knowledge of this.
  • SourceSafe opens its open file handle, and gets back 1.

Imagine the havoc this creates. Both BoundsChecker and SourceSafe are writing to the same file handle (1). They're writing completely different data. The upshot in this case was that the SourceSafe database got completely corrupted.

Man, did I take a lot of flack for that one!