Using DuplicateHandle to help manage the ownership of kernel handles


A customer was using a third party I/O library that also gave you access to the underlying HANDLE, in case you needed it. The customer needed that underlying HANDLE so it could pass it to _open_osfhandle() and get a C runtime file descriptor, which could then be used by other C runtime functions to operate on the I/O object.

Everything was great until it came time to close the I/O object, because both the I/O library and the C runtime tried to close the handle. and that resulted in assertion failures due to invalid handles.

The problem here is that both the I/O library and the C runtime think that they are responsible for closing the handle. The I/O library wants to close the handle because it created the handle in the first place, and the special method to obtain that underlying HANDLE wasn't transferring ownership of the handle to you; it merely gave you a handle that you could borrow. On the other hand, the _open_osfhandle() function will close the handle when the file descriptor is closed, because the function assumes that you're giving it not only the handle, but also the responsibility to close the handle.

Neither library has a way to change the handle semantics. There isn't a way to tell the I/O library or the C runtime, "Hey, don't close that handle."

The solution here is to use the Duplicate­Handle function to create a brand new handle that refers to the same underlying kernel object. You can then pass the duplicate to _open_osfhandle(). Both the I/O library and the C runtime library will close their respective handles. Since each handle is closed exactly once, balance is restored to the universe.

Exercise (easy): Suppose you have a C runtime file descriptor, and you want to take the underlying kernel handle and give it to another library, which will close the handle you give it. How do you manage this without running into a double-close bug?

Exercise (slightly harder): Suppose your program needs more than 2048 C runtime file descriptors, which is more than _setmaxstdio accepts. Fortunately, your program doesn't actively use all of the descriptors at the same time, so you're thinking that you can virtualize the file descriptors by "paging them in" and "paging them out" from underlying kernel handles. How would you do this?

Comments (12)
  1. ZLB says:

    Easy Excercise: use _get_osfhandle(...) to get a HANDLE from a file descriptor, the DuplicateHandle() it and pass the duplicate to the library.

  2. SimonRev says:

    To the harder question -- This feels a lot like when someone asks why they cannot have more than 10000 GDI objects open at the same time. To which the answer is almost always: You are doing it wrong. Sure you could use DuplicateHandle with some of the machinery in the stdio library to get the underlying OS handle, but it still feels very wrong. If you don't need 2048 simultaneous file descriptors, then close them when you don't need them. If you do need 2048 simultaneous file descriptors, redesign your program.

    1. What if you're designing a file server or CDN?

      1. Dmitry says:

        If the main point of your design is doing heavy lifting in some specific area, then you probably will not get away with one-liner. Standard libraries are designed for convenience in the first place, not for some extreme edge cases.

    2. Antonio Rodríguez says:

      I first though "if you have to asks the limit, you are doing it wrong". Then I remembered cases like file servers, database servers, etc. Even with caching, it's reasonable to have hundreds or thousands of files opened at the same time in those cases. It's not the 80s anymore, and Windows is used as a server OS, so its API and C runtime have to support those scenarios.

  3. Darran Rowe says:

    For the harder one, I would say that you keep a table of your Windows handles and permissions and ask for a C runtime file handle based on that Windows handle. You can then duplicate the handle and use _open_osfhandle to get the file handle.
    Once the handle has been used, you can then close the C runtime file handle. This allows reuse of the C runtime while keeping everything open.

    1. Klimax says:

      I don't think there's better answer. (Well, I though I might get my answer before other... :D )

  4. Florian S. says:

    Correct me if I'm wrong, but wouldn't it be better if the third-party library created a duplicate of the handle before it actually returns it to the caller? I mean, if you were to implement it... :)

    1. Joshua says:

      Nope. Almost never does the callee need to hang on to the handle. Usually stuff like this exists for calling DeviceIoControl.

    2. Antonio Rodríguez says:

      As Joshua says, it's reasonable that the library provides the original handle: if you need it for short-term use, you use it directly. And if you need to make longer use, it's your responsibility to duplicate and close it. As long as it is documented, it's perfectly logical and valid.

      In any case, it's futile discussing whether the library should duplicate the handle or not. It's a third party library, and you can't modify it. All you can do is to use it or to work around its bugs or design problems.

  5. Josh B says:

    I'm hoping Raymond comes in to give his answer to the hard one before comments close, rather than sticking it on the queue for 2020. That sounds like something that might come in handy someday.

    1. Klimax says:

      It's not really hard and most of tools are already present. The only missing piece is _get_osfhandle.

Comments are closed.

Skip to main content