Kernel handles are not reference-counted


Here's a question that floated past some time ago:

In my code, I have multiple objects that want to talk to the same handle (via DeviceIoControl). Each time I create an object, I use DuplicateHandle to increment the reference count on the handle. That way, when each object calls CloseHandle, only the last one actually closes the handle. However, when I run the code, I find as soon as the first object calls CloseHandle, the handle is no longer valid and nobody else can use it. What flags do I need to pass to CreateFile to get this to work?

In other words, the code went something like this:

// h is the handle that we want to share with a new CFred object

CFred *MakeFred(HANDLE h)
{
 // "Duplicate the handle to bump the reference count"
 // This code is wrong - see discussion
 // All error checking removed for expository purposes
 HANDLE hDup;
 DuplicateHandle(GetCurrentProcess(), h,
                 GetCurrentProcess(), &hDup,
                 0, FALSE, DUPLICATE_SAME_ACCESS);
 return new CFred(h);
}

Kernel handles aren't reference-counted. When you call CloseHandle, that closes the handle, end of story.

From the original problem statement, we know that the CFred object closes the handle when it is destroyed. Just for argument's sake, let's say that the caller goes something like this:

CFred *pfred1 = MakeFred(h);
CFred *pfred2 = MakeFred(h);
delete pfred1;
delete pfred2;

What actually happens when you run this fragment?

The first time we call MakeFred we take the original handle h and duplicate it, but we give the original handle to the CFred constructor and leak the hDup! The original poster assumed that duplicating a handle merely incremented the handle's imaginary reference count, so that h == hDup. (Which would also have made the original poster wonder why we even bother having a lpTargetHandle parameter in the first place.)

When pfred1 is deleted, it closes its handle, which is h. This closes the h handle and renders it invalid and available to be recycled for another CreateFile or other operation that creates a handle.

When pfred2 is deleted, it also closes its handle, which is still h. This is now closing an already-close handle, which is an error. If we had bothered calling a method on pfred2 that used the handle, it would have gotten failures from those operations as well, since the handle is no longer valid. (Well, if we're lucky, we would have gotten a failure. If we were unlucky, the handle would have been recycled and we ended up performing a DeviceIoControl on somebody else's handle!)

Meanwhile, the calling code's copy of h is also bad, since pfred1 closed it when it was deleted.

What we really want to do here is duplicate the handle and pass the duplicate to each object. The DuplicateHandle function creates a new handle that refers to the same object as the original handle. That new handle can be closed without affecting the original handle.

// h is the handle that we want to share with a new CFred object

CFred *MakeFred(HANDLE h)
{
 // Create another handle that refers to the same object as "h"
 // All error checking removed for expository purposes
 HANDLE hDup;
 DuplicateHandle(GetCurrentProcess(), h,
                 GetCurrentProcess(), &hDup,
                 0, FALSE, DUPLICATE_SAME_ACCESS);
 return new CFred(hDup);
}

The fix is one word, highlighted in blue. We give the duplicated handle to the CFred object. That way, it gets its own handle which it is free to close any time it wants, and it won't affect anybody else's handle.

You can think of DuplicateHandle as a sort of AddRef for kernel objects. Each time you duplicate a handle, the reference count on the kernel object goes up by one, and you gain a new reference (the new handle). Each time you close a handle, the reference count on the kernel object drops by one.

In summary, a handle is not a reference-counted object. When you close a handle, it's gone. When you duplicate a handle, you gain a new obligation to close the duplicate, in addition to the existing obligation to close the original handle. The duplicate handle refers to the same object as the original handle, and it is the underlying object that is reference-counted. (Note that kernel objects can have reference from things that aren't handles. For example, an executing thread maintains a reference to the underlying thread object. Closing the last handle to a thread will not destroy the thread object because the thread keeps a reference to itself as long as it's running.)

Comments (16)
  1. Medinoc says:

    I love the DuplicateHandle() function since the very first time I managed to make an "big" Interprocess Communication work (excluding sockets).

    It also helped me learn that a file object has only one file pointer, regardless of the number of handles. In fact, I can’t really think of any good reason to call DuplicateHandle() on a file handle. But I did know a dirty trick on duplicated console input buffer handles…

  2. dgr says:

    so many people get it wrong. even ones who understand pointers.

  3. Maks Verver says:

    @CornedBee: the handle is not just a reference to the kernel object, but also a token that signifies that a process has access to the object and is allowed to decrease the internal reference count. The kernel must be able to enforce that such a token is genuine, to make sure the internal books remain balanced.

    Suppose a handle was just a pointer to a kernel object that has a single reference count. Then processes would be able to delete references they didn’t create (maybe that would do for Windows 3.11, but not in any secure system) and you wouldn’t be able to remove outstanding references on process exit.

    So, you’ll have to do some per-process bookkeeping, which means that you’ll effectively have multiple handles to the same kernel object anyway (because handles differ between processes).  As Raymond points out, POSIX file descriptors work in the same way and that is not surprising since the scenario is similar: different processes (which must be isolated) share access to a single object.

    Given that you have to keep track of references, it’s easier to keep track of simple ones than having a two-level scheme of reference counting per process, and per kernel object. Ofcourse, if you want to have only one kernel reference and do per-process reference counting too (which is arguably somewhat more space-efficient) you can implement that in user mode just fine. (But don’t bother unless you’re creating hundreds of thousand handles.)

  4. Sohail says:

    Or you could use a funny thing called a reference counted pointer! std::tr1::shared_ptr

  5. Ed Massey says:

    The fact is, this stuff is not really that well documented.  In an effort to abstract the internals of the system (taken to the extreme), many of the nuiances are left inadequately explained, leading to many hours of developer frustration.

    Even a simple picture paints a thousand words, so here’s the data structures from the Windows kernel (from WinDbg 6.6 running under Windows XP).  Each (executive) process in the kernel is represented by the EPROCESS structure.  The ObjectTable field points to a HANDLE_TABLE structure that maintains the per-process handle table.  Each entry in this table is represented by the HANDLE_TABLE_ENTRY structure, whose Object field points to the object, and indirect, the OBJECT_HEADER, which contains the actual reference (and handle) counts.  The Body field is the first byte of the type-specific object body…

    0:000> dt _EPROCESS

    —snip—

      +0x0c4 ObjectTable      : Ptr32 _HANDLE_TABLE

    —snip—

    0:004> dt _HANDLE_TABLE

      +0x000 TableCode        : Uint4B

      +0x004 QuotaProcess     : Ptr32 _EPROCESS

      +0x008 UniqueProcessId  : Ptr32 Void

      +0x00c HandleTableLock  : [4] _EX_PUSH_LOCK

      +0x01c HandleTableList  : _LIST_ENTRY

      +0x024 HandleContentionEvent : _EX_PUSH_LOCK

      +0x028 DebugInfo        : Ptr32 _HANDLE_TRACE_DEBUG_INFO

      +0x02c ExtraInfoPages   : Int4B

      +0x030 FirstFree        : Uint4B

      +0x034 LastFree         : Uint4B

      +0x038 NextHandleNeedingPool : Uint4B

      +0x03c HandleCount      : Int4B

      +0x040 Flags            : Uint4B

      +0x040 StrictFIFO       : Pos 0, 1 Bit

    0:004> dt _HANDLE_TABLE_ENTRY

      +0x000 Object           : Ptr32 Void

      +0x000 ObAttributes     : Uint4B

      +0x000 InfoTable        : Ptr32 _HANDLE_TABLE_ENTRY_INFO

      +0x000 Value            : Uint4B

      +0x004 GrantedAccess    : Uint4B

      +0x004 GrantedAccessIndex : Uint2B

      +0x006 CreatorBackTraceIndex : Uint2B

      +0x004 NextFreeTableEntry : Int4B

    0:004> dt _HANDLE_TABLE_ENTRY_INFO

      +0x000 AuditMask        : Uint4B

    0:004> dt _OBJECT_HEADER

      +0x000 PointerCount     : Int4B

      +0x004 HandleCount      : Int4B

      +0x004 NextToFree       : Ptr32 Void

      +0x008 Type             : Ptr32 _OBJECT_TYPE

      +0x00c NameInfoOffset   : UChar

      +0x00d HandleInfoOffset : UChar

      +0x00e QuotaInfoOffset  : UChar

      +0x00f Flags            : UChar

      +0x010 ObjectCreateInfo : Ptr32 _OBJECT_CREATE_INFORMATION

      +0x010 QuotaBlockCharged : Ptr32 Void

      +0x014 SecurityDescriptor : Ptr32 Void

      +0x018 Body             : _QUAD

    Hope that clears things up.

  6. CornedBee says:

    So let me get this straight.

    There are kernel objects.

    There are handles to kernel objects. When all handles to an object have been closed, the kernel object is destroyed.

    With DuplicateHandle, you can create a new handle to a given kernel object. The new handle has a different value, but references the same object.

    Hmm …

    It seems to me like this would be more complicated to implement than really doing it the way this guy thought it worked: by having all handles to the same object be the same, and just incrementing/decrementing reference counts. Any specific reason it wasn’t done this way?

    Offhand, I can only come up with one: defense against multiple calls of CloseHandle on the same handle invalidating other handles. Given that several processes would share the same handle values (after all, what’s the point of simplifying the system when you still need the complicated version for some use cases), that’s probably a very valid reason.

    dgr: That’s hardly surprising. After all, the pointers themselves aren’t reference-counted, it can only be the underlying memory that is. The problem here is that handles do *not* behave like simple, reference-counted pointers to kernel objects.

    OK, came up with another reason: there is handle-specific information to be stored, like open mode.

    [You’ll have to ask the unix folks. They had dup(2) decades ago. (Good luck sharing objects across processes if the handle number must be globally unique.) -Raymond]
  7. Luke Amery says:

    I hope this isn’t offtopic…

    I have always wondered about the accounting associated with DuplicateHandle when creating handles for other processes.

    For example, two processes A and B. B is just an unwitting process standing around minding it’s own business. Process A is nasty and wants Process B to either go away or at least look extremely bad.

    Process A starts Duplicating handles with Process B as the target process. Process B never knows about these handles as there is never any IPC between A and B.

    Unix solves this problem by limiting handle duplication to an IPC mechanism (the kernel can then know each process is in agreement and any failure can be cleanly dealt with – either the IPC op worked or it didn’t).

    Just wondering what other people think…

  8. Mark Steward says:

    Luke: to duplicate a handle into Process B, you need a handle to Process B with PROCESS_DUP_HANDLE access.  Which, as http://msdn2.microsoft.com/ms684880.aspx notes, allows you to just as easily duplicate (HANDLE)-1, and do what you like with the target process.

  9. HS says:

    You’ll have to ask the unix folks. They had dup(2) decades ago.

    Those who do not understand unix, are doomed to reinvent it. Poorly.

  10. nksingh says:

    @HS

    And those who do understand UNIX long for something better.

  11. Jules says:

    Perhaps a more interesting question is why, when something is working, people tend to assume it’s because they’ve not include the correct magic incantation of flags in their system calls.

    It’s a perplexing question, actually.  A quick scan of the CreateFile documentation shows that there isn’t a FILE_ALLOW_CLOSE_ORIGINAL_HANDLE_INSTEAD_OF_DUPLICATE that could plausibly fix this problem, so why didn’t the poster realise this wasn’t the problem?

  12. Jules says:

    nksingh> Time for Plan 9, then? ;)

  13. pdohara@smallwarz.org says:

    Jules wrote:

    "Perhaps a more interesting question is why, when something is working, people tend to assume it’s because they’ve not include the correct magic incantation of flags in their system calls."

    Perhaps because so many attributes are not documented and must be discovered :-).

    Pat O

  14. BryanK says:

    Sohail:  "a reference counted pointer! std::tr1::shared_ptr"

    So you’re OK with requiring user programs to be written in C++?  And you’re OK with forcing all programs to use a specific C++ standard library (that includes the ref-counted pointer), and if they use some other C++ standard library, they’d be incompatible with the kernel?

    And, after all that, you’re happy having the library do all the same work that the kernel does right now *anyway*, to give you the *sole* benefit of having this particular situation work?

    If so, that’s fine.  But that seems like a massively, massively over-engineered solution to a nonexistant problem to me.  Just realize that objects have refcounts instead of handles, and each additional reference can have a different handle value.  Now it’s obvious that the lpTargetHandle parameter’s out value could be different, and poof, the problem disappears.

    (Granted, I don’t know whether it’s documented that objects have refcounts instead of handles.  But I must have picked it up *somewhere*.)

  15. Jules says:

    "Granted, I don’t know whether it’s documented that objects have refcounts instead of handles.  But I must have picked it up *somewhere*."

    It’s made pretty clear by the documentation on CloseHandle:

    "CloseHandle invalidates the specified object handle, decrements the object’s handle count, and performs object retention checks. After the last handle to an object is closed, the object is removed from the system."

  16. It didn’t do quite as much back then.

Comments are closed.

Skip to main content