The OVERLAPPED associated with asynchronous I/O is passed by address, and you can take advantage of that

When you issue asynchronous I/O, the completion function or the I/O completion port receives, among other things, a pointer to the OVERLAPPED structure that the I/O was originally issued against. And that is your key to golden riches.

If you need to associate information with the I/O operation, there's no obvious place to put it, so some people end up doing things like maintaining a master table which records all outstanding overlapped I/O as well as the additional information associated with that I/O. When each I/O completes, they look up the I/O in the master table to locate that additional information.

But it's easier than that.

Since the OVERLAPPED structure is passed by address, you can store your additional information alongside the OVERLAPPED structure:

// in C
 CClient *AssociatedClient;

// or in C++
 CClient *AssociatedClient;

When the I/O completes, you can use the CONTAINING_RECORD macro or just static_cast the LPOVERLAPPED to OVERLAPPEDEX* and bingo, there's your extra information right there. Of course, you have to know that the I/O that completed is one that was issued against an OVERLAPPEDEX structure instead of a plain OVERLAPPED structure, but there are ways of keeping track of that. If you're using a completion function, then only use an OVERLAPPEDEX-aware completion function when the OVERLAPPED structure is part of an OVERLAPPEDEX structure. If you're using an I/O completion port, then you can use the completion key or the OVERLAPPED.hEvent to distinguish OVERLAPPEDEX asynchronous I/O from boring OVERLAPPED I/O.

Comments (26)
  1. Vilx- says:

    Does this not fall in the category of "undocumented implementation-specific hacks that you should avoid"?

  2. Joshua says:

    Nice. By using the prefix rule this even avoids undefined behavior in C98.

  3. Francis Gagne says:


    If this is a hack, then object-oriented programming (and virtual functions, in particular) is a hack. :)

  4. Yuhong Bao says:

    Reminds me of async I/O in classic Mac OS, where people did the same thing.

  5. AsmGuru62 says:

    Isn't everything passed in Win API by address?

    Looks like it — however, there is PtInRect()… weird exception.

    {The point is that it is both passed and returned by address. You are basically using a lookup table where the lookup key is the address itself. It turns out that implementing such a table is easy since it's a nop. -Raymond]
  6. James Schend says:

    @yuhong2: IIRC (it's been awhile for obvious reasons) Mac Classic not only "allowed" you to do this, but it was officially supported, if you were coding in C.

  7. Seems like this technique would be vulnerable to slicing, especially if there's a filter driver in the middle which assumes the OVERLAPPED pointer points to, um, an OVERLAPPED-size chunk of memory.

  8. Joshua says:

    @Maurits, well in that case the application programmer will wring the filter driver programmer's neck.

    Slicing only happens if the address changes, which also causes the lookup table and other reasonable methods to break.

  9. Jeff says:

    @Maurits: that wouldn't be slicing, which would be trying to force an OVERLAPPEDEX sized struct through an OVERLAPPED size hole. As it happens, pointers to either of these types do actually point to an OVERLAP-sized chunk of memory.

    On the other hand, and as Raymond pointed out, anyone who wants to use the -EX information has to know that the pointer actually points to something thats OVERLAPEX sized, via some mechanism, including any intermediate filter drivers.

  10. Alex Grigoriev says:


    The GetQueuedCompletionStatus documentation says explicitly that it returns the same OVERLAPPED address that was passed for the IO operation call. Thus, it's not an implementation-specific hack.

    @Maurits, waleri:

    Drivers don't even see or care about OVERLAPPED. It's IO manager concept, hidden from the drivers. And it's associated with the original IRP.

  11. Sure… I'd just be a lot more comfortable expanding the size of a structure if it was a bona fide variable-sized structure (like SP_DEVINFO_DATA, say) with a .cbSize member that said how big the chunk of memory was.

  12. Sunil Joshi says:


    You're missing the point. This is no different from the C example (where the variables just happen to follow the OVERLAPPED) it just saves typing and the need to use CONTAING_RECORD macro. The ReadFile etc. functions don't carre that you've expanded the structure. They just use the bit that's for them and ignore the rest. It's perfectly safe.

  13. scorpion007 says:

    @Kyle, do you also shudder when casting from an int to a float generates code for the conversion? Surely you do this all the time (perhaps without realising it).

  14. They (ReadFile, WriteFile, DeviceIoControl) happen not to care today, but there's nothing in the interface declaration that says they will continue not to care forever.  Plenty of functions take members by reference and then marshal the memory; passing extended blocks of memory to such functions will suffer from slicing.  If you're lucky you'll AV when the "wrong" memory is accessed; if you're unlucky you'll stagger on for a while with garbage data being passed around.

    Now that Raymond has outed this technique, I suppose the likelihood that a future version will break this drops substantially.

    [Even if the OVERLAPPED is marshalled, the documentation says that the lpOverlapped passed to the completion function or the completion port is numerically identical to the one passed to ReadFile/WriteFile/whatever. So say it's marshalled. The marshaller copies only sizeof(OVERLAPPED) bytes into kernel mode. Kernel mode operates on sizeof(OVERLAPPED) bytes. When the call completes, kernel mode copies sizeof(OVERLAPPED) bytes back to the original buffer. You then receive a pointer to that original buffer, and you can access the bonus data you stored immediately after it. As far as the kernel is concerned, your bonus data is totally unrelated to the OVERLAPPED structure. It just happens to occupy adjacent memory. -Raymond]
  15. Thorsten says:

    @Maurits,the documentation already explicitly states that the pointer you get back is the same pointer that was passed in. This automatically excludes any possibility of slicing. There is absolutely nothing new in Raymond's post here except explicitly stating the consequences of the documented API behaviour and the pattern this behaviour enables for the few people that can't figure it out on their own.

  16. Joseph Koss says:

    I dont think that some of you understand.

    When you pass the pointer by its value (memory offset to where it points), there is no way for the called function to alter your codes perception of that value (where it points) unless it is the returned value of the function.

    For example, x = foo(x)

    The C abstract machine does not pass changes to parameters back to the caller, regardless of what they are. When you wish to do that sort of thing, you pass the value of a pointer and then the function can write to the memory it points to but cannot change the callers pointer value itself (you would need a pointer to a pointer…)

  17. I see, the sample here (which counts as documentation) relies on this:…/aa365601(v=VS.85).aspx

    "The parameters of the ReadFileEx and WriteFileEx functions specify a completion routine and a pointer to an OVERLAPPED structure. This pointer is passed to the completion routine in its lpOverLap parameter. Because the OVERLAPPED structure points to the first member in the structure allocated for each pipe instance, the completion routine can use its lpOverLap parameter to access the structure for the pipe instance."

  18. Kyle says:


    Yeah, and multiple inheritance would be *really* hacky.  I shudder at the thought of type-casting actually resulting in code generation.

  19. waleri says:

    I don't think filter driver would be a problem, because:

    a) Driver will simply pass same OVERLAPPED* to the next driver

    b) Will create another (own) instance of OVERLAPPED, pass it along and upon completion will notify the original OVERLAPPED*

    I believe, in terms of drivers, IRP is the word and it is pretty much the same logic

  20. Marcel says:

    @Joseph: Sorry, but it appears that you didn't understand the topic… at all. What you say has nothing to do with the things discussed here. Nobody is even remotely arguing that e.g. ReadFile magically changes the pointer it was given as a parameter…

  21. Ricardo Costa says:

    Raymond, I believe the C++ example is not correct since the position of the base class subobject in the derived class is unspecified according to ISO C++ 2003 Standard (10-3, page 168), and you assume that the base class subobject is always at the beginning. The C example would be fine in C++ too, so I'd stick with it.

    [The code does not make this assumption. That's why it's important to use static_cast instead of reinterpret_cast. Try it: Add a virtual method to OVERLAPPED (so a vtable goes in front) and observe what the compiler does. -Raymond]
  22. Marcel says:

    @Ricardo: You might also want to read up on chapter 9-2, point 17, where a pointer to a POD structure is guaranteed to be pointing to its first member.

  23. Random832 says:

    "you assume that the base class subobject is always at the beginning." IIRC, if it's not, then the static_cast from one pointer type to another will alter the address accordingly.

  24. GWO says:

    static_cast<> is not intended for casting base types to derived.  That's error prone.  dynamic_cast<> does exactly what you'd like, and as long you check for a null result, actually provides type-safety, which competent programmers consider to be a good thing.

  25. Anonymouse says:

    GWO: You can't use dynamic_cast<> to cast from OVERLAPPED* to a derived class.  This is because OVERLAPPED doesn't have any virtual members, so it doesn't have a virtual function table, so there's nowhere for the compiler to put the type information needed by dynamic_cast<>.

    static_cast<> is the right thing to use here.

  26. waleri says:

    In DDK, there is a macro (can't remember the name right now) that calculates address of a base type from a member address. Same can be used in this case. Not as safe as dynamic_cast<> but pretty fast.

Comments are closed.