I know that an overlapped file handle requires an lpOverlapped, but why does it (sometimes) work if I omit it?


A customer observed that the formal requirements for the Read­File function specify that if the handle was opened with FILE_FLAG_OVERLAPPED, then the lpOverlapped parameter is mandatory. But the customer observed that in practice, passing NULL results in strange behavior. Sometimes the call succeeds, and sometimes it even returns (horrors!) valid data. (Actually the more horrifying case is where the call succeeds and returns bogus data!)

Now sure, you violated one of the requirements for the function, so the behavior is undefined. But why doesn't Read­File just flat-out fail if you call it incorrectly?

The answer is that the Read­File function doesn't know whether you're calling it correctly.

The Read­File function doesn't know whether the handle you passed was opened for overlapped or synchronous access. It just trusts that you're calling it correctly and builds an asynchronous call to pass into the kernel. If you passed a synchronous handle, well, it just issues the I/O request into the kernel anyway, and you get what you get.

This quirk traces its history all the way back to the Microsoft Windows NT OS/2 Design Workbook. As originally designed, Windows NT had a fully asynchronous kernel. There was no such thing as a blocking read. If you wanted a blocking read, you had to issue an asynchronous read (the only kind available), and then block on it.

As it turns out, developers vastly prefer synchronous reads. Writing asynchronous code is hard. So the kernel folks relented and said, "Okay, we'll have a way for you to specify at creation time whether you want a handle to be synchronous or asynchronous. And since lazy people prefer synchronous I/O, we'll make synchronous I/O the default, so that lazy people can keep being lazy."

The Read­File function is a wrapper around the underlying Nt­Read­File function. If you pass an lpOverlapped, then it takes the OVERLAPPED structure apart so it can pass the pieces as an Io­Status­Block and a Byte­Offset. (And if you don't pass an lpOverlapped, then it needs to create temporary buffers on the stack.) All this translation takes place without the Read­File function actually knowing whether the handle you passed is asynchronous or synchronous; that information isn't available to the Read­File function. It's relying on you, the caller, to pass the parameters correctly.

As it happens, the Nt­Read­File function does detect that you are trying to perform synchronous I/O on an asynchronous handle and fails with STATUS_INVALID_PARAMETER (which the Read­File function turns into ERROR_INVALID_PARAMETER), so you know that something went wrong.

Unless you are a pipe or mailslot.

For some reason, if you attempt to issue synchronous I/O on an asynchronous handle to a pipe or mailslot, the I/O subsystem says, "Sure, whatever." I suspect this is somehow related to the confusing no-wait model for pipes.

Long before this point, the basic ground rules for programming kicked in. "Pointers are not NULL unless explicitly permitted otherwise," and the documentation clearly forbids passing NULL for asynchronous handles. The behavior that results from passing invalid parameters is undefined, so you shouldn't be surprised that the results are erratic.

Comments (27)
  1. Ben Voigt says:

    "It just trusts that you're calling it correctly and builds an asynchronous call to pass into the kernel. If you passed a synchronous handle"

    I think sync and async are reversed here?

  2. nathan_works says:

    So many of these customer questions really seem like variations on the famous Babbage quote:

    "On two occasions I have been asked, "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" … I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."

  3. Joshua says:

    Is it just me, or does passing a spurious NULL to the KERNEL sound like something that should always be caught.

    I'm just remembering that everything in kernel needs to be soft realtime, handle constrained memory gracefully, and avoid faulting even in the instance of processes actively attempting to break it.

  4. jader3rd says:

    They should have kept it async only. Now programmers write programs which hang because everything's waiting on I/O.

  5. JoakimA says:

    I'd guess the reason is the ByteOffset parameter of NtReadFile. If you don't pass an overlapped to ReadFile, it sends NULL to the kernel to specify the current file position. And that won't work for an asynchronous file, because such a file doesn't keep track of the current position. Therefore it fails.

    But for pipes, mailslots, and I'd guess serial ports too, you don't need a read position. So I'd guess this is why it succeeds. Could this be it?

    '

  6. Billy ONeal says:

    @Joshua:

    No, the kernel just passes the parameters along to the driver responsible for actually handling the IO. More here -> stackoverflow.com/…/82320

  7. Ishai says:

    > For some reason, if you attempt to issue synchronous I/O on an asynchronous handle to a pipe or mailslot, the I/O subsystem says, "Sure, whatever." I suspect this is somehow related to the confusing no-wait model for pipes.

    For Synchronous I/O the ByteOffset parameter of NtReadFile can be NULL (the kernel keeps the file pointer).  For overlapped I/O the parameter is not optional and is taken from the OVERLAPPED structure.  Mailslots and named pipes do not have the concept of file pointers so ByteOffset is not needed regardless of sync / async nature of the handle.

  8. JS says:

    I wonder why synchronous kernel-mode calls were necessary; couldn't the user-mode layer issue the asynchronous call and then wait?

    My guess is it's a performance optimization.  Fewer trips into kernel mode.  Except that a context switch is noise compared to an I/O…

  9. Anonymous Coward says:

    I think the main reasons overlapped I/O is seldom used are that a) often whatever the program has to do next depends on the result of the operation and b) there is already another more popularised way of doing things asynchronously: threading.

    I am not saying that the latter would be better necessarily, since you'd have to write a thread function, create the thread and then wait for it to complete; for a single operation an overlapped call is much easier. But if you have to do multiple reads and writes in succession, overlapped I/O is pretty much a non-starter. You'd have too much to keep track of and be forced to make one bit of code do two unrelated tasks; much better to simply put all I/O in a separate thread and wait on that.

  10. Cheong says:

    It's easier for old DOS application to port to Windows if they can still assume that they can get synchronous I/O. It's important fact that if you port application, you shouldn't alter application structure at the same time.

    Now people can take it for granted that they can continue to write "functions using I/O calls" in synchronous way.

  11. Anonymous Coward says:

    Cheong, that doesn't make sense. If you're porting old DOS applications, they won't be calling the Win32 file functions but rather CRT functions for example. And you could implement those by an asynchronous call followed by a wait; you don't really need synchronous calls or structure changes.

  12. Cheong says:

    The CRT libraries would also need the synchronous I/O support in Windows API to run in the same way as it were in DOS. So what doesn't make sense?

  13. Drak says:

    Nice, and sadly, .NET (I know, not a .net blog) is going to make the same mistake again.

    "As it turns out, developers vastly prefer synchronous reads. Writing asynchronous code is hard. So the kernel folks relented and said, "Okay, we'll have a way for you to specify at creation time whether you want a handle to be synchronous or asynchronous. And since lazy people prefer synchronous I/O, we'll make synchronous I/O the default, so that lazy people can keep being lazy.""

    TechDays: in the future everything in .Net will be asynchronous!

    Do we never learn from the past?

  14. @cheong00 says:

    "The CRT libraries would also need the synchronous I/O support in Windows API to run in the same way as it were in DOS."

    No, the CRT could have used an OVERLAPPED structure per FILE, using it in every fread/fwrite call and waiting then on the event. Event maintaining the file pointer can be simulated at this level. Because open(), read(), write() etc are not native system calls on Windows, you could also implement the wrapping inside this functions.

  15. Cheong says:

    @@cheong00: That would mean different implementation of CRT libraries for Windows (remember, other then Microsoft, Borland and a few more vendors are supplying C/C++ or other programming language libraries at that time) would possibly have different behaviour on how the I/O calls returns "for synchronous operations". That's not good.

    I'd perfer simple operation map to single API and complex operation map to multiple APIs, over simple operation map to multiple APIs and complex operation map to single API.

  16. ulric says:

    arguing for the sake of arguing. each compiler version already has its own implementation of the C runtime library.  and the fread behavior could not have come out different, it's either sync or not.  stdio is  already a totally different underlying implementIon on win16, win32, than on unix, and on different compilers, and everything works fine.

  17. SimonRev says:

    Surprised no one has mentioned that in Metro at least, all IO is asynchronous.  Looks like that bullet has finally been bitten.  Of course we had to wait 20 years before programming languages has constructs that made combining asynchronous I/O with the need for sequential I/O operations passably tolerable to the programmer.

  18. Killer{R} says:

    2JoakimA: confirm.

    A bit playing with NtReadFile/NtWriteFile shows that Io R/W fails on 'async' handle when no ByteOffset value passed. Its obviously why, but interesting point that NtWriteFile also fails even if file was opened with FILE_APPEND_DATA access flag set, that effectively makes current pos == current size for write op and it could work successfully. Funiest thing is that NtWriteFile requires but ignores ByteOffset when file opened with FILE_APPEND_DATA :) So, some 'if' in kernel's code written not in the best place…

  19. Cheong says:

    @ulric: Yes, it's pretty pointless for arguing that because no system at that time doesn't provide support to synchronous I/O on OS level, therefore don't have to implement synchronous I/O using asynchronous I/O themselves in their own library.

  20. Medinoc says:

    Re "basic ground rules": I just noticed the "basic ground rules" for COM are in conflict with the traditional implementation of AddRef/Release: The "minimum maximum count" rule requires an unsigned 32-bit integer, but InterlockedIncrement() works on signed ones…

  21. @cheong00 says:

    The CRT source code for Unix, OS/2, DOS, Windows etc is very different, not only but also because of the different approaches the systems offer to applications: starting processes, handling pipes, unlink files, as just a few examples.

    So the implementation of read() and write() as "issue I/O and wait" is absolutely no big deal, compared to the emulation of fdopen(), dup(), dup2() on systems that do not use integers as "file descriptors", or popen() on systems that do not support pipes nor multi-tasking (like DOS).

  22. Anonymous Coward says:

    @Cheong: And the implementation for the file calls has to be different anyway since even though the Win32 API's can be synchronous, the relevant function calls still have different names, arguments, &c. You're effectively saying ‘if Microsoft used the CRT as Windows API then writing a CRT would be really easy’ and I bet it would be. But a) the Windows API provides functionality not in the CRT but necessary for a lot of software and b) Microsoft has written a CRT implementation and thus has essentially given you what you want.

  23. pete says:

    "As originally designed, Windows NT had a fully asynchronous kernel. There was no such thing as a blocking read. If you wanted a blocking read, you had to issue an asynchronous read (the only kind available), and then block on it. "

    So why is there no asynchronous CreateFile?

  24. Anonymous Coward says:

    Pete, I'm just guessing here, but presumably this is because to be able to do something with it you'd always have to wait on it.

  25. Cheong says:

    @pete: I think there aren't any asynchronous function that return handles. The handles must be valid when being returned or you can't use them.

  26. Joshua says:

    @Medinoc: The signed-ness of an integer is all in your head. It turns out that casting a pointer to an integer of a different signed-ness works on all CPUs that Windows runs on.

  27. Andrew says:

    A favorite Raymondism applies here:  "Appearing to succeed is a valid form of undefined behavior, but it's still undefined."

Comments are closed.