All I/O on a synchronous file handle is serialized; that’s why it’s called a synchronous file handle

File handles are synchronous by default. If you want asynchronous file handles, you need to pass the FILE_FLAG_OVERLAPPED flag when you create the handle. And all operations on a synchronous file handle are serialized.

You'd think this was a simple and obvious rule, but "Someone" finds it "very surprising that operations can block which only handle file metadata."

Imagine if synchronous file handles were not serialized for metadata operations. First of all, it means that the documentation for synchronous file handles suddenly got a lot more complicated. "Some operations on synchronous file handles are serialized, but not others. Specifically, operations on file contents are synchronized, but operations on file metadata are not synchronized."

Now things get weird.

For example, the size of a file is metadata. Allowing file size operations to bypass serialization means that if you issue a Write­File operation, and then on another thread call Get­File­Size, you can get a size that is neither the old size nor the new size. Maybe that doesn't bother you.

Okay, how about the Device­Io­Control function? Does that operate on file contents or file metadata? Your initial reaction might be that Device­Io­Control is obviously a metadata operation. For example, the object ID you get from FSCTL_GET_OBJECT_ID has nothing to do with the file contents. On the other hand, some I/O control codes do affect file contents. FSCTL_SET_ZERO_DATA zeroes out chunks of a sparse file, FSCTL_FILE_LEVEL_TRIM tells the underlying storage that it may (but is not required to) throw away the current contents of a section of a file. Since some I/O control codes affect file contents and some don't, you would have to say that the Device­Io­Control function on a synchronous file handle is sometimes serialized and sometimes not. It's not very reassuring to read documentation that goes something like "If the file handle is a synchronous file handle, the I/O control operation might be serialized, or it might not."

Recall that all I/O in the kernel internally follows the asynchronous programming model. Synchronous file handles are a convenience provided by the kernel which converts operations on synchronous handles into the equivalent asynchronous operation, followed by a wait for the operation to complete; and all of these operations are serialized.

Since drivers are allowed to make up custom I/O control codes, the I/O subsystem cannot know for certain whether a particular control code affects file contents or not. It wouldn't know whether any particular control code issued on a synchronous file handle should be converted to a synchronous operation or allowed to proceed asynchronously.

So now you're in an even more confused situation, where the kernel doesn't even know whether it should serialize a control operation or not. What would the documentation say now? "Actually, the kernel doesn't know whether the operation should be serialized, so it just flips a coin. Heads, the operation is serialized. Tails, it isn't. Do you feel lucky, punk?"

So the kernel chooses a very simple algorithm: All operations are serialized. It doesn't care if an obvious file contents operation, a subtle file contents operation, a hidden file contents operation, or not a file contents operation at all.

Sometimes the best way to solve a problem is to stop trying to be clever and just focus on predictability and reducing complexity.

If you want to perform an operation that is not serialized, you can just create a second handle to the same file and perform the operation on the second handle. (The Re­Open­File function may come in handy.) Whether a file handle is synchronous or asynchronous is a property of the handle, not of the underlying file. There can be five handles to the same file, some synchronous and some asynchronous. Operations on the synchronous handles are serialized with respect to the file handle; operations on the asynchronous handles are not.

And with the decision to take the simple approach and serialize all accesses to a synchronous file handle, the kernel can wait on the hFile itself to determine when the I/O has completed, thereby saving it from having to create a temporary event for every single I/O operation.

Comments (30)
  1. Joshua says:

    I still say there's no excuse for GetFileType blocking. That should not even be I/O, just looking up a config byte in the kernel structures that was set when the file was opened. It can't even change while the handle is open.

    ["All operations on synchronous handles are synchronous, with one exception." Now you've opened the door. Tomorrow, there will be two exceptions. -Raymond]
  2. alegr1 says:

    Small clarification. The scope of I/O serialization is not simply a handle, but all I/O for a given FILE_OBJECT. Additional handles created by DuplicateHandle are included to the serialization scope.

  3. waleri says:

    I don't understand. Does that mean, that if two threads try to read from a file they will wait for each other? Also, does that mean, that if one uses async mode, the I/O requests will wait for each other too?

    [If you attempt to perform asynchronous I/O on a synchronous handle, it is performed synchronously. Synchronous-ness is a property of the handle (or, more accurately, file object), not the I/O request. -Raymond]
  4. Gabe says:

    Considering that the post "Someone" commented on was about different processes sharing a synchronous file (stdin in that case), and different processes clearly have different handles, it is obviously true that "synchronous or asynchronous" is not a property of the handle, but of some underlying object to which the handle refers (as alegr1 points out).

    So if DuplicateHandle returns a reference to the same FILE_OBJECT, that leaves one to wonder how ReOpenFile works. Does it return a reference to the same object, just with different options? Or does it copy the FILE_OBJECT and change some options? Or something else entirely?

    [It creates a new FILE_OBJECT that refers to the same file as the original. -Raymond]
  5. @waleri:

    The reason why CreateFile has the file share flags, and the LockFile(Ex) functions exist is because there is no synchronisation between two separate handles.

    So if you want any kind of guarantee that the carpet isn't pulled out from under you, you would either open the file with no file sharing set to not allow writing (either use FILE_SHARE_READ or 0) or you lock the region of bytes that you want to work on with LockFile(Ex) before you start, and then unlock them after you have complete.

    This guarantees that another I/O operation on another handle can't interfere with what you are doing.

  6. I should have read that again before posting.

    "you would either open the file with no file sharing set to not allow writing"

    Two different thoughts at the same time. That should be

    "you would either open the file with sharing set to not allow writing"

  7. Douglas says:

    "There can be five handles to the same file"

    I've never heard that limit before. Is that system-wide, per process, or per thread?

    [The context of that sentence was intended to be "You can, for example, create five handles to the same file…" It was not intended to mean "You can create at most five handles to the same file." -Raymond]
  8. @Douglas:

    Handles should be limited only by available memory. Or at least that is what the MSDN says.

  9. Joshua says:

    ["All operations on synchronous handles are synchronous, with one exception." Now you've opened the door. Tomorrow, there will be two exceptions. -Raymond]

    You already have 3 of them. CloseHandle, DuplicateHandle, ReOpenFile. Oh wait CloseHandle is blocked by somebody else's IO only on the single processor kernel.

    Another way of looking at it is NT4 and 2000 claimed POSIX compliance. This method would be the implementation of isatty(), which must not block.

  10. @Joshua:

    None of those three are what you claim them to be though.

    They would be meta meta operations.

  11. 640k says:

    5 open handles per file ought to be enough for everyone.

    @Crescens2k: Handles should be limited only by available memory. Or at least that is what the MSDN says.

    Where exactly? NT dies with 10k+ open handles. 32-bit and 64-bit. ram doesn't matter.

    [Not sure where you got that from. Mark Russinovich discovered that the actual limit is 16 million handles per process. -Raymond]
  12. Mark says:

    640k: [[Citation needed]]. As far as I know, the only limit is 16M.

  13. @640k:…/ms724469(v=vs.85).aspx

    Yes, that does talk about the system limit, but please remember the original context (handles open to a file).

  14. ErikF says:

    @Joshua: NT4/2000 only claimed POSIX compliance within the POSIX subsystem; Win64/32 still behaves like Windows (case is still folded for filenames, for example.) The kernel can behave any way it likes, as long as it can satisfy the requirements of the subsystems.

  15. alegr1 says:


    I see 39000 handles in my x86 laptop.

    Now, somebody tell me please, why does Outlook have to have 5K+ handles? and 1000+ GDI objects?

  16. alegr1 says:


    "You already have 3 of them. CloseHandle, DuplicateHandle, ReOpenFile. Oh wait CloseHandle is blocked by somebody else's IO only on the single processor kernel."

    You don't know what you're talking about. Number of processors doesn't affect CloseHandle blocking or not.

    DuplicateHandle doesn't have to block, because it doesn't affect the FILE_OBJECT state, or query its state. It doesn't issue any IRP.

    ReOpenFile may or may not have to block, depending on how it's implemented.

    CloseHandle HAS to block, only if it's the last handle, and there is synchronous I/O in progress (which can only happen on a different processor). When the last handle is closed, a IRP_MJ_CLEANUP request is issued for the given FILE_OBJECT. It have to be serialized with other I/O.

  17. alegr1 says:

    > (which can only happen on a different processor).

    *different thread, obviously

  18. .dan.g. says:

    Raymond, could we have a citation for the "Do you feel lucky, punk?"?

    I know where it comes from but there may be some youngsters who do not…

    [Fortunately, youngsters know how to use a search engine. -Raymond]

    Dirty Harry

    Harry Callahan: I know what you're thinking, punk. You're thinking "did he fire six shots or only five?" Now to tell you the truth I forgot myself in all this excitement. But being this is a .44 Magnum, the most powerful handgun in the world and will blow you head clean off, you've gotta ask yourself a question: "Do I feel lucky?" Well, do ya, punk?

  20. Silly says:

    > I know where it comes from but there may be some *young punks* who do not…


  21. j b says:

    @alegr1 (and others),

    This discussion about number of hsandles and other system objects reminds me of one developer back in the 1990s (in the days when NetNews was the place to air frustrations and seek advice) complaining that Windows crashed after just a few thousand windows were opened.

    He was developing a text processor, displaying every word in a separate window (with no border or any sort of decoration). That way he could justify the text by simply adjusting the window position for each word window, without doing any redrawing. To this developer, that was the most obvious and natural way of doing things, like, 'Wouldn't everybody do it that way? Why not?'

  22. Someone confused says:

    A few Questions:

    1 What state is shared accross duplicated handles? I'm especially concerned about file-pointer and IO-Aborts.

    2 How would you open an asynchronous handle for writing to a file opened with exclusive access, given a synchronous handle (having no/read/all access)? Wouldn't ReOpenFile be blocked?

    3 Is there a safe non-blocking way to do point 2?

    [File pointer position is a property of the file object, not the handle. (Easily verified by experiment.) I don't know about cancellation. And yes, if you open the file in exclusive mode, then you have exclusive access. That's sort of the point. -Raymond]
  23. waleri says:

    [If you attempt to perform asynchronous I/O on a synchronous handle, it is performed synchronously. Synchronous-ness is a property of the handle (or, more accurately, file object), not the I/O request. -Raymond]

    Thank you for the clarification. I asked, because it is not very clear who performs the conversion between synchronous and asynchronous. For example an API can simply internally allocate an OVERLAPPED and a *per thread* event object then wait on that event. In this case the thread still would perform synchronously yet other threads won't be blocked.

  24. alegr1 says:

    CancelIo(hFile) walks the list of IRPs (I/O Request Packets) for the current thread and calls IoCancelIrp for every IRP for this FILE_OBJECT. The IRP doesn't keep a corresponding HANDLE, so it's impossible to cancel on a separate handle basis.

    CancelIoEx walks all threads, and all IRPs in their pending lists, then matches them against the pOverlapped. Same as CancelIo, it has no means to see which one of duplicated handles was used to initiate the IO.

    Even if ReOpenFile blocks on a synchronous original handle (which requires verification), it won't block or will only block for short time, if all possible I/O on it complete promptly.

  25. alegr1 says:

    One example of per-handle state is file region locks. A duplicated handle doesn't give access to the region locked by another handle to the same FILE_OBJECT. At least it's what LockFile documentation says for inherited handles, without clarification for duplicated handles.

  26. Someone confused says:

    @Raymond: So your advice to reopen as (a)synchronous becomes impossible to follow, much of the time. Unless there's another way to get a handle with changed (a)synchronous mode option we don't know about yet. I acknowledge that generally evading the exclusive option is bad, but if that's the only way to get it done, maybe a Flag to ReOpenFile allowing all access and  same or tighter sharing modes available as if the passed FILE_OBJECT didnt exist yet…

    @alegr1 re CancelIoEx: I'm sure its restricted to the calling process, otherwise it would be a glaring security hole.

    @alegr1 re ReOpenFile: If it blocks on a synchronous handle at all, it could block 'forever': See "Don't let more than one process try to read from stdin at the same time" linked by Raymond as "create a second handle…".

    @alegr1 re file region locks: Looks like its keyed to process+FILE_OBJECT, not any kind of per handle data.

  27. Deduplicator says:

    I'm not sure: Is there any state/data/permissions in Windows ever associated with a specific handle instead of with the referenced Kernel object?

    As far as i can see, the api reference writers are generally not sure if any state/privilige/data is associated with the underlying object (the file itself), the kernel object (refered to as FILE_OBJECT here) or the handle…

    Its really frustrating when the one and only authoritative source is confused about the concepts involved.

  28. alegr1 says:


    Access mask is associated with a handle. This is why DuplicateHandle allows to create a handle with different access mask.

  29. Deduplicator says:

    Thanks, I should have recalled the access_mask.

    Should also have remembered bInheritable aka Unix O_CLOEXEC.

    Any more coming to mind?

  30. Hm says:


    All I/O on a synchronous file handle is serialized; that's why it's called a synchronous file handle


    But the sane expectation is that the meaning of "synchronous" is bound to the thread making an I/O call over the handle: The thread makes a I/O call, the kernel creates the I/O request and waits for its completition on behalf of the thread making the call, to make the call synchronous.

    The layer processing the I/O requests can serialize them when it makes sense, for example, by serving them in FIFO manner to read or write data from a byte stream that does not support a file position. If the byte stream supports file positions, the I/O layer is free to process the I/O requests in any order.

Comments are closed.