You can use an OVERLAPPED structure with synchronous I/O, too


Even if you didn't open a file with FILE_FLAG_OVERLAPPED, you can still use the OVERLAPPED structure when you issue reads and writes. Mind you, the I/O will still complete synchronously, but you can take advantage of the other stuff that OVERLAPPED has to offer.

Specifically, you can take advantage of the Offset and OffsetHigh members to issue the I/O against a file location different from the current file pointer. (This is a file pointer in the sense of Set­File­Pointer and not in the sense of the C runtime FILE*.) If your program does a lot of reads and writes to random locations in a file, using the synchronous OVERLAPPED structure saves you a call to Set­File­Pointer at each I/O.

Let's illustrate this by writing some code to walk through a file format that contains a lot of offsets to other parts of the file: The ICO file format. First, the old-fashioned way:

#define UNICODE
#define _UNICODE
#include <windows.h>

#include <pshpack1.h>
struct ICONDIRHEADER {
    WORD idReserved;
    WORD idType;
    WORD idCount;
};

struct ICONDIRENTRY {
    BYTE bWidth;
    BYTE bHeight;
    BYTE bColorCount;
    BYTE  bReserved;
    WORD  wPlanes;
    WORD  wBitCount;
    DWORD dwBytesInRes;
    DWORD dwImageOffset;
};
#include <poppack.h>

BOOL ReadBufferAt(__in HANDLE hFile,
    __out_bcount(cbBuffer) void *pvBuffer,
    DWORD cbBuffer,
    DWORD64 offset)
{
 LARGE_INTEGER li;
 DWORD cbRead;
 li.QuadPart = offset;
 return SetFilePointerEx(hFile, li, nullptr, FILE_BEGIN) &&
        ReadFile(hFile, pvBuffer, cbBuffer, &cbRead, nullptr) &&
        cbBuffer == cbRead;
}

int __cdecl wmain(int argc, wchar_t **argv)
{
 HANDLE hFile = CreateFile(argv[1], GENERIC_READ,
  FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
  nullptr, OPEN_EXISTING, 0, nullptr);
 if (hFile != INVALID_HANDLE_VALUE) {
  ICONDIRHEADER hdr;
  if (ReadBufferAt(hFile, &hdr, sizeof(hdr), 0) &&
      hdr.idReserved == 0 && hdr.idType == 1) {
   for (UINT uiIcon = 0; uiIcon < hdr.idCount; uiIcon++) {
    ICONDIRENTRY entry;
    if (ReadBufferAt(hFile, &entry, sizeof(entry),
                     sizeof(hdr) + uiIcon * sizeof(entry))) {
     void *pvData = LocalAlloc(LMEM_FIXED, entry.dwBytesInRes);
     if (pvData) {
      if (ReadBufferAt(hFile, pvData,
                       entry.dwBytesInRes, entry.dwImageOffset)) {
       // process one image in the icon
      }
      LocalFree(pvData);
     }
    }
   }
  }
  CloseHandle(hFile);
 }
 return 0;
}

Run this program with the name of an icon file on the command line, and nothing interesting happens because the program doesn't generate any output. But if you step through it, you can see that we start by reading the ICON­DIR­HEADER to verify that it's an icon and determine the number of images. We then loop through the images: For each one, we read the ICON­DIR­ENTRY (specifying the explicit file offset), then read the image data (again, specifying the explicit file offset).

We use the Read­Buffer­At function to read data from the file. For each read, we first call Set­File­Pointer to position the file pointer at the byte we want to read, then call Read­File to read it.

Let's change this program to take advantage of our newfound knowledge:

BOOL ReadBufferAt(__in HANDLE hFile,
    __out_bcount(cbBuffer) void *pvBuffer,
    DWORD cbBuffer,
    DWORD64 offset)
{
 OVERLAPPED o = { 0 };
 o.Offset = static_cast<DWORD>(offset);
 o.OffsetHigh = static_cast<DWORD>(offset >> 32);
 DWORD cbRead;
 return ReadFile(hFile, pvBuffer, cbBuffer, &cbRead, &o) &&
        cbBuffer == cbRead;
}

We merge the Set­File­Pointer call into the Read­File by specifying the desired byte offset in the optional OVERLAPPED structure. The I/O will still complete synchronously (since we opened the handle synchronously), but we saved ourselves the hassle of having to call two functions when it could be done with just one.

Comments (19)
  1. Skyborne says:

    The "Set:FilePointer" typo in the last paragraph catches my eye: are you using the Dvorak keyboard layout?  Mostly-idle curiosity from a Dvorak user.

    [If you do a View-Source, you'll see that it was supposed to be a semicolon. Shift key slip. (Well you can't see it now because I fixed it.) -Raymond]
  2. Relevant MSDN snippet: "Considerations for working with synchronous file handles:

    • If lpOverlapped is not NULL, the read operation starts at the offset that is specified in the OVERLAPPED structure and ReadFile does not return until the read operation is complete. The system updates the OVERLAPPED offset before ReadFile returns."

    Interesting… I learned something new!

  3. Koro says:

    Unrelated, but personally when I have to do random access to lots of offsets in a file, I will just memory-map it and do pointer arithmethic, and let the kernel do more efficient caching than having to do lots of kernel calls and copies around.

  4. Joshua says:

    @Koro: Every try to memory map a file bigger than 2GB on a 32 bit processor?

  5. Paul says:

    @Joshua – Have you ever actually memory mapped a file?  You don't have to map in the whole thing at one time.  CreateFileMapping creates a section object, and them MapViewOfFile actually maps in some range of the file specified by an offset and number of bytes.

  6. Igor says:

    Yes, but then you're back in the seeking/remapping business (if you need to cover the whole file).

    I'm curious about the "let the kernel do more efficient caching" thing… is it necessarily the case?

  7. alegr1 says:

    This reading (and writing) in small random pieces is all fun and games, until your file is on a USB drive, or compressed. And then you get the painfully slow JPG file "turn 90 degree" operation in windows image viewer because of that.

  8. alegr1 says:

    Another advantage of using OVERLAPPED is that the operation becomes thread-safe.

  9. Neil says:

    The bit-twiddling reminds me that I still hate the MAKELONG and HIWORD macros; my 16-bit compiler at the time implemented them using function calls. Oops.

    I can't imagine why the OVERLAPPED structure didn't use a LARGE_INTEGER structure.

  10. Gabe says:

    Neil: Back when the OVERLAPPED structure was being conceived, your average HD was probably 80MB. Creating a 4GB file would require tens of thousands of dollars worth of SCSI disk array. I can imagine that the average programmer would never be expected to even see such a file, let alone have to write a program to manipulate one. Obviously they didn't expect that 20+ years later we would all be downloading 4GB video files from our smartphones and such.

    Anyway, using a LARGE_INTEGER would make using the structure ever so slightly more complicated. I can easily imagine that they wanted to make it just a bit simpler for the 99.999% of programmers who would never have to worry about a 4GB file, while making it ever so slightly more work for the 0.001% of us who do have to be concerned about it.

  11. JoakimA says:

    I tried to post a comment the other day and now, but it's not getting through. Is there some kind of automatic spam filter or am I unlucky to be in a blacklisted IP range? Or are the comments just moderated?

  12. JoakimA says:

    There is one thing regarding files opened as overlapped I've been wondering:

    The ReadFile documentation says about the lpNumberOfBytesRead arameter

    "Use NULL for this parameter if this is an asynchronous operation to avoid potentially erroneous results.". However, KB article 156932 says:

    "If, on the other hand, an operation is completed immediately, then &NumberOfBytesRead passed into ReadFile is valid for the number of bytes read. In this case, ignore the OVERLAPPED structure passed into ReadFile; do not use it with GetOverlappedResult or WaitForSingleObject."

    But the ReadFile documentation said I was supposed to pass in NULL for asynchronous operations. So what do I do if the operation on the file opened as overlapped finishes immediately? Do I follow the ReadFile documentation and pass in NULL, in which case I'll have to use GetOverlappedResult even though the KB article says I shouldn't? Or do I pass in a pointer as the KB article says and get the "potentially erroneous results" the ReadFile documentation speaks of? Got to love that warning… Also, what happens to the hEvent in OVERLAPPED when completing synchronously?

  13. NeedInfo says:

    Can we please provide some API for telling whether a given HANDLE is open for synchronous IO? You can get this information with the NT native API, but poor mortals outside Microsoft can't figure out the same thing. Given a HANDLE of unspecified origin, then, these people have to use an OVERLAPPED structure, yet prepare for all IO to be done synchronously anyway.

  14. Gabe says:

    NeedInfo: Is there any time when you shouldn't expect any particular I/O to be done synchronously?

  15. Glitchy says:

    From my experience with synchronous use of OVERLAPPED is that its very buggy and glitchy.

    I always get nervous when i see code that use it.

  16. @JoakimA: I was wondering about this same exact thing the other week.  I'd say the documentation and support articles are ambiguous; it is really not clear what one should do in this situation.  You could always decompile or check the .NET Framework source code on this; they probably got it right.

  17. @JoakimA says:

    I believe the canonical implementation reads like this (in Delphi):

    // Returns when <BytesToRead> bytes are read into <Buffer> or when the timer <TimeoutTimer> expires,

    // whatever comes first. The number of bytes read is returned.

    // I/O errors are reported as EOSError exception.

    function TTestObj.Read(TimeoutTimer: TWaitableTimer; out Buffer; BytesToRead: cardinal): cardinal;

    var

     Handles: array [0..1] of THandle;

    begin

     // ReadFile can complete immediately:

     if not Windows.ReadFile(FHandle, Buffer, BytesToRead, Result, @FOverlapped) then begin

    Win32Check(Windows.GetLastError = ERROR_IO_PENDING, &#39;ReadFile&#39;);
    
    // ReadFile will complete asynchronous =&gt; wait for it with a timeout:
    
    Handles[0] := FOverlapped.hEvent;
    
    Handles[1] := TimeoutTimer.Handle;
    
    case Windows.WaitForMultipleObjects(length(Handles), pointer(@Handles), false, INFINITE) of
    
    WAIT_OBJECT_0 + 0:  // ReadFile is complete
    
     &nbsp;{do nothing};
    
    WAIT_OBJECT_0 + 1:  // timeout is expired =&gt; terminate the ReadFile operation:
    
     &nbsp;Win32Check( Windows.CancelIo(FHandle), &#39;CancelIo&#39;);
    
    else
    
     &nbsp;RaiseWin32Error(Windows.GetLastError, &#39;WaitForMultipleObjects&#39;);
    
    end;
    
    // Get the result of the ReadFile operation. ERROR_OPERATION_ABORTED means CancelIO was called, but even in this case,
    
    // we want to report the number of bytes read:
    
    if not Windows.GetOverlappedResult(FHandle, FOverlapped, Result, false) and (Windows.GetLastError &lt;&gt; ERROR_OPERATION_ABORTED) then
    
     &nbsp;Win32Check(false, &#39;GetOverlappedResult/ReadFile&#39;);
    

     end;

    end;

    FHandle and FOverlapped are instance members. Win32Check() raises the EOSError exception when the condition (the first argument) is not true. <Result> is the implicit variable that represents the return value of the function.

    I use this code in a productive application without problems.

  18. JoakimA says:

    I dug into kernel32.dll a little, and the implementation is such that the overlapped struct is where the system service NtReadFile returns bytes read and final status. ReadFile then copies the number of bytes read into its return parameter if supplied and the operation completed synchronously. Otherwise it leaves 0 which it put in at the start. GetOverlappedResult will work regardless of whether the operation completed synchronously or not. This may be an implementation detail though, but this is how it is here in XP SP3 at least.

    @@JoakimA: That's similar to what I've seen in sample code too. Though I think you should set wait=TRUE in your GetOverlappedResult because CancelIO only requests cancellation without waiting for it to finish. GetOverlappedResult may return ERROR_IO_PENDING otherwise, and things will break badly if the overlapped struct and/or buffer goes away before the operation completes.

  19. Yuhong Bao says:

    Gabe: Yea, SetFilePointer was designed the way it was for a similar reason. At least it fared much better than GetDiskFreeSpace.

Comments are closed.

Skip to main content