Memory-mapped files and how they work


A key Windows facility that’s been available since NT shipped is the support of memory-mapped files.  A memory-mapped file is a file that has been mapped (i.e., not copied) into virtual memory such that it looks as though it has been loaded into memory.  Rather than actually being copied into virtual memory, a range of virtual memory addresses are simply marked off for use by the file.  You (or Windows) can then access the file as though it were memory, freely changing parts of it without having to make file I/O calls.  Windows transparently loads parts of the file into physical memory as you access them.  You don’t have to concern yourself with which parts are and are not in memory at any given time.

 

As each page of the file is accessed and copied into memory so that the CPU can access it, it bypasses the paging file.  In this sense, the file takes the place of the paging file as the backing storage for the particular range of virtual memory addresses into which it has been mapped.  Typically, the backing storage for virtual memory is the system paging file.  But with memory-mapped files, things change.  The files themselves act as virtual extensions of the paging file and serve as the backing storage for their associated virtual memory address range for unmodified pages.

 

How are memory-mapped files used?  Most commonly by Windows itself.  Every time a binary image (an .EXE or .DLL, for example) is loaded into memory, it is actually loaded there as a memory-mapped file.  The DLLs you see loaded into a process’s virtual memory address space are actually mapped there as memory-mapped files by Windows. 

 

As I’ve mentioned, memory-mapped files are also used to process files as though they were data.  If you’d rather search a file or perform minor in-place modifications of it using simple pointer dereferencing and value assignment rather than invoking file I/O operations, you can certainly do so.  Have a look at the MapViewOfFile() Win32 API for some basic info and some pointers on how to get going with this.  Something to be careful of, though, is depleting virtual memory.  When mapping large files into virtual memory to perform I/O on them, be cognizant of the fact that every address you burn in virtual memory is another than can’t be used by your app.  It’s usually more efficient to use regular file I/O routines to perform read/write operations on large files.

 

I’ll continue tomorrow with a discussion of how this relates to SQL Server.


Comments (8)

  1. Calin Iaru says:

    Hi,

    it would be great to know more on this topic. I’m also interested in how SQL Server behaves on a clustered environment. There are not so many topics on this, so please tell us (me) more. And what’s the latest on TPC tests?

    Best regards,

    Calin

  2. Neil Weicher says:

    Memory mapped files, in general, cannot be used with databases (except maybe readonly databases) as they would be highly prone to corruption in the event of a system crash

  3. Ris says:

    Hi,

    I would also like some more information about the inner workings of this system.

    I maintain a data-intensive application for Windows Mobile platforms (GPS navigation software which depends on mass quantities of geographic data stored in a proprietary data file format) and we use memory mapped files to "pretend" to load this data into RAM, allowing the OS to handle the resource management.  The problem for me is that there is not enough RAM hardware available to load over a 1GB of data for searching and processing into real hardware RAM all at the same time.  If we are forced to rely on File I/O operations, then we must create our own caching system in which we open/seek/read only selected parts of selected data files as-needed (which requires custom logic to define "as-needed" and equally importantly "no longer needed") in order to acquire access to 1GB+ of data with only 32MB of RAM available.

    What I find is that if I memory-map a file that is larger than 5MB, let’s say 16MB for an example (such as our road map of Los Angeles) then the performance of the system grinds to a very slow unresponsive halt.  However, if I map the same quantity of data as 4 separate files each under 5MB in size (in this case NE, SE, NW, SW quadrant maps of LA, each 4MB in size) then the performance is fine, even if I’m randomly accessing all of this data.

    It seems to me that I have mapped the same quantity of data into virtual memory, which should use the same number of virtual addresses.  That is, I have the same number of addresses burned in virtual memory either way.  Right?  So why then is the large file problematic, but the equivalent amount of data spread across more smaller files does not reach this slow performance limitation?

    What I assume is that there is a sort of cache in real hardware RAM at the OS level which handles the paging file and does actually COPY the virtually mapped memory data from slower storage memory into faster RAM, so that I can continue to access this data at the speed of RAM rather than the slower speed of File I/O.  However, it seems that something related to using larger files results in clearing the cache, which results in cache misses, which are bad and make things slow.  But my understanding of this is lacking in detail and full of assumptions and speculation.  Could you talk a bit more about what happens in this case, and why the same quantity of data spread throughout multiple memory-mapped files does NOT cause slow performance, but putting the same quantity of data all in a single, larger memory-mapped file DOES cause slow performance?

    Given this experience with large files causing performance problems, I have broken my >1GB of data into about two thousand files all smaller than 5MB in size each.  However, if I attempt to open too many of these small files, the operations to CreateFileForMapping, and MapViewOfFile, etc begin to fail as if I am out of memory (off-hand, I’m not sure exactly which API fails – there is a series of about 6 APIs you must call to accomplish the task)  The memory control panel on the device indicates >20MB of available RAM for programs, and the storage card has enough space to store all this data, so I’m clearly not out of memory in reality.  Is this because I have run out of virtual addresses and the virtual address table has become full?  

    I feel if I understood more about the inner workings of this system, I might be able to accomodate its limitations and continue to work within this framework.  The alternative is going to be the creation of a customized caching logic tailored specifically for my program, which is going to be a lot of work that I would rather avoid if I can simply make some changes to the way I use memory mapped files instead.

    I appreciate you offering a website like this because technical information at this level is generally difficult to find.  Thank you.

    Best Regards,

    Ris

  4. Vincent says:

    Ris,  I did a quick test of MMF performance by mapping to a 261mb file, and accessing bytes at every 100th KB increment forward and back.  The first run took 4.5 seconds to complete.  Subsequent runs took 0 seconds to complete probably due to OS caching.

    I did the test on Windows 2000, not Windows Mobile, and I have 512mb of RAM rather than 32MB so I upped the file size to 1GB, and reran the test.  This took 17.6 seconds to complete.  I think this is reasonable: the time it took to test a 1GB file is about 4 times what it took for a 261mb file.

    Where is the slow performance coming from — when you access an address that is generating a page fault, or some place else ?  How random is your data access ?  On W2K, each page fault causes a read of at least 4kb from file to RAM.  If your data access is causing a lot of page faults and you are running low on RAM so pages are being swapped out and in, it will slow performance down.

    As for your problem with opening 2000 files and running out of memory, you’re probably running out of handles.   The memory used to store handles are quite limited and it is very possible that they will run out on a small device.  I’ve tried opening as many handles as I can on a 4gb RAM machine and the program ran out of memory way before 4gb was reached (or even 2gb).  I’m no expert on this, but I suspect that the bulk of what it takes to store a handle may reside in system memory rather than user process memory, and this system memory is quite limited.

    Vincent

    2006/7/16

  5. Ris says:

    Hi Vincent,

    Thanks for the extensive testing, it sounds like you went to a bit of trouble to look into this, and I appreciate the extra info.  Unfortunately, I am no longer involved with that Mapopolis project, so I can’t access the code anymore for reference purposes or experimentation.  However, I am still interested in my own on-going learning, so I’m happy to keep up a discussion about this.

    To answer your questions, I never figured out exactly where the slow performance was coming from, i.e. whether I was generating a page fault or what else it might have been or where.  Though I always suspected a paging or caching problem.

    My memory access was more sequential than random.  I’m not really sure how I would report a measurement of exactly how random, but I can describe in more detail how it worked.

    Consider the case of creating a file for each of the 88 counties in Ohio.  Each of these files would be 1.5MB to 4.5MB in size.  A window would show a portion of one to four counties at a time (usually just one or two at a time).  When painting the window, I would perform a linear search on an array of the 88 counties to find which counties were in the view area (all indexing data in RAM, a buffer of structs allocated with malloc)  This would yield results of which memory-mapped files had to be accessed in order to paint the screen.  I would then perform a linear search of those memory-mapped files that were at least partially in the view area in order to find the data elements that had to be drawn to the window.

    Now consider the alternate case of showing one densely populated region such as the monolithic 16MB county map of Los Angeles.  The first linear search to find which county is applicable is instantaneous because there is only the one county available.  The second linear search to find which part of the 16MB file should be shown on-screen was painfully slow.  Whereas the case of small Ohio county files would perform instantaneously, the case of accessing the single large file with these linear searches would presumably overflow the cache repeatedly, alternately caching off-screen data before caching the on-screen data, and then repeating the linear search for the next paint, thus accessing the off-screen data, which flushed the on-screen data from the cache and then back in again because of the linear nature of the data access.

    It would be interesting to generate some test projects as you have done, to experiment with randomly generated files of different sizes.  I might try that one of these days just for a personal learning experience.

    To compare with some of your tests, and give some more numbers… On a Windows Mobile system with 32MB, I was able to load up to about 500MB of data separated into over a hundred files all smaller than 5MB each, before the performance started to decay.  At that point, the performance hit came mostly from iterating a monolithic array of several hundred data structures containing the file handles and some basic indexing data describing the file whose handle was stored in that structure.  These files would be stored on an SD card (what is that, NAND flash?) and I’m not sure how that compares to the speed of a hard drive used for virtual memory in a Windows 2000 machine.  These files would be opened for read-only access.

    — Ris