The case of the unexpected ERROR_ACCESS_DENIED when calling MapViewOfFile


A customer was trying to figure out how to use shared memory, but even their simplest program couldn't work. The customer shared their code and asked for help.

The first process creates a named file mapping object backed by the page file. The second process opens the file mapping object by name, and then maps a view of that file mapping object. But the attempt to map the view always fails with ERROR_ACCESS_DENIED. The file mapping object was created by the first process as read/write, and it was opened by the second process as read/write. The two processes are running in the same session as the same user. And yet, the second process can't get access. What's wrong?

To simplify presentation, error checking has been deleted. Instead, we will describe what happened with comments.

// code in italics is wrong
//
// Program 1

#include <windows.h>

int main(int, char**)
{
 // This succeeds with a non-null handle.
 HANDLE fileMapping = CreateFileMapping(
    INVALID_HANDLE_VALUE, // backed by page file
    nullptr,              // default security
    PAGE_READWRITE,       // read-write access
    0,                    // high part of size
    65536,                // low part of size
    L"Local\\FileMappingTest"); // name

 // This succeeds with a non-null pointer.
 void* view = MapViewOfFile(
    fileMapping,
    FILE_MAP_READ | FILE_MAP_WRITE, // desired access
    0, 0,                   // file offset zero
    0);                     // map the whole thing

  Sleep(5000); // pause to let user run second process

  UnmapViewOfFile(view);
  CloseHandle(fileMapping);
 
  return 0;
}

// Program 2
#include <windows.h>

int main(int, char**)
{
 // This succeeds with a non-null handle.
 HANDLE fileMapping = OpenFileMapping(
    PAGE_READWRITE,       // read-write access
    FALSE,                // don't inherit this handle
    L"Local\\FileMappingTest"); // name

 // This fails with a null pointer.
 // GetLastError() returns ERROR_ACCESS_DENIED.
 void* view = MapViewOfFile(
    fileMapping,
    FILE_MAP_READ | FILE_MAP_WRITE, // desired access
    0, 0,                   // file offset zero
    0);                     // map the whole thing

  UnmapViewOfFile(view);
  CloseHandle(fileMapping);
 
  return 0;
}

The customer added that the second process successfully opened the file mapping object, so presumably the handle does have read/write access. Otherwise, the Open­File­Mapping would have failed with ERROR_ACCESS_DENIED right away, rather than waiting for the Map­View­Of­File.

Study these programs and see if you can find the problem.

(Time passes.)

The problem is that the first parameter to Open­File­Mapping is not supposed to be a PAGE_* value. It's supposed to be a FILE_MAP_* value. This is easily overlooked because you are tempted to just do a copy/paste of the Create­File­Mapping call's parameters, and just delete the parameters related specifically to creation, like file size and security descriptor.

However, it is a common¹ pattern that Create functions return a handle with full access and do not have an explicit access mask parameter, whereas Open functions accept an access mask parameter that controls what level of access the returned handle has.

The numeric value of PAGE_READ­WRITE is 4, which happens to match the numeric value of FILE_MAP_READ. Therefore, the second program successfully opened the file mapping for read, but when it tried to map it for read and write, it got ERROR_ACCESS_DENIED because it's trying to obtain a mapping for writing, even though the mapping was opened only for read.

This is one of the nasty pitfalls of using plain old integers for flags. There's no type safety: Integers look the same.

¹ Note that the pattern is common but not not universal. The most notable exception is Create­File, which takes an explicit access mask. But if you think about it some more, Create­File is an open-like function, because if the file already exists, Create­File opens a handle to it, and it uses the requested access mask to evaluate whether your attempt to open that handle will succeed.

Comments (13)
  1. Darran Rowe says:

    I suppose RegCreateKeyEx is also an exception to that pattern too. It works in a similar way to CreateFile.

  2. pc says:

    I was hoping for a reference to an easy way to prevent these kind of errors. I generally work in the Java world nowadays, where there are still some “old-fashioned” APIs like this, but modern IDEs use annotations on such things and show warnings when using the wrong type, even when it’s just using an int and would compile fine. Is there still nothing similar in the C/C++ Windows API world? Some sort of compiler flag or #define or something to detect these kinds of problems?

    If I were to start work on a brand-new Windows application, is there a Raymond-approved template I should start from of symbols to define and files to include to use “current” development instead of all the defaults that are in there for backwards compatibility?

    1. Zan Lynx' says:

      Well yes. C++11 has strongly typed enums. There’s some good information here http://stackoverflow.com/questions/12581064/enum-vs-strongly-typed-enum

      But fixing this in the Windows API is not possible because of the lack of a time machine. In theory Microsoft could produce yet another API wrapper. But would anyone use it?

      1. Karellen says:

        The “problem” with C++ enums is that they’re mutually exclusive. You can’t use them for bitwise flags that are supposed to be ORd together.

        1. smf says:

          It is possible to use them for flags as well, a standard for the boiler plate code would be good (or an extension to c++ so it’s not even necessary).

          https://github.com/jweyrich/cxx-boilerplate/blob/master/include/cxxboiler/enum_flags.h

          1. Karellen says:

            That particular implementation doesn’t help, as it (as far as I can tell) uses “operator int() const { return i; }” to return the value back to the rest of the application as an “int”… which completely defeats the goal of using an enum to distinguish between PAGE_* values and FILE_MAP_* values.

            I’m also very suspect of it implementing operator+()/operator-()/operator++()/operator–(). Those operations make no sense in terms of switching groups of bits on and off.

            It’s also a bad overload of the enum abstraction, as the point of an enum is to enumerate all possible (and only those possible) values. If “a | b” is an allowed value, the enumeration would need to contain a member with that value for it to make sense as an enumeration. At which point you might as well use that member, rather than ORing two other members together. A separate abstraction (“bitflags”?) might make sense, but given C++’s aversion to introducing new keywords so as to not break existing code, that seems unlikely. (But “nullptr” made it, so… Or, “bitor enum” maybe – would make about as much sense as some of the historic overloads for “static” or “auto”. :-)

          2. smf says:

            “That particular implementation doesn’t help, as it (as far as I can tell) uses “operator int() const { return i; }” to return the value back to the rest of the application as an “int”… which completely defeats the goal of using an enum to distinguish between PAGE_* values and FILE_MAP_* values.”

            Any type safe solution would require you to change the API calls to take an enum instead of an int, Then int() won’t be called.

            “It’s also a bad overload of the enum abstraction, as the point of an enum is to enumerate all possible (and only those possible) values.”

            No, that isn’t the point of an enum. However I agree that a type that contains bit fields (including enum bit fields) is a better solution.

        2. voo says:

          Providing an API that is a collection of enum values with simple ways to query and set value is pretty trivial. In Java there is EnumSet (https://docs.oracle.com/javase/7/docs/api/java/util/EnumSet.html) which does exactly that.

          I don’t really see where compiler support would be necessary except to avoid some small syntactic sugar.

  3. Myria says:

    At the system call interface level, FILE_MAP_READ (SECTION_MAP_READ) is just an access mask bit like SYNCHRONIZE. So yes, OpenFileMapping has an access mask that you want to use for the handle.

    However, why MapViewOfFile takes FILE_MAP_* instead of PAGE_* is a mystery to me: NtMapViewOfSection takes PAGE_* flags. So MapViewOfFile is internally just translating the flags rather pointlessly.

    Also, Raymond’s footnote only applies to Win32, not the NT API. NtCreateSection takes both SECTION_MAP_* and PAGE_* flags. The SECTION_MAP_* flags are the access mask for the handle returned by NtCreateSection.

    1. I guess you’d better tell the Win32 team that it’s time to dust off that time machine.

  4. Mixing flags is an issue a static analyzer could probably find. It would require the function to be annotated with the flags it accepts.

  5. Darran Rowe says:

    Well, this is a bit of a grey area. First, you can fill in all of the blanks if you really want, so:
    enum class rw {none, read, write, read_write};
    This means that you can if you want. The other thing is that you can also write bitwise operator functions for this type of enum. This allows you to use enums as bitmasks.
    The problem here is the first bit. Writing out all states of an enum is a pain, so if you are not planning on reading the value directly then you can skip the writing out of all the states and have just the bitwise operators.
    But I understand though, I would be more than tempted to use a class type to do this instead.

    1. Darran Rowe says:

      Meh, this was meant to be a reply to Karellen’s post further up, but somehow it got displaced down here.

Comments are closed.

Skip to main content