How can I force memory to be allocated above the 4GB boundary (for debugging purposes) on Windows 7?


A customer discovered that their program has 64-bit-to-32-bit pointer truncation bugs. They are working hard to track down all the places where this happens. On Windows 8 and higher, they can build their program with high entropy VA to make ASLR be super-aggressive about putting heap and stack memory above the 4GB boundary. However, they still have some customers running Windows 7, and on Windows 7, even though ASLR randomizes the location of the heap and stack, it does not push them above the 4GB mark.

The customer asked for suggestions on what steps they can take to help flush out their pointer truncation bugs on Windows 7 machines.

Unfortunately, there's no easy solution for Windows 7. You can set the system Allocation­Preference to MEM_TOP_DOWN. This requires a reboot, and the setting applies to the entire system (not a single process), and it makes memory allocation slower, so this approach is suitable only for testing purposes.

The customer wanted to force the heap and stack above 4GB on production machines, so the above approach is not recommended. We saw in the earlier discussion that there exists a poor-man's solution: Reserve all the available memory below the 4GB mark. That will push new allocations above the 4GB mark. The initial stack and initial heap segment will still be below 4GB, and future allocations in the process will be slower because it increases the amount of address space under management. (Searching for an open address takes longer.) The exact impact will vary depending on the application's memory allocation patterns, so you will want to measure the performance before and after to see if the degradation is acceptable. But it's better than nothing,

Comments (12)
  1. kantos says:

    We found that setting LARGEADDRESSAWARE explicitly on x64 builds seemed to do the trick without needing to do that. Windows 8 didn't care about the flag, but Windows 7 seems to act as if it's set to false by default for some reason.

    1. Joshua says:

      I thought that was on by the linker by default for x64 and turning it off restricted the process to 2GB even if it was 64 bit. Maybe that too stopped in W8.

      1. kantos says:

        We thought so too, or more to the point we assumed it was ignored unless explicitly set to NO. But we found that explicitly it to YES resulted in Windows 7 allocating above the 4gb line. In this particular case it was quite annoying since I had a developer senior to me on Windows 7 that couldn't reproduce a crash bug I was seeing. We messed around with the flags and found this behavior was very consistent. From that point on we made sure all components had it explicitly set to YES even though it should be ignored.

  2. Adam says:

    There's some sample code at https://randomascii.wordpress.com/2012/02/14/64-bit-made-easy/ that tries to reserve all the address space below 4GB, so that future allocations come from address above 4GB.

  3. Pierre B. says:

    The linker no longer support specifying the location of your code and stack? Maybe I'm remembering wrong, I thought there were ways to force the addresses you wanted.

    Also, I would have thought that since address space is private to the process, the OS could simply add a fixed offset to all allocations from a range it would know is free (say, by design). For example, adding 2^35 to all virtual addresses, causing no slow-down.

    1. Matthew w. says:

      @Pierre
      I doubt the vm code is optimized for this kind of use case, although what you're saying is probably possible.

    2. smf says:

      It would appear the customers problem was with the heap, not the stack or code segments.

  4. Joe says:

    Try PVS-Studio (http://www.viva64.com/en/pvs-studio/) which can find many of these bugs through static analysis.

  5. Without seeing the code for the affected program I'm guessing it won't be nice to have to maintain. They are resorting to testing in production?!?

    1. Dave Bacher says:

      TDD is great, good engineering too -- preview programs like Windows Insider are good too.

      No amount of testing can replicate the unexpected chain of events caused by users and the third party software they have installed. You don't know what is injecting into the process -- and you can ask, but the customer won't know, either. You have a tremendous diversity, and third party drivers, etc.

      And so sure, sometimes the only solution is to test on production, or do some beta program

  6. Myria says:

    Use a parent process to help with this. Have the parent process call CreateProcessW with CREATE_SUSPENDED, then do:
    for (std::uintptr_t addr = 0; addr < 0x100000000ULL; addr += 0x10000) {
    VirtualAllocEx(process, reinterpret_cast(addr), 0x10000, MEM_RESERVE, PAGE_NOACCESS);
    }
    ResumeThread(thread);
    Intentionally ignore VirtualAllocEx errors. Certain things will end up below 4 GB, such as the environment and command line, but a bug with handling these is unlikely.

    1. Myria says:

      Just realized a bug in my code: addr should start at 0x10000. The zero page should already be reserved, and passing zero would have allocated the reserved region at a random address.

Comments are closed.

Skip to main content