Debugging virtual memory problems

Today I looked at a virtual memory usage bug. I determined that the function to set a breakpoint on is {,,ntdll}_ZwAllocateVirtualMemory@24. This function is called by the heap APIs, and by the VirtualAlloc APIs.

Comments (4)

  1. Pavel Lebedinsky says:

    By the way, here’s how one could figure it out on their own:

    c:debuggers> cdb notepad

    0:000> * Let’s see what VirtualAlloc does:

    0:000> u kernel32!VirtualAlloc


    77e7ac72 55 push ebp

    77e7ac73 8bec mov ebp,esp

    77e7ac75 ff7514 push dword ptr [ebp+0x14]

    77e7ac78 ff7510 push dword ptr [ebp+0x10]

    77e7ac7b ff750c push dword ptr [ebp+0xc]

    77e7ac7e ff7508 push dword ptr [ebp+0x8]

    77e7ac81 6aff push 0xff

    77e7ac83 e89cffffff call kernel32!VirtualAllocEx (77e7ac24)

    0:000> * Now let’s disassemble VirtualAllocEx:

    0:000> u kernel32!VirtualAllocEx

    0:000> u


    77e7ac52 ff158811e677 call dword ptr [kernel32!_imp__NtAllocateVirtualMemory (77e61188)]

    0:000> * Dump import address table entry at 77e61188:

    0:000> dds 77e61188

    77e61188 77f5b548 ntdll!ZwAllocateVirtualMemory

    The cool thing about windbg/cdb is that you can do debugging, poking around in the OS internals and a ton of other things all from the same tool. For example, here’s how you can use the above info to trace all VM allocations:

    0:000> .symfix

    0:000> bp ntdll!ZwAllocateVirtualMemory "k;g"

    0:000> g

  2. Gregg Miskelly says:

    One could easily figure this out in VS as well, which is what I did. The next version of Visual Studio will also support tracepoints (breakpoint that print a message and continue when hit), if thats what you want.

  3. Vince says:

    I had the same problem my computer would not even start past the welcome screen. I reinstalled windows and everything was fine.

  4. Gregg Miskelly says:

    Opening an EXE as a project is done by the VC project system, so you need to have VC installed. As long as you have that it should ‘just work’ in 2002, 2003 or 2005 versions.