In Windows XP, even when DEP is on, it’s still sometimes off


As we saw last time, there are a variety of ways you can control DEP, one of which is to turn it on for all system processes. But even if you turn on DEP, it still sometimes turns itself off temporarily. It goes back to those bad versions of ATL.

The application compatibility team found that there were so many programs written with application frameworks that were not DEP-compatible (ATL mostly, but a few others) that nobody would actually enable DEP because the odds were close to 100% that there would be some program on the system that was not DEP-ready. Even DEP-fan Leo Davidson runs a couple of programs that don't work with DEP enabled. And it takes only one program to foul an upgrade.

When the kernel encounters a DEP exception, it checks whether thunk emulation is enabled, and if so (which it usually is), it checks whether the code sequence is one of the "well-known DEP-violating thunks". If so, then it simulates the actions the thunks would have performed and resumes execution instead of raising the exception. For example, if thunk emulation is enabled and you just took a DEP exception on the code sequence

mov ecx, 12345678
jmp 43218765

the kernel thunk emulator will perform the moral equivalent of

pContext->Ecx = 0x12345678;
pContext->Eip = 0x43218765;
return EXCEPTION_CONTINUE_EXECUTION;

You can mark your program as "I am so okay with DEP that not only do I want you to enable it, but I don't even want you to do this thunk emulation stuff" by setting the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag in your PE header. (The Visual Studio linker uses the /NXCOMPAT command line switch to set this flag.)

Comments (19)
  1. Psa says:

    That’s kind of ironic.  Code uses thunks to save a few cycles.  The thunk emulator and associated exception come along and steal rather more than a few cycles.

  2. Oleh Hadash says:

    IIRC, VB6 is another of those DEP-incompliant frameworks.

  3. Anthony Wieser says:

    You’ve got me worried now Raymond.

    I’ve been marking my latest code as /NXCOMPAT, because I’m sure my code doesn’t get the IP on the stack.

    But now, I assume that if some bad printer driver, or hook proc hooks into my software, and is based on the old ATL stuff, my software will now crash.  

    Am I right to be worried?

  4. Anthony Wieser says:

    I should have added Anti-virus software to my previous list of potential ways to take down my software.

  5. Ken Hagan says:

    "if some bad printer driver, or hook proc hooks into my software … my software will now crash."

    As I read it, the NXCOMPAT flag is per-DLL and so you are OK.

  6. Ken Hagan says:

    Whoops, scrub that, the offending EIP is on the heap and so the kernel can’t possibly trace it to the DLL that put it there.

    Hmm. Nasty.

  7. Pierre B. says:

    If I understand what you are saying, then doesn’t it defeat the purpose of DEP? The evil exploit writers will simply write their code so that it matches one of the “well-known thunk” pattern and pass through DEP.

    (I understand that DEP is not a cure-all for exploit anyway, but what it does protects against is defeated by this feature, it would seem.)

    (It still makes writing the exploit harder, though, and defeat earlier ones.)

    [The well-known thunks don’t do anything other than twiddle some registers and then jump somewhere else. DEP still applies to that “somewhere else”. All you gained was the ability to control the contents of the ECX register. That’s hardly arbitrary code execution. -Raymond]
  8. mafkees says:

    Just because you don’t see a way to exploit it doesn’t mean that there isn’t any. Even uninitialised variables can pose a security risk.

    Anyway, does anyone know how to set DllCharacteristics in mingw32? I think i’m gonna have to binary patch the exe from the makefile (maybe with dd), i’d rather not make my own build if i can avoid it.

  9. CN says:

    The point is that a memory overwrite resulting in a thunk lookalike is basically just as bad as any other data overwrite where you replace an existing function pointer directly, and there are lots of function pointers in writable memory. This doesn’t open a new class of exploits, it only extends the set of possible addresses for an existing one ever so slightly. As the whole point of DEP is to reduce the consequences of a security breach that’s already happened, that seems quite ok.

  10. Dean Harding says:

    "Just because you don’t see a way to exploit it doesn’t mean that there isn’t any."

    If you always think like that, it’s probably better if you don’t write any code at all then…

  11. Johnny DEP says:

    The pocketpc emulator in visual studio 2003 doesn’t run if windows have dep (globally) enabled. Why? Wasn’t XPSP2 developed with this compiler? How incompetent must someone be to develop software with a compiler that doesn’t work with the compiler?

    [Huh? Windows XP SP2 isn’t a compiler. -Raymond]
  12. Dean Harding says:

    Er, I don’t think Windows was developed with the Pocket PC emulator…

  13. DriverDude says:

    "Just because you don’t see a way to exploit it doesn’t mean that there isn’t any."

    If you always think like that, it’s probably better if you don’t *run* any code at all then…

  14. Sidebar Willy says:

    Interesting to learn about thunk emulation on DEP exceptions.

    I wonder why the Windows Vista Sidebar team doesn’t do this for Sidebar gadgets that use ActiveX controls that were built with old ATL.  Old ActiveX controls that work fine under IE crash Vista Sidebar.

  15. Yesterday I had an interesting case that I thought of sharing, even though there's nothing very new

  16. Hello, my name is Xiang Fan and I am a developer on the C++ Shanghai team. Today I’d like to talk about

Comments are closed.

Skip to main content