New NX APIs added to Windows Vista SP1, Windows XP SP3 and Windows Server 2008

In the interests of helping secure the platform, we want more people to opt-in to using Data Execution Prevention (aka DEP aka NX), and we have lowered the barrier to entry for application developers in Windows Vista SP1, Windows XP SP3 and Windows Server 2008.

We've added some new APIs that allow a developer to set DEP on their process at runtime rather than using linker options. The new APIs also give developers some more flexibility if your application uses an older version of the Active Template Library (ATL.) Before I explain the new APIs, let me give you a little history behind ATL and NX.

Some ATL History

ATL has been around for a long time; it's reasonably light-weight and allows developers to build COM components rapidly. It also includes classes for manipulating security descriptors and such; to be honest, it makes working with Windows security objects open to mere mortals.

Older versions of ATL, and by older I mean pre-Visual C++ 2005, used dynamically generated code in small isolated cases. Obviously, without the appropriate APIs this is going to cause problems on a DEP-enabled computer, because you can't execute data. This code is referred to as a "thunk" and versions of ATL in VC++ 2005 and later work correctly with DEP.

The APIs

The most important API added is SetProcessDEPPolicy,   which sets the DEP policy for the running process. You would normally use this function pretty early in main.

The function takes only one flag argument: the policy setting. 

If your program loads 3rd party plug-ins or makes use of older ATL libraries you should use the flag below, as this enhances compatibility:

  • PROCESS_DEP_ENABLE Enable DEP for the process and allow ATL thunk emulation.

If your program does not use legacy or 3rd party plug-ins, nor make use of older ATL libraries, you should use:

  • PROCESS_DEP_ENABLE | PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION Enable DEP for the process, and disallow ATL thunks.

There are two other functions: GetSystemDEPPolicy and GetProcessDEPPolicy; I'm not going to insult your intelligence and explain what they do.

The only negative to these APIs is they must be dynamically loaded because they don't exist on all supported versions of Windows. The following code shows how you can use the functions regardless of Windows version:


If you OR the two flags together, it's virtually the same as linking with /NXCOMPAT.

When to use the NX APIs

There are three main reasons to use these new APIs:

  • If your application has some form of in-process extensibility mechanism, and some applications might use older ATL, then you can enable DEP for your process, and the extensibility mechanisms using ATL will function correctly.
  • If you support DEP but want to allow customers to disable DEP if there are serious compatibility issues, then this is the API to use because the argument can be a configuration option.
  • If your application uses an old version of ATL, and you still want to do the right thing by DEP, then use this function. Of course, you really ought to use an updated version of ATL!

One Caveat

I'm only telling you this because it bit me.

There is one caveat that you should know; SetPRocessDEPPolicy often returns error 5 (Access Denied) but this error does not mean the operating system is denying access, it means you are attempt to change DEP policy in a way that is not appropriate. For example, if you link with /NXCOMPAT, and then use this API, you'll get the error. Or, if the operating system is configured to use DEP for all processes all the time no matter what, then you'll see the same error. Finally, you'll get an access denied error if you attempt to call SetPRocessDEPPolicy twice in one application; once the policy is set, it's set for the process lifetime.

In short, don't be overly alarmed if you see this error.

#define PROCESS_DEP_ENABLE                          0x00000001


       HMODULE hMod = GetModuleHandleW(L"Kernel32.dll");

       if (!hMod) return FALSE;

       typedef BOOL (WINAPI *PSETDEP)(DWORD);

       PSETDEP procSet = (PSETDEP)GetProcAddress(hMod,"SetProcessDEPPolicy");

       if (!procSet) return FALSE;

       return procSet(dwFlags);


Comments (27)

  1. Curt Nichols says:

    > an older version of the Abstract Type Library (ATL.)

    a.k.a. (by most of us) Active Template Library.

    Thanks for an informative article.

  2. Tim says:

    What about Windows 2k3?  Are there plans for a service pack to add this functionality there as well?

  3. IronGutsMorla says:

    Why provide the option to disable it? it seems that makes easier the job of shellcode exploits.

  4. Curt, you are 100% correct – I will correct the name.

  5. IronGuts

    If you’re running shellcode, then you must have already defeated NX!!

  6. jenny says:


    all Windows components have DEP enabled, so this doesn’t affect the security of Windows. Only 3rd party applications can break DEP security

  7. A while a go when I posted about the .NET Framework 3.5 and 2.0 SP1 being available for download, Kima

  8. o.s. says:

    Michael your One Caveat section was something I found truly disturbing. You mentioned three distinct error condtions there and the system actually only responds with one single code. Error 5 (Access Denied)!

    Hey you work at Microsoft can’t you just reach out and smack the developers in the head and have them at least attempt to use error messages and codes that are SPECIFIC to the error condition that actually occurred? 🙂

  9. Nice, but the APIs missing in Windows are the ones to patch the system i.e. to force downloads of Emergency (critical/wormable ones with an exploit in the wild) or Critical patches

    Patching APIs would make a HUGE difference in how systems are protected as they could be called by installers (even third party ones) or system management software …

    or even tempt people to write inoculating viruses (vaccines);-)

  10. stefan, updating is built into the OS, it’s not an app thing. that’s why we default new OSs to check for updates every 24hrs

  11. o.s. – *THINK* the issue relates to the granularity of the underlying APIs, it has a fixed set of errors, and Err5 is one.

  12. Per faciliare la pianificazione di un corretto processo di update di Vista oggi Renato, sul blog di Technet

  13. Per faciliare la pianificazione di un corretto processo di update di Vista oggi Renato, sul blog di Technet

  14. antivirus says:

    Thank You For Sharin very inforamtive materials with us

  15. Chris says:

    There are other libraries besides old ATL that use thunking.  For instance, our app is build with OWLNext which uses thunking for windows in a way similiar to ATL.  Is there a way to turn on DEP but allow these specific thunks to work?

  16. Chris says:

    I have DEP problems with Server 2003 Enterprise. Is there a possiblility to get the current state of DEP-settings (via an alternative way for GetProcessDEPPolicy / GetSystemDEPPolicy)?

    We use a translation tool that modifies the address/code of LoadResString and at that point  our program is being kicked without exception.

    When the program is in the list it works fine.

    But that is not acceptable for clients. They shall know what went wrong…

    Any hint would be welcomed.

  17. Igor Levicki says:

    1. Error code should be invalid parameter or something, not access denied.

    2. NX/DEP can be easily defeated.

    NX = wasted silicon.

  18. IronGutsMorla says:

    "’If you’re running shellcode, then you must have already defeated NX!!"

    Not really, in return to libc attacks you are not there yet. If you can change the return address to point to this function you can disable NX in one more convenient way than before. It would be a two stage attack of course.

    maybe we can go from other side, what cases does it support flipping this flag over and over?

  19. Peter Westerström says:

    I’m getting a bit confused about the flag PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION.

    If I have an ATL application using old ATL, shall I set flag to PROCESS_DEP_ENABLE | PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION or  PROCESS_DEP_ENABLE only?

  20. IEBlog says:

    Hi, I’m Eric Lawrence from the Internet Explorer Security Team. With the RSA security conference kicking

  21. ASPInsiders says:

    First, let me remind you that in my new ongoing quest to read source code to be a better developer ,

  22. Scott Hanselman has a look under Chrome’s hood and how it uses the new NX/DEP APIs we added to Windows

  23. Hello, my name is Xiang Fan and I am a developer on the C++ Shanghai team. Today I’d like to talk about

  24. こんにちは、五寳です。 IE7 から実装されているメモリ保護 ( DEP/NX Memory Protection ) の機能ですが、IE8 からは (条件がそろえば) デフォルトで有効になっています。

Skip to main content