Why is Explorer opted out of Data Execution Prevention and termination on heap corruption, and how effective is the policy to opt it back in?


Every so often, somebody will report a security issue with the fact that Explorer is linked with /NXCOMPAT:NO, which opts it out of Data Execution Prevention (DEP), also known as no-execute (NX). Explorer is also opted out of termination on heap corruption, but that's harder to detect from a static analysis. Why does Explorer opt out of these things? Doesn't that make it less secure?

Explorer opts out of DEP and termination on heap corruption as a backward-compatibility concession to shell extensions which are not DEP-compatible, or which corrupt the heap and managed to get away with it in earlier versions of Windows.

This compatibility concession applies only to the x86 version of Explorer. Explorer on all other processors (such as x64) enables both data execution prevention and termination on heap corruption.

Furthermore, this compatibility concession is controlled by a pair of group policies, both under Computer Configuration, Administrative Templates, Windows Components, File Explorer:

  • Turn off Data Execution Prevention for Explorer
  • Turn off heap termination on corruption¹

This means that even though x86 Explorer does not have DEP or termination on heap corruption enabled in its file header, it will manually enable them as soon as it verifies that neither policy is in effect. This is one of the first things Explorer does, so the window of opportunity is relatively small. (And, as noted earlier, this situation exists only on x86. All other processor architectures enable DEP and termination on heap corruption from the get-go.)

Windows defaults to the safe setting of the policy, namely to re-enable DEP and termination on heap corruption. The policy is there so that organizations can selectively enable the policies if they have a shell extension that is not DEP-compliant or which contains heap corruption bugs.

Explorer is a single-instance application, and this means that if you are running the x64 version of Windows, and you manually run the x86 version of Explorer, that copy of Explorer doesn't do much other than hand the request to the existing 64-bit copy of Explorer. So the 32-bit version is there, but it doesn't hang around for long.

So go ahead and disable those policies in your organization. The x64 version of Explorer already has DEP and termination on heap corruption enabled. Disabling these policies (or leaving them unconfigured) will make sure you cover the x86 systems as well.

Bonus chatter: But wait, if I run an x86 application on an x64 version of Windows, and I use the File.Open dialog, then doesn't that give me an x86 Explorer? And wouldn't the policies take effect in that case?

No, the policies don't take effect in that case because the File.Open dialog is not running inside the Explorer process. It's running inside Notepad, or whatever your x86 process is. The DEP and termination on heap corruption policies of the host process are the ones that control behavior.

¹ You might have noticed the incorrect wording of the policy. The policy text says "heap termination on corruption", but that doesn't make sense because it's not the heap that is terminating; it's the process that is terminating. The phrase should be "termination on heap corruption". The reason for the messed up wording is that the heap feature is known as "terminate on corruption", and people lazily glue the word "heap" onto the phrase "terminate on corruption".

Comments (16)

  1. Yukkuri says:

    As a highly paid security consultant that only knows how to check header flags I will keep telling my clients that explorer is dreadfully vulnerable. Sorry but I need to justify those consulting fees somehow :)

  2. middings says:

    Would the phrase “termination on corruption of heap” satisfy everybody?

  3. BZ says:

    @middings,
    no, that would need to be “corruption of the heap” at which point it might be too long

  4. Yuhong Bao says:

    It used to be possible to run the 32-bit version of Explorer in 64-bit Windows. I wonder why this was broken in Win7 (I think).

    1. You weren’t ever supposed to be able to run the 32-bit version of Explorer on 64-bit Windows. If you could, it was a bug. I guess Windows 7 finally fixed that bug.

      1. alegr1 says:

        Bob: “So we just went ahead and fixed the glitch.”

    2. Joshua Schaeffer says:

      Fortunately you can still run a 32-bit Explorer clone just fine. All those IShellFolder and IContextMenu posts pay off sooner or later.

  5. Re: single instance… so what would happen if explorer crashes, and task manager is used to specifically open the x86 build? would the x86 recognize that it’s running on x64 platform and hand off the instance, or would it just look to see if ‘explorer.exe’ is running (finding that it’s not) and allow it to start?

    1. I don’t remember. You could always try it and see.

    2. GL says:

      If you start C:\Windows\SysWoW64\explorer.exe, it will instead start an C:\Windows\explorer.exe. My guess is that the 32-bit explorer will always start the 64-bit explorer with appropriate command line and leave the “handling the single-instance thing” to the 64-bit one. Not verified though.

  6. Yuhong Bao says:

    Don’t forget that a lot of shell extensions use ATL too, and /NXCOMPAT might break them without ATL thunk emulation.

    1. Isn’t that what I said in the article? “as a backward-compatibility concession to shell extensions which are not DEP-compatible.”

  7. 640k says:

    What an unusual move by MS, now sloppy developers need to actually fix bugs in their plugins when they recompile the code for x64. Not just pressing Ctrl+Shift+B with the eyes closed.

  8. IanBoyd says:

    How do i opt into *Termination on Heap Corruption*?

    I know how to opt into NX:

    MapAndLoad(pathToExe, null, out li, false, false);
    LI.FileHeader.OptionalHeader.DllCharacteristics ^= IMAGE_DLLCHARACTERISTICS_NX_COMPAT;
    UnMapAndLoad(li);

    Microsoft Security Research and Defense blog says:

    > These improvements are enabled by default (with the exception of termination on heap corruption)

    **Note**: There are compilers besides Visual Studio and GCC that create native Windows executables. I’m using one such compiler (which is why i have to edit PE flags directly)

    1. laonianren says:

      Raymond wrote: “Explorer is also opted out of termination on heap corruption, but that’s harder to detect from a static analysis.”

      Which implies that the option isn’t configured with a flag in the PE file. Call HeapSetInformation.

  9. Barteks2x says:

    Probably not very relevant but reading this remineded me of something:

    Back when I used windows XP (32-bit), after a clean system reinstall + (maybe, I don’t remember) instaling an antivirus, I started getting message that said that “Data Execution Prenention” prevented Explorer.exe from working (or something like that, worded differently). It was the first time I’ve seen tat windows even has such feature!

    Even now if I got such error, I would di the same thing as I did back then – reinstall windows again.

Skip to main content