Is DEP on or off on Windows XP Service Pack 2?

Last time, we traced an IP_ON_HEAP failure to a shell extension that used an older version of ATL which was not DEP-friendly. But that led to a follow-up question:

Why aren't we seeing this same crash in the main program as in the shell extension? That program uses the same version of ATL, but it doesn't crash.

The reason is given in this chart. Notice that the default configuration is OptIn, which means that DEP is off for all processes by default, but is on for all Windows system components. That same part of the page describes how you can change to OptOut so that the default is to turn on DEP for all processes except for the ones you put on the exception list. There's more information on this excerpt from the "Changes to Functionality in Microsoft Windows XP Service Pack 2" document.

The program that comes with the shell extension is not part of Windows, so DEP is disabled by default. But Explorer is part of Windows, so DEP is enabled for Explorer by default. That's why only Explorer encounters this problem.

(This little saga does illustrate the double-edged sword of extensibility. If you make your system extensible, you allow other people to add features to it. On the other hand, you also allow other people to add bugs to it.)

The saga of the DEP exception is not over, however, because it turns out I've been lying to you. More information tomorrow.

Comments (11)
  1. movl says:

    So DEP was not made to prevent IP_ON_HEAP?

    [“What does DEP do?” and “When is DEP enabled?” are two different questions. -Raymond]
  2. Leo Davidson says:

    I’ve configured my Vista machine to use DEP by default for everything and only a handful of things crash on it. From memory, just a couple of games so far had needed adding to the whitelist to prevent crashes during startup.

    Of course, YMMV and it may just be I’ve been lucky with the things I run.

    I can see why it’s off by default for non-system processes but tech-savvy people who want that extra bit of protection, or to know when things are going wrong so they can fix/report the problems, probably shouldn’t fear turning it on. Of course it could cause you to lose some work when using a program that otherwise would have continued okay but my experience is that DEP rarely kicks in except when something was about to crash anyway, or in cases where it always happens. (In the latter case you add the program to the whitelist the first time you use it, or ditch the program, and carry on.)

  3. ak says:

    As we know from the securitynow podcast, optout is not really optout, securom and other evil stuff in windows ignore everything except always off

  4. Dean Harding says:

    ak: what do you mean? You mean that SecuROM adds itself to the whitelist? I don’t see what the big deal with that is…

  5. ak says:

    sorry, not securom, safedisc and starforce(even worse;)

    "The second check it performs is to look in the application database for the process to see if NX support should be disabled or enabled. Lastly, it checks to see if the DLL that is being loaded is flagged as having an NX incompatible section (such as .aspack, .pcle, and .sforce)"

  6. Leo Davidson says:

    One problem is that protection code may (definitely does in some cases) run as a privileged service. In those cases, disabling DEP would make it easier for something to exploit bugs in the protection code in order to escalate privileges.

    (I do not know whether or not there are protection systems which both disable DEP on themselves and run with high privileges but if there are then that is a bothersome combination.)

  7. Dean Harding says:

    Leo: That’s certainly a problem. But most copy protection applications have more security problems than just being easy to exploit ;)

  8. Dean Harding says:

    ak: but DEP is not *trying* to protect against those sorts of programs anyway. I’m not sure what the problem with having safedisc and starforce in the "compatibility whitelist" is..?

  9. Rune Moberg says:

    I read an IE team blog a couple of months ago, and IIRC IE runs with DEP disabled by default. It seems the Java Runtime Engine does not run too well with DEP enabled, and so the IE team was forced to bow out. :(

    I hate Java. I really do. Nasty language, nasty class library and worse performance than a cat in water.

  10. wqw says:

    I thought (old versions of) ATL and MFC thunks are all recognized by the DEP exception handler? Otherwise using DEP on user processes would be a disaster, considering all the VC6 compiled apps.



Comments are closed.

Skip to main content