Execution protection (NX) and PAE

A comment from the earlier memory management entry posed a good question. How does PAE factor into the new No Execute (NX) mechanism enabled by the Opteron, Athlon64, and new Prescott-based Xeon?

In Windows XP SP2 and Server 2003 SP1, the two are inexorably linked. The two level address translation scheme used by the non-PAE kernel does not have enough room to accommodate any further descriptive information about individual pages of memory. The three-level scheme that PAE necessitates allows the new NX attribute to be used. (It is simply a bit in the Page Table Entry (PTE) that indicates the memory in this location is not allowed to be referenced by the instruction pointer. “Do Not Run Under Penalty of Death”.)

When you use the /NoExecute switch on these OS’s, ntldr now loads the PAE kernel, but in a special mode. You don’t get access to over 4GB of RAM, and more importantly, your drivers don’t get physical addresses over the 4GB mark either. This is important because we’ve found that many devices and device drivers, especially in the consumer space, happily assume they’ll never have to address memory at an address over the 4GB boundary.

While you may only get to use a total of 4GB RAM in XP, that doesn’t mean that some of it can’t have a physical address above the 4GB boundary. The BIOS or devices may re-map memory up there with the assumption that it won’t be seen or used. When the PAE kernel starts handing out addresses to those pages of memory, things can get ugly. The easiest way to ensure everything works like it did in the past, while allowing the new feature, is to make sure we don’t hand out any addresses over that boundary.

If you add the /PAE switch, you get the normal PAE behavior, and all bets are off. Of course, this is exactly what you want in the server space; after all, you got that extra RAM for a reason, right? Also note that the properties aren’t transitive. While both /PAE and /NoExecute use the same kernel file (ntkrnlpa.exe or ntkrpamp.exe) and address translation mechanism, you need both switches in place to enable both features.

[Added 8/6/2004 @ 2:56PM, thanks to Adam]

Note that the above information applies only to NX on processors in x86 mode. The IA-64 and the x64 platforms natively support NX, which is being enabled for the first time with the release of Server 2003 and XP for 64bit. Those platforms both already use a 3-level address translation scheme, but they're not related to PAE in any way. They address 64bits natively, and the memory structures have room to include the NX information. We just needed to add the support to the OS.