PE resources must be 4-byte aligned, but that doesn’t stop people from trying other alignments


Resources in PE-format files must be stored at offsets which are a multiple of four. This requirement is necessary for platforms which are sensitive to data alignment.

That doesn’t stop people from breaking the rules anyway. After all, it sort of works anyway, as long as you’re careful. I mean, sure maybe if somebody running a non-x86 version of Windows tries to read your resources, they will crash, but who uses non-x86 versions of Windows, right?

In Windows Vista SP1, additional hardening was added to the resource parsing code to address various security issues, but the one that’s important today is that tests were made to verify that the data were properly aligned before accessing it. This prevents a file with a misaligned version resource from crashing any program that tried to read its resources. In particular, it is common for programs to read the version resources of arbitrary files—for example, Explorer does it when you view the file’s properties or if you turn on the Description column in Details view—so enforcing alignment on resources closes that avenue of remote denial of service.

And then the bug reports came in. “Program XYZ fails to install” because the program tries to read its own version resources and cannot, because the tool they used to build the program cheated on the alignment requirement and stored the resources at offsets that aren’t multiples of 4. “I mean, come on, that wastes like three bytes per resource. Everything still worked when we removed the alignment padding, so we went ahead and shipped it that way.”

Another example of a program that stopped working when the alignment rules were enforced was a computer game expansion pack which could not install because the code that tried to verify that you had the base game found itself unable to read its version resources.

Multiple programs refused to run, preferring to display the error message “AppName is not a valid Win32 application.” Presumably, as part of initialization, they tried to read their own version resources, which failed with ERROR_BAD_EXE_FORMAT, which they then showed to the user.

The fix was to relax the enforcement of the rules back to the previous level, and impose the stricter requirements only on architectures which raise exceptions on misaligned data. It does mean that you can have a program whose resources can be read on one machine but not on the other, but that was deemed a lesser evil than breaking all the programs which relied on being able to misalign their data without consequence.

[Raymond is currently away; this message was pre-recorded.]

Comments (38)
  1. Alex Grigoriev says:

    After all that done, Windows 2008 R2 stopped supporting that alignment-enforcing architecture at all. Good riddance.

  2. Mordachai says:

    Such a pain trying to thread the needle of backwards-compatibility & reasonable enforcement of the rules.

    Too many developers still fail to grasp the importance of following the rules, even when you don't grok their meaning.  Just because you're too dim, doesn't make it okay to break the system and impose your stupidity on the OS and everyone else down the road… which is exactly what happens (or MS has to break compatibility, and then they look like the bad guys).

    Granted, the reasons behind the rules should be made as available as possible, so that folks can understand the reasons why (or why not).

    But failing to generate consistent output means that integrity checks, security checks, and interoperability are all compromised going forward.  It also means your own software is a fragile POS.

    Lame.

  3. Leo Davidson says:

    I bet a lot of things like this are as much people not knowing/remembering the (exact) rules as much as it is people wilfully breaking them.

    Alignment problems were a pain back in my 68000 assembler days. I assumed way back then, what with newer chips not having a problem with alignment (albeit at a performance cost) and with OS being able to trap the exception and emulate the operations in other cases (at a bigger performance cost) that the problem would be a thing of the past by now.

    Maybe, with those hardware+OS combinations where it's still a problem, we need to fire those platforms into the sun? :)

    Unless we're talking sensitivity to alignment w.r.t. atomic reads/writes, which is another story.

  4. Joseph Koss says:

    With the introduction of SSE on x86, alignment issues grew to became a matter of practical, rather than just theoretical, concern on the platform.

  5. laonianren says:

    @Leo Davidson: in this case (indeed, in most cases) you don't have to remember these kind of low-level rules because the tools take care of them for you.

    Which makes me wonder how this happens.  Writing your own resource compiler must be very unusual.  Is there some third-party compiler or post-build resource patcher that's getting this wrong?

  6. Mike Dimmick says:

    @Alex Grigoriev: ARM also requires alignment, so it'll be back in Windows 8.

  7. Ivo says:

    @laonianren – I was wondering the same thing. If I use the resource compiler or the UpdateResource family of functions I expect the alignment to be taken care of for me. How did the resources got misaligned in the first place?

  8. SimonRev says:

    Ivo:  Raymond answers that in his post.

    For whatever reason you don't use the resource compiler.  One example he gives is a third party installer creator.  A typical installer creator will take the original files, compress them and emit an executable that can be run on the target machine to install some software.  

    Such software would almost certainly store some resources in the exe that it creates (at a minimum an icon and UI resources for its install wizard.  Likely the actual files themselves as well).  

    If I am an installer creator vendor, I can't exactly distribute Microsoft's resource compiler so I might choose to write my own.  In that case it might be easy to overlook (or ignore) the alignment restrictions, when everything seems to work fine.  In fact, I might be highly tempted to ignore them if I am trying to make the claim of "Creates smaller install files than the leading competitor" (Ok, nowadays that seems silly, but in 1993 it might not have — especially since I used a lot of software back then that had hundreds of tiny files — those gaps could have been a couple of percent of the total space)

  9. Ivo says:

    Yes, I was thinking about an installer as well. But an installer creator should probably use the resource API (like BeginUpdateResource), and not do PE surgery.

    As for purposely misaligning data to save memory – you are saving 2 bytes per file on average. Hundreds of files translates into two hundreds of bytes. I don't think that's worth boasting about, even in the 93 era. :)

  10. Alex Grigoriev says:

    @Mike D:

    Fortunately, there are no legacy applications for ARM.

  11. Pierre B. says:

    The simple law is "An unenforced rule is not a rule."

    Works in software, works in real life, works when raising children.

    Anecdote: the Amiga had such a problem: the memory allocation had a flag telling if the memory was shared or not with other processes. The original Amiga didn't have a CPU with memory protection, so nobody cared or got it right. So the Amiga was doomed to never have memory protection, at least not without ditching almost entirely its existing software base.

  12. Anonymous Coward says:

    Pierre, that sounds like something that could be easily dealt with using a relatively simple shim.

    Also, with regards to Windows enforcing alignment issues on x86, that's pointless. A PE tends to contain code for one platform only; nothing is gained by enforcing alignment for x86 PE files in the vast majority of cases.

    Given this, and given that enforcing pointless rules is also a bad thing, I'd just leave it.

    And maybe hardware designers should take note of this story, and many similar ones. Information wants to be byte-addressable.

  13. steveg says:

    @Anonymous Coward: "Information wants to be byte-addressable."

    Information is inherently analog. Discuss.

  14. dave says:

    I think this turns Postel's Law(*) on its head.  I give you "dave's dictate":  be absolutely stingy on what you accept, because some fool will always do the wrong thing and you'll be stuck with supporting it forever after.

    (*) en.wikipedia.org/…/Robustness_principle

  15. David Matthew says:

    "Information dislikes being anthropomorphized."

  16. Evan says:

    @Anonymous Coward: "Information wants to be byte-addressable."

    Hmm, I'm not sure I buy that. On platforms that allow byte-addressable memory, there can be issues because of it. Just a couple days I was forwarded an article about a company who had a race condition brought about because they used a field in a structure as a spinlock, packed the structure, and then had non-atomic accesses to that spinlock because it crossed a cache line. Less dramatic, of course, are the performance costs.

    Though I haven't looked into it enough to have any remotely-strong opinion on the matter, I tend to think that the platforms that do require a "reasonable" alignment are closer to being right nowadays. (I consider 4 bytes to be "reasonable", and *might* consider 8 bytes to be reasonable on a 64-bit platform.)

    BTW, why stop at bytes? Why shouldn't information be bit-addressable?

  17. Drizzt says:

    I'm loosing something.

    I never had to work with PE format, but should I have to do it, I'd pick its specs and follow them. What's difficult about that?

  18. NB says:

    How do you create resources that don't follow the rules?

    I would assume all resource editors – Visual Studio and the like – follows the rules, so does this mainly concern resources created "manually"?

  19. Burak KALAYCI says:

    Resources in PE-format files must be stored at offsets which are a multiple of four.

    I find this quite unacceptable in this day and century. Certainly a leaked abstraction, if there was an abstraction in the first place… What kind of a primitive format will have such requirements? What's next, compilers imposing code formatting rules? (Please don't make me remember).

    Don't harden the rules, instead relax the specs. Vista did it wrong again and why I'm not surprised at all that I'm not surprised at all should be pretty obvious to anyone.

    Why shouldn't information be bit-addressable?

    Exactly. It should be. Period.

  20. NB says:

    Isn't the multiple of four requirement included in order to support direct memory-mapping of the resource data?

    Which would seem very desirable in many cases.

  21. dancol says:

    Speaking of enforcement: the x86 has an alignment check mode (turned on via the "AC bit", oddly enough) that, when enabled, will cause the processor to treat misaligned accesses as errors. Unfortunately, it hasn't been very useful in practice because quite a bit of code actually uses unaligned accesses, causing *your* bugs to be lost in the noise — but I last tried a long time ago. I wonder whether alignment checking would be more useful today.

  22. Tim says:

    @Anonymous Coward: "A PE tends to contain code for one platform only; nothing is gained by enforcing alignment for x86 PE files […]"

    "On my new 'Windows 8' running on ARM, Explorer crashes reproducibly when navigating to a folder containing backups of my old x86 installation. Oddly enough this only happens when I switch to details view with the Description column enabled."

  23. Anonymous Coward says:

    Steveg: This has never been proven, and our developing knowledge regarding quantum mechanics and QFT suggests otherwise.

    Tim: You're assuming that the coders of the resource extractor functions are idiots. They'll put a try clause around the resource reader on 4 B aligned platforms, and the worst that'll happen is that you don't see versions for a small number of PE files that won't even work on your platform anyway.

  24. 640k says:

    Why not enforce alignment on binaries compiled after a set date? Like 20121221?

  25. Tim says:

    @Anonymous Coward: I understand that there are ways to solve the problem that never existed. I am however having difficulty following your "won't even work on your platform" way of thinking. You're making it sound like a PE-image will only ever be accessed for the sole reason of executing its code. As was described in the article this is not the case. Care for another demonstration? "Dear support team, ever since migrating our centralized update server to 'Windows 8' on ARM it stops working unexpectedly. It appears to happen whenever an x86 client queries for updated executables. The event log has lots of ERROR_BAD_EXE_FORMAT entries. All executable files however have been verified to run on the target platform without any problems. Is this is a bug in 'Windows 8'?"

    Burak KALAYCI: "[the alignment rules are] certainly a leaked abstraction":

    The rules are an implementation detail that never surfaced. Both the resource compiler as well as the resource access API do not expose this detail while still producing conforming results.

    Burak KALAYCI: "> Why shouldn't information be bit-addressable?

    Exactly. It should be. Period."

    Wouldn't you complain if 32bit Windows' address space was limited to 16MB per process?

  26. ErikF says:

    @640k: That won't prevent devs from gaming the system, as anyone can roll back the date prior to compilation. Also, what happens when a newer executable tries to load in resources from an older DLL? It's probably better to either enforce this stuff globally or not at all (where allowed.)

  27. Burak KALAYCI says:

    Tim: The rules are an implementation detail that never surfaced.

    Hardware implementation details leaked to the specs, as rules that don't make sense on their own… How about this one: All resources must have byte size that has 3 as remainder when divided by 7?

    Tim: Wouldn't you complain if 32bit Windows' address space was limited to 16MB per process?

    Probably yes. But what does it have to do with bit-addressability? What I state is an (ideal) idea, not related to any hardware or OS. A good OS need not magnify shortcomings of hardware, on the contrary, it must shield the users.

  28. yuhong2 says:

    "the release after Windows 2008 R2 will stop supporting that alignment-enforcing architecture at all."

    FTFY.

  29. yuhong2 says:

    Daniel Colascione: And unfortunately the Windows kernel itself don't turn the CR0.AM bit on that is required for EFLAGS.AC to work.

  30. 640k says:

    @Burak: 16mb is as useless as 4gb. 64-bit bit addressing is the way to go.

  31. Ben Voigt says:

    Clearly we need a new acronym for PE files, since "portable, executable file" just isn't accurate.

  32. Evan says:

    @Burak: "What's next, compilers imposing code formatting rules? (Please don't make me remember)."

    Ever used Python? Or to a lesser extent, Haskell?

    Here's my take: from a hardware side, alignment restrictions make sense. Allowing misaligned accesses means you've got more hardware to deal with it, and doing it *properly* (i.e. not a leaky abstraction, which it still is; see the performance degradation and atomicity issues with misaligned accesses) would take even more. Spend those transistors on something that actually matters and will help the end user, like a larger cache or something instead.

    From the PE side, it probably makes less sense. It's not like PE files require a file-system block alignment so that the system can set up a mapping from a memory page to a block (or more; not sure how big blocks are) of the file. I'm not sure what the 4-byte alignment restriction gains in that case; I don't know what's going on behind the scenes well enough to know for sure. I guess when copying the file contents from one location to another in memory, it allows the system to assume that the source address will be aligned. That doesn't sound quite convincing enough to me if that's it.

  33. ficedula says:

    @Evan: The topic is specifically talking about *resources* in PE files needing alignment. So generally, we're dealing with a section of a PE file that's been copied (or, more likely, mapped) into memory that contains a number of different resource items; e.g. bitmaps, icons, menus, …

    If you can mandate that each resource is aligned on (say) 32-bit boundaries, you can access that resource data (which probably has its own internal structure; a bitmap header at the start of a bitmap resource…) through a simple pointer and assume that all the structures within the resource in question are correctly aligned (your 32-bit field in the bitmap header is aligned to a 32-bit boundary). So you can access the data through a pointer *without* copying it around: that's the point.

    If the resource could be arbitrarily aligned, then to access the data, you either have to copy it to an aligned destination (inefficient), or access every single field in the resource one-byte-at-a-time (unaligned data, so pointers to anything bigger than a byte won't necessarily work). That's quite a pain.

    On x86, of course, your pointers *will* still work on unaligned data – but they'll be slower, so there's a benefit to mandating alignment even there (although the consequence of failing to do so is a speed penalty rather than a crash). All things considered, the PE spec requiring alignment on resources makes a lot of sense: even on the CPUs of the time, it made things faster, and as an added benefit paved the way for running on architectures that weren't so friendly to unaligned pointers.

  34. Burak KALAYCI says:

    @ficedula: How can you assume a certain hardware architecture and then say PE spec makes sense because of that unmentioned assumption?

    If there will be a performance hit, you make it a recommendation, not a rule. If you make it a rule, you enforce it. If you didn't need to enforce it, it means the rule was not necessary in the first place. If you program using a high level language like C, not to mention any toy scripting languages that use a GC, mentioning a performance hit does not make sense.

  35. Neil says:

    @Ivo: Sure, everyone has access to BeginUpdateResource now, but older installer generators that needed to run on platforms that didn't provide it would have had to have a custom resource compiler, and therefore older installers could have had unaligned resources.

  36. ficedula says:

    @Burak: It makes sense because virtually *all* architectures either insist on alignment or impose a speed penalty on unaligned access. So pretty much any "data must be aligned to X byte boundary" requirements in pretty much any structure or format has at the very least a *plausible* explanation.

    Also, bearing in mind that some Windows formats date back to the 386 days or earlier, you have the problem that enforcing every single rule (including the ones that are there for performance reasons) under every situation takes memory and time which kind of defeats the point of having them. Sure, the alignment one would be pretty easy to check, but if Windows had verified that from the beginning, Raymond would be posting about how some common non-MS resource compiler instead failed to enforce the other future-proofing rules (some of which would be quite expensive to check at runtime, like anything that involves actually examining the resource data: paging in a bunch of resources that might never actually be required by the program can be horrifically slow).

    Net result: Enforcing all the rules is often not possible, assuming sane objectives. The alignment one *could* have been enforced, but the general rule for early versions of Windows was "assume the programmer knows what they're doing, we can't afford to waste time and memory checking/cleaning up after them".

  37. Reid says:

    @ficedula: You are right. But at the same time, what Pierre B. says, also applies:

    >

    The simple law is "An unenforced rule is not a rule."

    Works in software, works in real life, works when raising children.

    <<

    As long as we talk about humans who write software, this is almost generally true.

    The final conclusion depends on the very details of the subject at hand.

  38. Mordachai says:

    There are an awful lot of comments that boil down to:

    But I don't understand why hardware isn't an idealized perfection without any trade-off compromises between infinite ease of use and infinite speed?

    And then they go on to whine that this is intolerable, and their ignorance proves that the hardware guys have it all wrong, and that they shouldn't have to learn anything or think more deeply than their idealized projection of what should be.

    And I assume your code is this perfect, has never made trade-offs between any opposed and intrinsically exclusive possibilities in the real (as in reality) world?

    Go get and education.  Learn why alignment is an issue at the hardware level.  Become a more worthy programmer who understands his tools more deeply, instead of whining that they're imperfect*.

    *BTW, news flash: the world is imperfect.

Comments are closed.