Getting MS-DOS games to run on Windows 95: Working around the iretd problem

Today's story is the story of Speed Racer in the Challenge of Racer X. Here goes. The really scary thing is that I still remember the details.

To this day, I can't bear to listen to the Speed Racer theme song because I spent over a week debugging why the program froze up right after the title sequence music. The crashes were completely nonsensical and random.

Windows 95 uses the iretd instruction to return from the kernel back to the application. After days of frustrating head-scratching, I eventually discovered that if you use the instruction to return from the kernel back to the application, and the application is running 32-bit protected-mode code on a 16-bit stack, then only the bottom 16 bits of the esp register are updated by the iretd instruction. The upper 16 bits remain unchanged and continue to hold the value they had while you were in kernel mode. This behavior doesn't appear to be documented anywhere in Intel's reference books.¹

The effect of this is that 32-bit protected-mode code running on a 16-bit stack will observe that the upper 16 bits of the esp register are spontaneously corrupted randomly. (Sound familiar?) Unfortunately, Speed Racer was counting on the upper 16 bits of the esp register remaining zero.

To fix this, I had to counter insanity with more insanity.

At the last moment before restoring all the general purpose registers and executing the iretd instruction, Windows 95 does a check to see whether the troublesome scenario is about to occur. If so, the kernel sets up a temporary stack selector whose base linear address matches the high 16 bits of the kernel esp register, then switches to that stack while simultaneously zeroing out the high 16 bits of its own esp register. This double-switch rewrites the ss:esp value such that it points to the same memory, but shuffles the bits around to arrange for the high 16 bits of esp to be zero. In other words, it rewrote SS:ESP = 00000000 + xxxxyyyy as SS:ESP = xxxx0000 + 0000yyyy. (Sound familiar?)

At this point, the kernel is set up to restore the general purpose registers and perform the iretd. This returns control back to the application with the high 16 bits of the esp register set to zero, as the application expects.

Now, this may seem like an awful lot of work just to get a single game to work, and it's not like Speed Racer was a blockbuster game like DOOM. However, this particular problem was not intrinsic to Speed Racer. Rather, it was a problem in the client-side library code that came with the MS-DOS extender they were using, and that MS-DOS extender was one of the major players in the MS-DOS extender market, so fixing this issue actually fixed a lot of programs. It's just that Speed Racer was the first one discovered to exhibit the problem, so it was the one I ended up debugging.

¹Maybe I'm missing it. You tell me if you see it in there. The pseudocode at the RETURN-TO-OUTER-PRIVILEGE-LEVEL label talks about raising an exception if the stack doesn't have at least 8 bytes of data in it, but it doesn't appear to discuss what happens to the esp register. The discussion says "If the return is to another privilege level, the IRET instruction also pops the stack pointer and SS from the stack," but it doesn't mention what happens if the destination stack pointer is a different size from the current stack pointer.

Comments (11)
  1. Karellen says:

    > the application is running 32-bit protected-mode code on a 16-bit stack

    The obvious question here is, why would you use a 16-bit stack? So I had a bit of a dig, and found the following quote on the Wikipedia page of one of the major players in the 32-bit DOS extender market:
    > The compiler generates 32-bit code, which runs natively in 32-bit protected mode while switching back to 16-bit DOS calls for basic OS support.
    > While [the platform] runs in 32-bit protected mode, its stub and library heavily rely on many 16-bit DOS and BIOS calls.

    Just in case anyone else was wondering.

  2. Antonio Rodríguez says:

    The world of 32-bit DOS extenders was weird. Since the native OS only supported a 16-bit model and it wasn’t multitasking, the 32-bit developer thought it was free to do as s/he whished, as far as it switched back to 16-bit before calling into DOS. And I suspect most of the hacks involved performance (after all most of its “clients” were games of heavy-load applications). Having a single 16-bit stack would avoid moving data back and fort two 16-bit and 32-bit stacks, and would greatly speed calls and interrupts.

    Debugging those 32-bit extenders to make them work in Windows 95 must have been everything but fun :-( . I send my condolences, Raymond.

  3. Myria says:

    I wish that modern NT had NtSetLdtEntries, so that I could create my own NTVDM for 64-bit Windows. Maybe the Linux on Windows thing will allow modify_ldt and arch_prctl so that it could be implemented now.

  4. DWalker says:

    I hate the theme music too. And “Speed Racer” is just a silly name. who would name their child “Speed”? Especially if your last name was already “Racer”? As a kid, I thought about these things…

  5. Joshua says:

    I found it long ago. Whenever you transition from 32 bit back to 16 bit, the 32 bit portions of all registers are preserved in the 32 bit portions of the registers and will be there next time you transition back to 32 bit. Now obviously this is pretty bad and v86 interrupt to protected mode should have protected the bits but I guess Intel never thought a process would move back and forth between v86 mode and 32 bit protected mode.

  6. Martin Bonner says:

    Huh! I can still remember the details of a bug switching between 16- and 32-bit mode on a Pr1me mini-computer before 19*8*5. (It was a slightly simpler bug though).

  7. HiTechHiTouch says:

    I don’t know any of the design players in in Intel 286 “brain dead” birth, or the Intel 386 “32 bits and never look back” delivery.

    But, as an old math prof of my used to say, “It is blatantly obvious to the most trivial of observers that…” that _Things were Missing_. Why else did the triple-fault become so important?

    {insert link to story about Intel visit to MS asking programmers for 386 suggestions and hearing about the ingenuity required to get the processor out of 32 bit mode.}

    {insert link to story about original IBM PC’s use of A31 line as a mark on the wall} so Intel won’t feel alone.

  8. Cesar says:

    Wait, didn’t I see this issue somewhere before? *searches around* Yup, the Linux folks also hit the same issue, and their fix was similar. The keyword is “espfix” if anyone wants to know more.

    1. Scarlet Manuka says:

      “espfix” makes it sound like it’s those famous psychic powers in play again (I assume Linux devs have them too). :)

      I’m glad I wasn’t the one who had to work out what was going on. It sounds like it would have been a nightmare to debug. Even with the psychic powers.

  9. Scarlet Manuka says:

    Also – I love this series. Can’t wait for next Monday’s adventure!

Comments are closed.

Skip to main content