Adventures in application compatibility: Following a detour off a cliff


The application compatibility team reported that a program was crashing in its installer. Debugging the installer revealed that it actually triggered three exceptions.

The first exception was due to passing an invalid pointer to the lstrlen function. However, this didn't crash the installer because the lstrlen function contains an application compatibility mitigation: If you pass a pointer to invalid memory, it handles the access violation exception by reporting that the string length is zero.

It did this twice, so that is the second exception as well.

The third exception is the one that crashed the installer: A system function wanted to report a runtime issue. It did this by checking if a debugger was present, and if so, it printed a diagnostic message to the debugger so that the developer could see that there was a problem and get a chance to investigate it.

The problem is that the call to Is­Debugger­Present was crashing.

And it was crashing because the installer used a DLL which detours the Is­Debugger­Present function when it is loaded, presumably to make it harder to reverse-engineer (though since the source code is freely available, it's not sure why they are so concerned about it).

But notice that the DLL does not un-detour the function when it is unloaded.¹

Therefore, if you load the DLL and then unload it, you leave the Is­Debugger­Present function detoured to a function that no longer exists. And the next person to call the Is­Debugger­Present function will crash.

(Also curious is that the DLL is licensed under the GPL, yet the product that uses it makes no mention of the GPL in its license agreement.)

The fix was to remove the debugging message for the particular case that this program's installer was tripping over.

¹ Not that un-detouring is a perfect solution either, because it runs into the What if two programs did this? problem: Suppose another DLLs also detoured the Is­Debugger­Present function. When this DLL unloads, it restores the original function, which also removes the other DLL's detour. And then when the other DLL unloads, it restores what it thought was the original function, but which was actually the detour installed by this DLL. As a result, this DLL's detour gets reinstalled.

Comments (14)
  1. Koro says:

    This is why I think Windows should come with a built-in method for API hooking, so everyone would stop hand-rolling their own thing.
    Something like SetWindowSubclass. Coupled with a “load this DLL in all processes right now” call (requiring administrator of course, but bypassing UIPI, sessions, and all other roadblocks) would be perfect. Something like RegisterUserApiHook, but public and less theme-specific.

    And yes I know the “but it could be used by malware!!!111” excuse applies, but there are many legitimate uses to be able to tweak one’s system completely.

    1. Joshua says:

      Load DLL into all processes actually exists, and needs to go away. Application DLLs don’t behave when loaded into specialized memory-layout processes. The level of compatibility required to make it work is absurd, and the programming shops who want to use this feature aren’t willing to even try to get it right.

      1. Koro says:

        AppInit_DLLs is a bad system, forces you to do stuff in your DllMain.
        And most DLLs misbehaving are doing so exactly because there is no proper way of doing API hooking.

      2. Harry Johnston says:

        Ideally there would be a separate subsystem for processes that want complete control over their memory space. But I’m thinking this is a case of -100 points; apart from Cygwin, I doubt such a subsystem would have seen all that much use.

        1. Joshua says:

          A long time ago I dealt with something that needed a gigabyte byte array and 64 bit was not yet available. A random 3rd party dll would be a major headache unless I could rebase it.

          1. hyperman_ says:

            Got exactly your problem: A 32 bit JVM of windows has a heap limit of the maximum contiguous block of addres space it can allocate. So of course our company bought some security contraption which injects a DLL in every process to lock down the proxy for crying out loud. It doesn’t enable ASLR and decided on a base adress somewhere around the 1GB mark to cause maximal damage. No heaps above 600 mb possible.

        2. Harry Johnston says:

          On second thoughts, although minimal/pico processes would be nice to have for special cases, the more common problems (such as those Joshua and Hyperman describe) could have been more easily solved by making rebasing mandatory for any DLL loaded via an API hook.

          (And perhaps some mechanism for an executable to say “reserve me a single chunk of X megabytes of address space before you start loading any DLLs”?)

          1. Harry Johnston says:

            … which of course you can already do. D’oh!

            @Joshua, wouldn’t making the array static have solved your problem?

  2. Joshua says:

    I looked at the actual Detours code being used for IsDebuggerPresent and ChangeDisplaySettingsEx, and that code makes a lot of sense. But whoever wrote that code forgot one thing. If you’re going to detour something, call LoadLibrary on yourself so that you don’t get unloaded.

    Hint: IsDebuggerPresent() => false;

    1. Neil says:

      Presumably if an actual debugger is present it’s not beyond the wit of the programmer to set a breakpojnt on IsDebuggerPresent and simulate return true;…

      1. R P (MSFT) says:

        Back in the day, a program which shall not be named (but which, remarkably, wasn’t a virus) did IsDebuggerPresent their own way: they’d do an int 3 and use (presumably) a hardware clock to see how quickly it executed. If it took too long, the program would terminate. And they sprinkled these checks liberally throughout the code, so they quickly got annoying.
        If you’re good enough with your debugger, you can work around that check, too, and without having to find where they’re checking the clock.

  3. adrianm says:

    I had the same problem back in the Amiga days. Two programs patched the same function and if you closed them in the wrong order it crashed.
    I solved it by patching the patch-function (SetFunction?) to return a trampoline which could be modified when the programs unhooked themselves. Caused extra indirections of course but lacking memory protection the alternative was a guru meditation. Worked well until anti-virus became a thing

    1. John Styles says:

      Quis patchiet ipsos patches?”

Comments are closed.

Skip to main content