Static hooking through predefinition


A customer had a program that incorporated source code from two different third parties, let's call them Contoso and LitWare. These libraries were originally written for Linux, and they are trying to port them to Windows.

Contoso's library implements some useful feature that they want to use. LitWare's library implements some fancy memory management and wants to intercept all calls to malloc, free, and related functions. In particular, it wants to intercept the calls from Contoso.

The customer knew that they could use Detours to do the intercepting, but that would require them to obtain a professional license, and the cost was a concern.

Fortunately, since the customer is building all the libraries themselves, they can make changes to the code and recompile.

I suggested using this header file:

// interceptable.h
extern void* (*intercepted_malloc)(size_t);
#define malloc interceptable_malloc

extern void (*intercepted_free)(void*);
#define free interceptable_free

... repeat as necessary ...

Include this header file after stdlib.h so that all calls to the functions you care about are redirected to the intercepted_... wrappers.

The implementation file is simple:

// interceptable.c
#undef malloc
void* (*intercepted_malloc)(size_t) = malloc;

#undef free
void (*intercepted_free)(void*) = free;

... repeat as necessary ...

When the LitWare library wants to intercept the functions of interest, it does this:

void* (*original_malloc)(size_t);

void* replacement_malloc(size_t size)
{
 ... replacement can call original_malloc() ...
}

void install_malloc_wrapper()
{
  original_malloc = intercepted_malloc;
  intercepted_malloc = replacement_malloc;
}

Now, when the Contoso library calls intercepted_malloc, it ends up calling replacement_malloc, which can do whatever it wants (including calling the original malloc).

Comments (9)
  1. Pierre B. says:

    Small typo: your #define declare intercept*able* instead of intercept*ed*.

  2. xcomcmdr says:

    How timely !

    I was wondering how to redirect calls to undocumented functions (like ddraw's AcquireDDThreadLock), and how DDrawCompat did it (it turns out it uses Detour Express to simply "jump" to those functions by calling the real ddraw.dll).

    I was hooking calls to functions such as DirectDrawCreate easily thanks to the types and functions defined in ddraw.h,

    As it was very early in my project, I simply did this for each function :
    - define a function with the same name in EXPORTS section of the DEF file (example : "DirectDrawCreate = DirectDrawCreateHook")
    - define a function with matching return type (from ddraw.h) that simply calls the real function imported from ddraw.h

    But for those undocumented functions I could not call them by name (AcquireDDThreadLock is not defined in ddraw.h) nor know which type they returned (again, not defined).

    As I was seeing it, Run'Time Dynamic Linking was only an answer to the "call the real function" part (of course I was wrong).

    But I wondered how people did it before Detour / EasyHook, and this give such an "of course !" solution that I'm a little ashamed I didn't think of it sooner.

    Thanks a lot !

  3. pc says:

    The wonderful book "Working Effectively with Legacy Code" by Michael Feathers describes all kinds of techniques like this (usually in a language-neutral way) for "hooking" into places and making "seams" to be able to test individual pieces of things. Usually, one can find some way to abuse a compiler to point things to your own code when the code you're testing isn't expecting things, even in a non-OO language.

  4. Adrian says:

    So many projects try to globally replace malloc and free that I'm mildly surprised the C and C++ standards committees haven't provided official hook points.

    Maybe it's because so many projects don't actually have a good reason. Library implementations and/or compiler instrumentation seem good enough for finding bugs (leaks, double-frees, pathological allocation patterns).

    The system allocators are about as efficient (and secure) as general purpose allocators can be. But many project still insist on changing the allocator for performance. The way you get better performance is to exploit something specific about your application's allocation patterns. For any non-trivial app, though, those patterns are specific to only a portion of your application, so hooking at the global level is overkill and surrenders the other benefits of the system allocator. So, sure, it might make sense to allocate all your fribblediwidgets with an arena allocator, but hijacking all allocations is fraught with peril.

    I hope that one day, the C++ compiler folks will change new and delete to use something other than malloc and free, just to screw with all the C++ developers who've replaced malloc and free.

    1. Joshua says:

      They do, it's called ld. The C standard doesn't define dynamic linking, so they expect that defining the symbol redirects callers in the standard library to it.

    2. Kevin says:

      > For any non-trivial app, though, those patterns are specific to only a portion of your application, so hooking at the global level is overkill and surrenders the other benefits of the system allocator.

      Playing devil's advocate, if you're going to reinvent the allocation wheel for part of your application, then your allocator probably wants to have some understanding of the memory usage of the rest of the application. That means hooking malloc for accounting purposes.

      But the real question is why you're reinventing the wheel in the first place. Is the performance issue really big enough to justify all that developer time, and the fragility that comes with it (see: Heartbleed, etc.)?

  5. Billy ONeal says:

    Don't be silly, Contoso would never have had something on Linux in the first place :P

  6. Peter Esik says:

    It's all fun and games until they start encountering nasty stuff, like an MSVC header where memory is allocated with _malloc_dbg, but freed with regular free (such as strstream, for example).

  7. John Wiltshire says:

    An aside on Linux, how dynamic linking is different, and LD_PRELOAD would have been fun. ;)

Comments are closed.

Skip to main content