How to modify an application behavior when you don't have the source

From time to time we need to help customers change the way an application interacts with the operating system or SDKs. The challenge is often the access to the code. Sometimes neither party may own the application in question and none of the parties have access to the source. Luckily, the Microsoft Research team came up with the Detours SDK to address this problem a number of years ago and the latest version makes it easy to implement a solution to a situation like this. In short, Detours allows you to create a DLL that hooks one or more operating system functions, so that when that function is called, the caller will actually invoke your custom Detours code instead.

The process is very simple:

· Download the detours SDK https://research.microsoft.com/sn/detours/ and build it.

· You can start with the SIMPLE Sample or our included sample that builds in the Visual Studio command-line environment.

· Create a function pointer prototype for the API you want to detour (TrueCreateFile in the example below). It should have the same parameters and return value as the function you will detour. As part of the declaration set the function pointer value to the real API Address. In the following sample we will detour the CreateFile API.

· You will also need to create your own version of the API you are detouring (ModifyCreateFile below). In this case we are creating our own Createfile, which will call the original CreateFile with the FILE_FLAG_WRITE_THROUGH flag OR’d into the dwFlagsAndAttributes parameter.

static HANDLE (WINAPI * TrueCreateFile)(LPCTSTR lpFileName, DWORD dwDesiredAccess, DWORD dwShareMode, LPSECURITY_ATTRIBUTES lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile) = CreateFile;

HANDLE WINAPI ModifyCreateFile(LPCTSTR lpFileName, DWORD dwDesiredAccess, DWORD dwShareMode,

    LPSECURITY_ATTRIBUTES lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile)

{

    dwFlagsAndAttributes |= FILE_FLAG_WRITE_THROUGH;

    return TrueCreateFile(lpFileName, dwDesiredAccess, dwShareMode, lpSecurityAttributes,

           dwCreationDisposition, dwFlagsAndAttributes, hTemplateFile);

}

· You will need to write your detour code in the DLLmain of your dll. This should be executed when your DLL loads and dwReason is == DLL_PROCESS_ATTACH. In our call to DetourAttach we pass our TrueCreateFile pointer (the real CreateFile address), and the address of ModifyCreateFile (our custom create file api). The detour API handles the intercept for us.

        DetourRestoreAfterWith();

        DetourTransactionBegin();

        DetourUpdateThread(GetCurrentThread());

        DetourAttach(&(PVOID&)TrueCreateFile, ModifyCreateFile);

        DetourTransactionCommit();

· When the DLL_PROCESS_DETACH happens you will need to clean up the detour and unhook the real API.

        DetourTransactionBegin();

        DetourUpdateThread(GetCurrentThread());

        DetourDetach(&(PVOID&)TrueCreateFile, ModifyCreateFile);

        DetourTransactionCommit();

So how do you get the DLL loaded into the target process? There are a couple ways. I recommend using the setdll tool that comes as part of the Detour SDK. In the following case we are modifying NTBackup to automatically load our detoured DLL when NTbackup runs.

C:\test>setdll /d:nocache.dll ntbackup.exe

Adding nocache.dll to binary files.

  ntbackup.exe:

    nowritethru.dll

    MFC42u.dll -> MFC42u.dll

    msvcrt.dll -> msvcrt.dll

    ADVAPI32.dll -> ADVAPI32.dll

    KERNEL32.dll -> KERNEL32.dll

    GDI32.dll -> GDI32.dll

    USER32.dll -> USER32.dll

    ntdll.dll -> ntdll.dll

    COMCTL32.dll -> COMCTL32.dll

    SHELL32.dll -> SHELL32.dll

    MPR.dll -> MPR.dll

    comdlg32.dll -> comdlg32.dll

    NETAPI32.dll -> NETAPI32.dll

    RPCRT4.dll -> RPCRT4.dll

    ole32.dll -> ole32.dll

    SETUPAPI.dll -> SETUPAPI.dll

    USERENV.dll -> USERENV.dll

    NTMSAPI.dll -> NTMSAPI.dll

    CLUSAPI.dll -> CLUSAPI.dll

    query.dll -> query.dll

    sfc_os.dll -> sfc_os.dll

    SYSSETUP.dll -> SYSSETUP.dll

    OLEAUT32.dll -> OLEAUT32.dll

    VSSAPI.DLL -> VSSAPI.DLL

Note that if you modify a binary that is protected by Windows File Protection the modified binary will be replaced by the OS with the original binary. I recommend keeping your modified version in another directory so it does not get replaced.

Click here for the sample source code and Makefile, which will build from a Visual Studio command prompt.

Jeff Dailey