How to modify an application behavior when you don’t have the source


 


From time to time we need to help customers change the way an application interacts with the operating system or SDKs.  The challenge is often the access to the code.  Sometimes neither party may own the application in question and none of the parties have access to the source.   Luckily, the Microsoft Research team came up with the Detours SDK to address this problem a number of years ago and the latest version makes it easy to implement a solution to a situation like this.  In short, Detours allows you to create a DLL that hooks one or more operating system functions, so that when that function is called, the caller will actually invoke your custom Detours code instead.


 


The process is very simple:


 


·         Download the detours SDK http://research.microsoft.com/sn/detours/ and build it.


·         You can start with the SIMPLE Sample or our included sample that builds in the Visual Studio command-line environment.


·         Create a function pointer prototype for the  API you want to detour (TrueCreateFile in the example below).   It should have the same parameters and return value as the function you will detour.  As part of the declaration set the function pointer value to the real API Address.  In the following sample we will detour the CreateFile API.


·         You will also need to create your own version of the API you are detouring (ModifyCreateFile below).   In this case we are creating our own Createfile, which will call the original CreateFile with the FILE_FLAG_WRITE_THROUGH flag OR’d into the dwFlagsAndAttributes parameter.


 


static HANDLE (WINAPI * TrueCreateFile)(LPCTSTR lpFileName, DWORD dwDesiredAccess, DWORD dwShareMode, LPSECURITY_ATTRIBUTES lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile) = CreateFile;


 


HANDLE WINAPI ModifyCreateFile(LPCTSTR lpFileName,  DWORD dwDesiredAccess, DWORD dwShareMode,


    LPSECURITY_ATTRIBUTES   lpSecurityAttributes, DWORD dwCreationDisposition, DWORD dwFlagsAndAttributes, HANDLE hTemplateFile)


{


    dwFlagsAndAttributes |= FILE_FLAG_WRITE_THROUGH;


    return TrueCreateFile(lpFileName, dwDesiredAccess,  dwShareMode, lpSecurityAttributes, 


           dwCreationDisposition,      dwFlagsAndAttributes,  hTemplateFile);     


}


 


·         You will need to write your detour code in the DLLmain of your dll.   This should be executed when your DLL loads and dwReason is == DLL_PROCESS_ATTACH.   In our call to DetourAttach we pass our TrueCreateFile pointer (the real CreateFile address),  and the address of ModifyCreateFile (our custom create file api).  The detour API handles the intercept for us. 


        DetourRestoreAfterWith();


        DetourTransactionBegin();


        DetourUpdateThread(GetCurrentThread());


        DetourAttach(&(PVOID&)TrueCreateFile, ModifyCreateFile);


        DetourTransactionCommit();


 


·         When the DLL_PROCESS_DETACH happens you will need to clean up the detour and unhook the real API.


 


        DetourTransactionBegin();


        DetourUpdateThread(GetCurrentThread());


        DetourDetach(&(PVOID&)TrueCreateFile, ModifyCreateFile);


        DetourTransactionCommit();


 


So how do you get the DLL loaded into the target process?  There are a couple ways.  I recommend using the setdll tool that comes as part of the Detour SDK.  In the following case we are modifying NTBackup to automatically load our detoured DLL when NTbackup runs. 


 


C:\test>setdll /d:nocache.dll ntbackup.exe


Adding nocache.dll to binary files.


  ntbackup.exe:


    nowritethru.dll


    MFC42u.dll -> MFC42u.dll


    msvcrt.dll -> msvcrt.dll


    ADVAPI32.dll -> ADVAPI32.dll


    KERNEL32.dll -> KERNEL32.dll


    GDI32.dll -> GDI32.dll


    USER32.dll -> USER32.dll


    ntdll.dll -> ntdll.dll


    COMCTL32.dll -> COMCTL32.dll


    SHELL32.dll -> SHELL32.dll


    MPR.dll -> MPR.dll


    comdlg32.dll -> comdlg32.dll


    NETAPI32.dll -> NETAPI32.dll


    RPCRT4.dll -> RPCRT4.dll


    ole32.dll -> ole32.dll


    SETUPAPI.dll -> SETUPAPI.dll


    USERENV.dll -> USERENV.dll


    NTMSAPI.dll -> NTMSAPI.dll


    CLUSAPI.dll -> CLUSAPI.dll


    query.dll -> query.dll


    sfc_os.dll -> sfc_os.dll


    SYSSETUP.dll -> SYSSETUP.dll


    OLEAUT32.dll -> OLEAUT32.dll


    VSSAPI.DLL -> VSSAPI.DLL


 


Note that if you modify a binary that is protected by Windows File Protection the modified binary will be replaced by the OS with the original binary.  I recommend keeping your modified version in another directory so it does not get replaced.


 


Click here for the sample source code and Makefile, which will build from a Visual Studio command prompt.


 


Jeff Dailey

Comments (3)

  1. neongreen says:

    Well Thanks a lot for writing this post…it’s amazing and you might say even enlightening for me..cause I do some programming..mostly <a href="http://www.notionsolutions.com">vb.net</a&gt;…but have never even heard that one could modify an application behavior without the source –didn’t think that was possible…but now I know better :)…thx again..

  2. stephc_msft says:

    For info

    The ‘shimming’ available from the Application Compatibility Toolkit can also be used to modify application behaviour.

    But it can only catch and change certain things. If want to catch something more extensively then detours, as in the example above, is the way to go