What is the HINSTANCE passed to SetWindowsHookEx used for?


The SetWindowsHookEx function accepts a HINSTANCE parameter. The documentation explains that it is a handle to the DLL containing the hook procedure. Why does the window manager need to have this handle?

It needs the handle so it knows which DLL to load into each process when the hook fires. It injects the DLL you pass, then calls the function you pass. Clearly the function needs to reside in the DLL you pass in order for its code to be there when the window manager calls it.

This is also why hook functions must reside in DLLs. You can't load an EXE into another EXE.

The WH_KEYBOARD_LL and WH_MOUSE_LL hooks are exceptions to this rule. These two are non-injecting hooks, as explained in their respective documentation pages. Rather, the hook function is called in its original thread context.

Okay, armed with this information, perhaps you can solve this person's problem with global hooks.

Comments (20)
  1. Anonymous Coward says:

    Sounds like -h- thinks his DLL is running in his process, when it’s actually running in the hooked program’s process…

  2. Speaking of hooks, does any know if the HHOOK parameter passed to CallNextHookEx is manditory or not? The MSDN documentation indicates it’s required, but 99% of the sample hook code I’ve seen they just pass NULL. Thanks.

  3. Ben Cooke says:

    Just the other day I was writing a program which used WH_KEYBOARD_LL. I did it with an external DLL because that’s what I’d always done before. Nice to know it didn’t have to be so hard. ;)

    I did see something along those lines in the documentation while I was doing it, but I was being a cocky programmer and assuming I already knew it all. Lesson learned, I think.

  4. Daev says:

    OK, the WH_KEYBOARD_LL and WH_MOUSE_LL hooks perform a context switch AND a message-loop negotiation every time there’s a keyboard or mouse event. There are an awful lot of WM_MOUSEMOVE events. How does the system avoid thrashing?

  5. Anonymous says:

    Is it because the hInstance he is passing is actually the module instance of his EXE, and not his actual DLL?

  6. Brian, the current CallNextHookEx documentation says the HHOOK argument is ignored.

  7. Thanks Doug, I guess it’s time to install a newer version of the MSDN. :)

    Really this just raises further questions. The older documentation clearly states you pass the handle returned from SetWindowsHookEx. So either:

    1) Older OS like 9x and NT require this handle but the newer MSDN assumes you’re coding for only newer OSs.

    2) The original documentation was wrong and this parameter has always been ignored.

    Ugh, what’s a programmer to do?

  8. oldnewthing says:

    Pass it anyway. It doesn’t hurt.

    This is what happens when MSDN starts documenting implementation instead of contract. The contract is "pass the hook handle". Whether any particular implementation requires it is irrelevant to writing correct code. You write code to the contract, not to the implementation.

  9. Gordo says:

    Has anyone tried playing with some global hooks (like WH_SHELL) from managed code? Our team loves to send messages around via the Windows message service (the UDP one, not messenger!). Even though I keep reminding them that it’s 2005 and not 1995, they persist. I thought I’d try to write a nice WH_SHELL hook to listen for window created messages, scrape the screen, and send me an email with the message instead.

    Even with an unmanaged DLL to solve the HINSTANCE problem, I ran into lots of excitement trying to pin the return address of my callback function. Eventually I gave up, and went back to cursing and swearing every time a message pops up when I’m coding, and my return key closes it before I can read it.

    Where is my managed user32!?

  10. A says:

    Pass it anyway. It doesn’t hurt.

    But only the process that called SetWindowsHookEx has a handle. When a hook DLL is injected into another process, it’s not given any handle, AFAIK. So what is there for the DLL to pass other than NULL?

  11. Really this just raises further questions. The older documentation clearly states you pass the handle returned from SetWindowsHookEx. So either:

    1) Older OS like 9x and NT require this handle but the newer MSDN assumes you’re coding for only newer OSs.

    2) The original documentation was wrong and this parameter has always been ignored.

    <

    My bet is on (2). It’s nice to relax this requirement. It does away with the need to share the HHOOK between processes. It also seems to avoid a race condition in which one process uninstalls the hook while another goes on to CallNextHookEx(dead_hook_handle). I was never clear as to whether that was a real issue, and if it was, how to overcome it.

    >Ugh, what’s a programmer to do? <

    In this case, I’m content to listen to the documentation. :)

  12. Tim Smith says:

    > You write code to the contract, not to the implementation.

    Boy this is a drum I beat constantly at work. I just laugh at all the people who outsmart themselves by programming around ‘issues’ in the operating systems. Then when a new version comes out, their software fails.

    I also get onto people who design API based on the problem at hand. They don’t think about the bigger picture and how well rounded the contract might be. *sigh*

  13. The DLL can install the hook itself and store the returned handle in shared memory:

    #pragma data_seg(".SHARED")

    /* Shared between all instances of DLL */

    HHOOK hook = NULL;

    #pragma data_seg()

    #pragma comment(linker, "/section:.SHARED,RWS")

    You can protect the hook installation with a mutex.

  14. >The DLL can install the hook itself and store the returned handle in shared memory:

    #pragma data_seg(".SHARED")

    /* Shared between all instances of DLL */

    HHOOK hook = NULL;

    #pragma data_seg()

    #pragma comment(linker, "/section:.SHARED,RWS")

    <

    Yes, that’s the usual way it’s done.

    >You can protect the hook installation with a mutex. <

    You’d have to protect the CallNextHookEx call and hook uninstallation. It appears to me that for a CallNextHookEx that uses its HHOOK parameter, one shouldn’t want process X to uninstall the hook while process Y has just entered the hook function and is about to call CallNextHookEx. Wrapping that call in a mutex makes me nervous, so I’ve never done it, and this is partly why I was happy to read the function ignores its HHOOK parameter. I’ve never seen any code that uses a mutex to protect the HHOOK, nor have I seen this race condition discussed, but AFAICT, it would exist for a CallNextHookEx that uses its HHOOK parameter, and it would have to be accounted for in some way.

  15. DL says:

    Even if you use shared memory to share the HHOOK handle across processes, the handle itself would have to be global (e.g. consider sharing a file handle in a similar way. Obviously it wouldn’t work as the handle is per process).

    Treating the handle as global would have all sorts of security implications… for example, malicious processes could brute-force window hook handles and use them to "inject" data to other "hookers" in the chain, etc.

    With that said, it’s obvious that the HHOOK parameter was in fact *never* used by the system in Win32. My guess is that it’s another legacy from good-ol’ Win16.

    And finally, a question: Any reason why the SetWindowsHookEx function receives its argument as HMODULE, and not as LPCWSTR (i.e. dll filename)? Obviously, the first thing the SetWindowsHookEx implementation does is to convert the handle back to a filename!

  16. an0nym0us says:

    You write code to the contract, not to the implementation.

    A contract that is not enforced leads to app-compat issues once the implementation starts enforcing it.

  17. Because there is nowhere to inject them into.

  18. Because there is nowhere to inject them into.

Comments are closed.