How were window hooks implemented in 16-bit Windows?


The mechanism for keeping track of window hooks was very different in 16-bit Windows. The functions involved were SetWindowsHook, UnhookWindowsHook and DefHookProc. The first two functions still exist today, but the third one has been replaced with a macro:

// 16-bit prototype
DWORD WINAPI DefHookProc(int nCode, WPARAM wParam,
                         LPARAM lParam, HHOOK FAR *phk);

// 32-bit macro
#define DefHookProc(nCode, wParam, lParam, phhk)\
        CallNextHookEx(*phhk, nCode, wParam, lParam)

Disclaimer: All code below is "reconstructed from memory". The spirit of the code is intact, but the precise details may be off.

To install a windows hook in 16-bit Windows, you started by calling SetWindowsHook:

HHOOK g_hhkPrev;

g_hhkPrev = SetWindowsHook(WH_WHATEVER, MyHookProc);

The return value from SetWindowsHook must be saved in a global variable, which we gave the somewhat provocative name g_hhkPrev. The hook procedure itself went something like this:

// In Win16, hook procedures returned a DWORD, not an LRESULT.
DWORD CALLBACK MyHookProc(int nCode, WPARAM wParam, LPARAM lParam)
{
  if (nCode >= 0) { ... }
  return DefHookProc(nCode, wParam, lParam, &g_hhkPrev);
}

And then when you were finished, you removed the hook by calling UnhookWindowsHook:

UnhookWindowsHook(WH_WHATEVER, MyhookProc);
g_hhkPrev = NULL;

Internally, the chain of hook functions was managed as a linked list, but instead of using some internal data structure to keep track of the hooks, the linked list was managed inside the HHOOK variables themselves.

The internal implementation of SetWindowsHook was simply this:

// This array is initialized with a bunch
// of "do nothing" hook procedures.
HOOKPROC g_rgHook[NUMHOOKS];

HHOOK WINAPI SetWindowsHook(int nType, HOOKPROC pfnHookProc)
{
 HHOOK hhkPrev = (HHOOK)g_rgHook[nType];
 g_rgHook[nType] = pfnHookProc;
 return hhkPrev;
}

Installing a hook merely set your hook procedure as the head of the hook chain, and it returned the previous head. Invoking a hook was a simple matter of calling the hook at the head of the chain:

DWORD CallHook(int nType, int nCode, WPARAM wParam, LPARAM lParam)
{
 return g_rgHook[nType](nCode, wParam, lParam);
}

Each hook procedure did its work and then sent the call down the hook chain by calling DefHookProc, passing the HHOOK by address.

DWORD WINAPI DefHookProc(int nCode, WPARAM wParam,
                         LPARAM lParam, HHOOK FAR *phk)
{
 HOOKPROC pfnNext = (HOOKPROC)*phk;
 if (nCode >=0) {
  return pfnNext(nCode, wParam, lParam);
 }
 ... more to come ...
}

As you can see, it's all blindingly simple: Invoking a hook calls the first hook procedure, which then calls DefHookProc, which knows that a HHOOK is just a HOOKPROC, and it forwards the call down the chain by merely calling the next hook procedure directly.

The real magic happens when somebody wants to unhook. Recall that the rule for hook procedures is that a negative hook code should be passed straight to DefHookProc (or in modern times, CallNextHookEx). This convention allows the hook system to use negative codes to manage its own internal bookkeeping. In this case, we're using -1 as the "unhook this hook procedure" code.

BOOL WINAPI UnhookWindowsHook(int nType, HOOKPROC pfnHookProc)
{
 return DefHookProc(-1, 0, (LPARAM)pfnHookProc,
                    (HHOOK FAR*)&g_rgHook[nType]);
}

And then the real magic begins:

DWORD WINAPI DefHookProc(int nCode, WPARAM wParam,
                         LPARAM lParam, HHOOK FAR *phk)
{
 HOOKPROC pfnNext = (HOOKPROC)*phk;
 if (nCode >=0) {
  return pfnNext(nCode, wParam, lParam);
 }
 switch (nCode) {
 case -1: // trying to unhook a node
  if (pfnNext == (HOOKPROC)lParam) { // found it
   *phk = (HHOOK)pfnNext(-2, 0, 0);
   return TRUE;
  }
  // else keep looking
  return pfnNext(nCode, wParam, lParam);

 case -2: // report the next hook procedure
   return (DWORD)*phk;
 }
 return 0;
}

And there you have it, the entire window hook system in two dozen lines of code. You have to give 16-bit Windows credit for being small.

Let's walk through hook installation, dispatch, and removal to see how this all works. Suppose there is one WH_KEYBOARD hook in the system. Our variables are therefore set up like this:

// In USER
g_rgHook[WH_KEYBOARD] = Hook1;

// In HOOK1.DLL
HHOOK g_hhkPrev1 = DoNothingHookProc;

DWORD CALLBACK Hook1(int nCode, WPARAM wParam, LPARAM lParam)
{
 if (nCode >= 0) { ... work ... }
 return DefHookProc(nCode, wParam, lParam, &g_hhkPrev1);
}

Now suppose you want to install a new hook, Hook2.

// In HOOK2.DLL
HHOOK g_hhkPrev2;

g_hhkPrev = SetWindowsHook(WH_KEYBOARD, Hook2);

The SetWindowsHook function just puts your function in as the new "head" hook function and returns the old one.

// In USER
g_rgHook[WH_KEYBOARD] = Hook2;

// In HOOK2.DLL
HHOOK g_hhkPrev2 = Hook1;

DWORD CALLBACK Hook2(int nCode, WPARAM wParam, LPARAM lParam)
{
 if (nCode >= 0) { ... work ... }
 return DefHookProc(nCode, wParam, lParam, &g_hhkPrev2);
}

// In HOOK1.DLL
HHOOK g_hhkPrev1 = DoNothingHookProc;

DWORD CALLBACK Hook1(int nCode, WPARAM wParam, LPARAM lParam)
{
 if (nCode >= 0) { ... work ... }
 return DefHookProc(nCode, wParam, lParam, &g_hhkPrev1);
}

Now suppose the window manager decides it's time to fire the WH_KEYBOARD hook. It starts with CallHook which calls g_rgHook[WH_KEYBOARD] that takes us to Hook2. That hook function does its work, then calls DefHookProc(..., &g_hhkPrev2), which dispatches the hook to g_hhkPrev2 == Hook1. Similarly, the hook travels through Hook1, then DefHookProc(..., &g_hhkPrev1), where it finally reaches the DoNothingHookProc which does nothing and ends the hook chain.

Now suppose that HOOK1.DLL decides to uninstall its hook. It therefore calls UnhookWindowsHook(WH_KEYBOARD, Hook1). This starts off the hook chain with the internal hook code -1 and &g_rgHook[WH_KEYBOARD] as the first hook pointer. This activates the case -1 in DefHookProc code path, which dereferences its phk parameter and obtains g_rgHook[WH_KEYBOARD] == Hook2. Since this is not equal to Hook1, the call forwards down the chain to Hook2.

Like a good hook function, Hook2 reacts to the negative hook code by handing the call directly to DefHookProc(-1, ..., &g_hhkPrev2). This time, *phk == g_hhkPrev2 == Hook1, so the test succeeds and we dispatch the hook down the chain with a new internal code of -2, which means, "Tell me what the next hook procedure is".

This dispatch calls Hook1 which (since the notification code is negative) immediately passes the call to DefHookProc(-2, ..., &g_hhkPrev1). This now triggers the case -2 code path, which just returns *phk == g_hhkPrev1 == DoNothingHookProc. This value is returned to the DefHookProc(-1, ...) which stores the result into *phk == g_hhkPrev2; the result is that you have g_hhkPrev2 = DoNothingHookProc. Finally, DefHookProc returns TRUE to indicate that the hook was successfully uninstalled. This value is then returned out from all the nested function calls to the original caller of UnhookWindowsHook.

Observe that at the end of this unhook exercise, we get the desired result:

// In USER
g_rgHook[WH_KEYBOARD] = Hook2; // unchanged

// In HOOK2.DLL
g_hhkPrev2 = DoNothingHookProc; // updated!

DWORD CALLBACK Hook2(int nCode, WPARAM wParam, LPARAM lParam)
{
 if (nCode >= 0) { ... work ... }
 return DefHookProc(nCode, wParam, lParam, &g_hhkPrev2);
}

And Hook1 is out of the hook chain, as we desired.

This really isn't all that complicated. All we did was delete a node from a linked list. It's just that this particular linked list cannot be traversed by just dereferencing pointers. Instead, we have to issue a function call and ask the recursive function to perform the work on the "next" node for us. That's what the negative nCode values are for.

Every time I work through this exercise, I am impressed by how compactly 16-bit Windows was written. In just two dozen lines of code, we managed a linked list of function calls, including a dispatching system as well as arbitrary deletion from the middle of the linked list, and all without any memory allocation.

(And because I know people are going to try to change the topic: Remember, I'm talking about 16-bit Windows, not 32-bit window hooks.)

Next time, we'll look at one way people abused this simple system.

Comments (9)
  1. Nawak says:

    This is really clever!

    But it also ‘looks’ fragile as it relies on user functions doing the right thing in order for the whole mechanic to work!

    Since I am quite a nice guy (no, really ;) ), I am waiting for your tomorrow’s post to see what evil deeds people managed to do with that!

  2. Csaboka says:

    But it also ‘looks’ fragile as it relies on user functions doing the right thing in order for the whole mechanic to work!

    Well, if you look at it that way, the whole 16-bit Windows world was fragile since it relied on user programs playing nicely. Back in the 16-bit days, it would have been too expensive to protect against misbehaving user programs. If you wanted to bring the whole system down, all you needed to do is not yielding to the OS.

    This solution is ideal if you want to be memory efficient and you can assume user programs playing nicely together with you.

  3. How perceptive of you, Nawak. ;)

    I was rather frustrated by programming under 16-bit Windows, not because of Windows itself, but purely because other developers had chosen to implement strange custom solutions to problems that had established "correct" answers. It was reasonably common that someone would reverse-engineer the contents of a handle – which was deliberately undocumented and unsupported, because it could change at any time – and access it directly.

    In the HHOOK case, for example, someone might try to duplicate the above logic himself rather than use the standard Windows calls. If he happened to get something wrong, probably because he didn’t really understand the process, the system would behave oddly and nobody would know why. I suspect that may form the basis of Raymond’s next post.

  4. Tom says:

    I love opaque data structures.  It’s a relief not knowing how a class or data structure is implemented because otherwise you are tempted to program through the interface instead of to the interface (e.g. guess the implementation by the members of the class so you can ‘optimize’ how you call its methods).  

    I would have to say that if you are using a debugger to find undocumented members of structs/classes and relying on them being there: a) you’re doing something wrong; and b) you get what you deserve.

    I understand that Microsoft makes lots of thunks for broken software (see http://blogs.msdn.com/oldnewthing/archive/2006/01/09/510781.aspx  for more information), and I can understand why.  But it sure would be nice to say "to heck with them" when somebody pokes around internal undocumented structures and expects things to work that way forever.

    In regards to hook dispatching, I can’t imagine how expensive it is for more than one process to have a hook on a high-traffic message.  How does Win32 make this work without a ton of context switches?

  5. CN says:

    Tom: The hook function must reside in a DLL. That DLL must/will be loaded in each target process. So, in addition to message processing, you add clutter the virtual address space and some loading time. (More so if the hook function is just one function in a heavy DLL that’s then brought into every process.)

  6. swapping says:

    CN: 16-bit windows didn’t have "virtual address space", it had segmenting at best. Only the (4k) pages that is used is loaded into ram, usually not very much.

  7. Unhooking by hooking again.

  8. Chris Becke says:

    It seems an app that passed a copy of the HHOOK it originally got would defeat the unhooking mechanism. e.g.

    DWORD CALLBACK MyHookProc(int nCode, WPARAM wParam, LPARAM lParam)

    {

     …

     HHOOK hhkLocal = g_hhkPrev;

     return DefHookProc(nCode, wParam, lParam, &hhkLocal);

    }

    Why do this? Who knows: Given the lack of a user parameter I can imagine a scenario where parts of the code use their own copies of an HHOOK.

  9. ATZ Man says:

    @swapping, @DN

    Win16 enhanced mode had demand-paged virtual memory. In any mode, all running Win16 applications were in one memory space (accessed by the app through segment registers, it is true). The hook DLL would not have to be mapped into each target process, because the DLL was already jumpable into through a far call from any running application.

Comments are closed.