How can I investigate the possibility of a lot of leaked window classes (RegisterClass)?


A customer reported that they are seeing intermittent failures from the Register­Class function with the error ERROR_NOT_ENOUGH_MEMORY on some systems, typically after their program and other programs related to their program have been running for some time. They suspect that there is a leak somewhere caused by failure to unregister window classes, but they need help tracking this down.

We thought we could inspect the system to see how many classes have been registered indirectly via the following pseudocode:

Foreach running processId
  Foreach atom in (0xC000 .. 0xFFFF)
    If (GetClassInfo(HInstanceFromProcessId(processId), atom))
      Inspect the returned class name

But when we run this, we get a constant set of used atoms for all processes, and we can't see the class name. Is there something else that we can use to help track down this leak?

Okay, there are a few things wrong with the pseudocode above.

First of all, instance handles are meaningful only in the context of a process, and unless a specific process is specified (which is very rare), they are interpreted as in the context of the current process. So the call to HInstance­From­Process­Id is not all that useful because it gives you an instance handle you can't use. What you end up doing is taking the instance handle from that other process and then checking if anybody in the current process with the same instance handle has registered a class with the specified atom.

Second, the Get­Class­Info function gets information about the class which is registered against the provided instance/name pair. In order to get through all the classes that belong to a process, you have to enumerate through not only all the atoms, but also all the instance handles. And you have to include all the instances of modules that have ever been loaded (even if they were subsequently unloaded). This is not the sort of thing you can easily brute-force your way through, seeing as a module can be loaded at almost any 64KB boundary.

Fortunately, all is not lost.

It so happens that in current versions of Windows, registered class names, registered clipboard format names, and registered window message names all come from the same atom table. I wish to reiterate that this is an implementation detail which can change at any time, so don't take any dependencies on this. I provide this information for diagnostic purposes, which is what we have here.

The customer can do this once they encounter the problem:

Foreach atom in (0xC000 .. 0xFFFF)
  If (GetClipboardFormatName(atom, buffer, bufferSize))
      Print buffer

This will print out the names of all the classes, clipboard formats, and registered window messages. There is room in the atom table for 16,384 atoms, and in practice there aren't more than a hundred or so, so if you see more than 15,000 entries, that's a really good sign that you're leaking classes.

Leaking classes is generally hard to do because the class name is typically something you hard-code into your program, and it takes a lot of work to type 15,000 different strings into your progam (much less keep track of them all). Since you are unlikely to have 15,000 different window procedures in your program, you probably don't need 15,000 different classes.

There are some frameworks which generate class names programmatically, and it might be one of those frameworks that is the source of the large amount of class names (and the consequent leaking thereof).

The customer wrote back: "Thanks! That showed us exactly what was leaking: Tons of HwndWrapper[OurApp;;guid] classes, with all different GUIDs. It turns out that we were creating a dispatcher object on a background thread. Creating a dispatcher registers the class, which leaks when the background thread terminates."

The customer continued that they were able to fix the leak by using a synchronization context so that the object is created on the UI thread. One of my colleagues provided an alternative solution of calling Dispatcher.Begin­Invoke­Shutdown or Dispatcher.Invoke­Shutdown before terminating the background thread. That will shut down the dispatcher cleanly (instead of ripping the thread out from under it), which will destroy the window and unregister the window class.

Comments (9)

  1. Darran Rowe says:

    This is possibly one of those “time machine” situations where perhaps it would be nice if there was something like EnumWindowsClasses.

    1. Yuhong Bao says:

      Which also reminds me of window hooks. WIN31WH.HLP mentions that “Windows 3.1 no longer allows applications and DLLs to enumerate all the functions in a hook chain. In particular, Windows 3.1 no longer supports the HC_GETLPLPFN, HC_LPLPFNNEXT, and HC_LPFNNEXT values for the DefHookProc function. “

  2. Neil says:

    Along similar lines, I’ve always wondered why, having pointed out that RegisterWindowClass and RegisterClipboardFormat have the same ordinal in Windows 3.x, the book Undocumented Windows never made the leap to using GetClipboardFormatName to find out the name of a registered window message.

    1. Neil says:

      Sorry I meant RegisterWindowMessage of course…

  3. Andrew Rogers (ex-MSFT) says:

    Darran just asked for “time travel” debugging above – your wish is Microsoft’s command! There has just been a public release of the “time travel” debugging technology that’s been available internally for some years (you can find a few references to “iDNA” peppered on the web):

    https://blogs.msdn.microsoft.com/windbg/2017/09/25/time-travel-debugging-in-windbg-preview/

    It’s great to see this being shared with the community: there are some caveats though –

    * The trace files generated by it are huge, so using it over a long test may become impractical
    * It only records user-mode code

    That said, it’s a lot of fun to work with, and I’m glad that everyone will be able to get the benefit of working with it now.

  4. MacIn173 says:

    There’s some AtomMonitor tool available for it. We’ve used it once when our system failed because it got run out of atoms.

  5. ErikF says:

    I’m happy that the customer followed up with you. This is a refreshing change from many of your customer stories where they never bother to do that!

  6. Dan says:

    For better or worse, I have seen this pattern many times, as well as a related one in a .NET HWNDWRAPPER class. This is especially problematic in services, where you can easily deplete the desktop heap long before you run out of ATOMs. This could have been avoided if the code registering the class used a per-instance wndproc instead of a per-instance wndclass.

    1. Dan says:

      Quick correction: I meant NotificationWindow, not HWNDWRAPPER, which was called out in this blog. But both contain the same cause, which in my opinion, is a simple bug but not worth the risk of fixing with so many applications potentially dependent on a less than ideal behavior.

Skip to main content