Don’t forget to unregister your window classes when your DLL shuts down dynamically

If your DLL is unloaded dynamically, you need to make sure you have unregistered your window classes. (You can tell whether the DLL_PROCESS_DETACH is due to a dynamic unload or whether it's due to process termination by checking the lpReserved parameter to your DllMain function.) If you forget to unregister your window classes, all sorts of bad things can happen:

First, if you registered any of those classes as a CS_GLOBALCLASS, then people will still be able to create a window of that class by passing its class name to the CreateWindowEx function (or any other function that leads to CreateWindowEx). Since your DLL is no longer in memory, the moment it receives a window message (like, say, WM_NCCREATE), the process will crash since the window procedure has been unloaded. This manifests itself in crashes with the instruction pointer in no-man's land—these are typically not easy to debug, and the Windows error reports that are generated by these crashes won't even be assigned to your DLL since your DLL is long gone.

Second, even if you registered the classes as private classes, you are still committing namespace pollution, leaking the class into a namespace that you no longer own. If another DLL gets loaded at the same base address as your DLL (thereby receiving the same HINSTANCE, it inherits this dirty namespace. If that DLL wants to register its own class that happens to have the same name as the class you leaked, its call to RegisterClassEx will fail with ERROR_CLASS_ALREADY_EXISTS. This typically leads to the DLL failing to initialize or (if the problem is not detected) an attempt to create a window of that class creating instead a window of your leaked class, with a window procedure whose address now resides somewhere in the middle of this new DLL. This is even worse than an instruction pointer in no-man's land; instead, control goes to a random instruction in the new DLL and probably will manage to execute for a little while before finally keeling over. What's worse, not only does the crash not get reported against your DLL (which is no longer in memory), but it gets erroneously reported against the new DLL since it is the new DLL's code that was executing when the crash finally occurred. Congratulations, you just created work for somebody you never met. Those poor victims are going to be scratching their heads trying to figure out how control ended up in the middle of a totally random function with completely nonsense values on the stack and in the registers.

Third, the namespace you pollute can be your own. Suppose you registered a class as a CS_GLOBALCLASS, then your DLL gets unloaded and you forget to unregister the class. Later, your DLL gets reloaded, but due to changes in the virtual address map, it gets loaded at a new address. Now your DLL attempts to re-register its CS_GLOBALCLASS classes, and the call fails with ERROR_CLASS_ALREADY_EXISTS. If you're lucky, your DLL detects the error and fails to load, resulting in missing functionality. If you're unlucky, you fail to detect the error and succeed the load anyway. Then the code that did the LoadLibrary will try to create a window with that class, but instead of getting your DLL's window class (which failed to register), it gets the window class left over by that first copy of your DLL! Since that DLL no longer exists, you get a crash with the instruction pointer off in no-man's land.

This is not a purely theoretical problem. The shell common controls library contained this bug of neglecting to unregister all its classes when dynamically unloaded, and we had to issue a hotfix because the crashes caused by it were actually occurring on real users' machines. Don't be the one responsible for having to issue a hotfix for your product. Unregister your classes if the process is going to continue running after your DLL unloads. Because it's the right thing to do.

(Now, you might notice that this goes against the rule of not calling out to other DLLs during your DLL_PROCESS_ATTACH. The solution for this is to have a "cleanup" function that people must call before calling FreeLibrary on your library to balance the "initialization" function that they had to call to register your control classes. On the other hand, if you failed to plan ahead for this, such as the shell common control did with its InitCommonControlsEx function without a matching UninitCommonControls function, then you have to decide between the lesser of two evils.)

Comments (11)
  1. steveg says:

    Don’t be the one responsible for having to issue a hotfix for your product.

    Or you’ll be sitting all alone in the cafeteria after being shamed (but not named) right here. I feel for your pain, whomever you are.

    BTW Is Uninit really the opposite of Init?

  2. Niko says:

    Why Windows does not automatically unregisters them when a process terminate?

    Sure, the programmer should always ensures he close all resources (memory, handles, etc) he allocate in his program. But shouldn’t Windows automatically free these resources when the process terminate if the programmer has not done it?

  3. Niko says:

    Forget my previous comment. I missed the important part: "when the dll is dynamically unloaded"

  4. Norman Diamond says:

    Thank you very much for every bit you wrote here.  Also thanks to whoever permitted the hotfix to be released.

  5. me says:

    Citing: "and the Windows error reports that are generated by these crashes won’t even be assigned to your DLL since your DLL is long gone."

    In kernel mode, WinDBG shows the addresses of drivers already unloaded. If you are testing your own driver (and reloading it some times), you might even so it 6 or 7 times in the list, so you can recognize where the bug comes from.

    This info is even written into a crash dump, at least on Win 2000 and higher.

    Why can’t the reporting tools do similar for the user space? Wouldn’t this make sense for this type of crash?

  6. Tom M says:

    I had a series of DLLs with different CAxWindow based controls which caused a crash when you unloaded one and then loaded another. This only happened when statically linked with ATL, and didn’t occur when you dynamically linked with it. Turns out it was a bug in ATL because ATL does not unregister the window classes that it registers, at least when statically linked.

  7. Vipin says:

    Fundamentally, the problem here is with the window proceedure for a custom control class which will be pointing to a dangling memory when the dll gets unloaded. Microsoft could have solved this had their FreeLibrary(…) api implementation looked up in its internal data structures and figured out these window classes are going to have their wndprocs dangling post dll unload and hence need to kick them off, in fact unregister them.

    If you look at the UnregisterClass documentation:-

    It says this "Windows 95/98/Me: All window classes registered by a dynamic-link library (DLL) are unregistered when the DLL is unloaded."

    "Windows NT/2000/XP: No window classes registered by a DLL registers are unregistered when the .dll is unloaded. "

    What brings in the difference for NT, only Microsoft will know.

    Also another problem is let us say I create a window with a class registered a dll, then I have this window alive when I do a FreeLibrary of the dll. We are going to get the same crash, in fact an unavoidable crash.

  8. Kevin Eshbabch says:

    I also wondered why there was no way to uninitialize the common controls library and this finally explains it!  Somebody forgot to include this functionality.

  9. David Levins says:

    DLL function call crashed

  10. David Levins says:

    DLL function call crashed

Comments are closed.

Skip to main content