Some reasons not to do anything scary in your DllMain, part 3

In the same week, the shell team was asked to investigate two failures.

The first one was a deadlock in Explorer. The participating threads look like this:

  • Thread 1 called Free­Library on a shell extension as part of normal Co­Free­Unused­Libraries processing. That DLL called Ole­Uninitialize from its Dll­Main function. This thread blocked because the COM lock was held by thread 2.
  • Thread 2 called Co­Create­Instance, and COM tried to load the DLL which handles the object, but the thread blocked because the loader lock was held by thread 1.

The shell extension caused this problem because it ignored the rule against calling shell and COM functions from the Dll­Main entry point, as specifically called out in the Dll­Main documentation as examples of functions that should not be called.

The authors of this shell extension may never have caught this problem in their internal testing (or if they did they didn't understand what it meant) because hitting this deadlock requires that a race window be hit: The shell extension DLL needs to be unloaded on one thread at the exact same moment another thread is inside the COM global lock trying to load another DLL.

Meanwhile, another failure was traced back to a DLL calling Co­Initialize from their Dll­Main. This extra COM initialization count means that when the thread called Co­Uninitialize thinking that it was uninitializing COM, it actually merely decremented the count to 1. The code then proceeded to do things that are not allowed in a single-threaded apartment, believing that it had already torn down the apartment. But the secret Co­Initialize performed by the shell extension violated that assumption. Result: A thread that stopped responding to messages.

The authors of both of these shell extensions seemed be calling Co­Uninitialize/Ole­Uninitialize in order to cancel out a Co­Initialize/Ole­Initialize which they performed in their DLL_PROCESS_ATTACH. This is fundamentally unsound not only because of the general rule of not calling COM functions inside Dll­Main but also because OLE initialization is a per-thread state, whereas the thread that gets the DLL_PROCESS_DETACH notification is not necessarily the one that receives the DLL_PROCESS_ATTACH notification.

It so happens that in the second case, the DLL in question was a shell copy hook, and the hang was occuring not in Explorer but in an application which was using SH­File­Operation to delete some files. We could at least advise the application authors to pass the FOFX_NO­COPY­HOOKS flag to IFile­Operation::Set­Operation­Flags to prevent copy hooks from being loaded.

Previous articles in this series: Part 1, Part 2.

Comments (26)
  1. Joshua says:

    1. Don't screw around in DllMain.

    2. Don't call shell functions for automatic file operations. Shell plugins from non-GUI threads hate your guts.

  2. Mark says:

    Joshua: suggestion 2 is nonsense. That's not the conclusion to make from this article.

  3. skSdnW says:

    Why did these extensions even need to call Co­Initialize in the first place? COM should already be initialized by the host process before calling Co­Create­Instance to load the extension (Unless they are running on Win95 and the extension is registered to run without COM).

  4. Joshua says:

    @Mark: I take it you've never had a window open on the service session because some shell plugin decided it wanted to prompt about something.

  5. Jim Lyon says:

    1. Don't screw around in DllMain.

    2. Don't call CoInitialize / CoUninitialize on any thread that you don't own.

  6. Mark says:

    Joshua: the answer's in the post – FOFX_NO­COPY­HOOKS.

  7. MV says:

    The funny thing is, I've never in my life had any reason (that I know of) to implement a DllMain at all – literally not even once. I like to keep things simple, which means I avoid COM like the plague, so that's probably part of it.  The more of these articles I read, the more convinced I am that avoiding COM is the right thing to do.

  8. Roger says:

    In older times there were checked builds of Windows where developers could in theory find out about problems in their apps.  What is the current best practise for this sort of thing?  It looks like checked builds no longer happen, not that too many used them anyway.  The Windows Application Verifier also seems to not be too current.  Would it have caught this issue?

  9. Joshua says:

    @MV: The only thing I've needed to do is save HINSTANCE or possibly allocate a DLL heap if GetProcessHeap() was unsuitable for some reason.

  10. Behodar says:

    @Roger: There are still checked builds in MSDN Subscriber Downloads.

  11. Ben Voigt says:

    @MV, I suspect you mean you've never written a fake DllMain.  Almost certainly your toolchain implements the real DLL entry-point (which has no name; it's identified by its address stored in the PE header), which in addition to calling the function having the name DllMain which you chose not to provide, also performs construction and initialization of global variables.

  12. Mark S says:

    As a primarily managed-code developer who has nonetheless read this blog for many years and seen lots of "what *not* to do in DllMain", I'd like to ask: broadly speaking, what kinds of things *is* it appropriate to do in DllMain?

    [Initialize critical sections, allocate TLS slots, allocate memory, that's about it. -Raymond]
  13. cheong00 says:

    @MV: Completely agrees that when writing small tools, static linking makes more sense than dynamic linking.

    However, talking about "not linked to .NET runtimes in order to avoid DLL hell" as advantage is silly, it sounds as if you don't want to call Windows system DLL altogether.

  14. alegr1 says:


    Creating a thread is not a problem. Properly shutting it down is.

  15. MV says:

    @Ben, yes, I'm sure you're right.  I'm mostly talking C++, so there's definitely some initialization that has to happen when the DLL is loaded. But the toolchain handles that – and handles it correctly every time with zero effort from me, like a good toolchain should (OK, there's the order of global initialization problem, but that's not specific to DLLs, it happens just as much with EXEs).  But there's never been anything that *I* wanted to do when one of my DLLs was being loaded or unloaded.

    Actually, the last major project I designed, we avoided DLLs completely – we just compiled the common stuff into LIBs and then linked them directly into the EXEs.  Sure, the EXEs were consuming excess disk space, and excess physical memory (because the common code couldn't be shared between two processes like DLLs can), but all of my EXEs together were smaller than the .NET runtime, so who cares?  And it was *SO* worth it to stay completely out of DLL hell!

  16. In rare cases you might also want to create a thread.  (Which *is* safe, provided you remember that the thread won't start until DllMain has exited.)

  17. Neil says:

    @Ben Voigt The default CRT entry point does have a name, otherwise the linker wouldn't know where it was. The default name is chosen by the linker, but you can override it with /ENTRY (or /NOENTRY for a resource-only DLL).

  18. MV says:

    @cheong00: it was a pure C++ app (the architecture of which was cloned from a prior app which predates .NET and is still running strong), so there was really no need for .NET at all.  Most of the EXEs were 10-20MB, and they had ZERO dependencies – just drop the EXEs onto any pc running Win2K or higher, and they'd work.

    Every. Single. Time.  No code changes when Vista came out.  None for 7.  None for 8.  It was SOOOO nice.

  19. John Doe says:

    @MV, that sounds exactly like a huge Hello World to me.

  20. Gabe says:

    It sounds like DllMain is a lot like Unix signal handlers in this respect: It's a very attractive place to put fancy code yet there is very little that you can do safely. Since so few people understand that, they write all kinds of unsafe things in their DllMain (like invoke COM) or signal handlers (like allocate memory). And they get away with it too, because the race windows are so hard to hit.

  21. Cesar says:

    @Raymond: It would be *great* if there was a precise list of "safe to call in DllMain" Win32 API functions. That would make things simple: if you are calling anything else in your DllMain, you are wrong.

    @MV: One useful thing to do in a DllMain: DisableThreadLibraryCalls(). Makes every CreateThread() in the whole process faster.

  22. Azarien says:

    0. don't use DllMain. It's borken beyond repair and you never know if your DllMain breaks one of gazillion DllMain don'ts.

  23. Kevin Eshbach says:

    @Mark S

    Use the DisableThreadLibraryCalls function.

  24. John Doe says:

    @Riggs, the problem is that the other thread will be running your DLL's code, how do you kill it?  TerminateThread will leave some things in an inconsistent state, and signaling the thread and waiting for it to exit is prone to deadlocking with the dreaded loader lock (or whatever it actually is).

Comments are closed.