The NT DLL Loader: DLL_PROCESS_ATTACH reentrancy – step 1 – LoadLibrary()

So what happens if you call back into the loader when you’re inside a loader callout (DllMain) for DLL_PROCESS_ATTACH?

I’ll be addressing teardown (DLL_PROCESS_DETACH) after completing the DLL_PROCESS_ATTACH series.

The first issue is: what about LoadLibrary()?  I’ll address GetProcAddress() and FreeLibrary() later.

We already know how LoadLibrary() works.  It walks the dependency graph starting from the DLL that is being loaded and trims away any portions already in the loader’s tables.  It then effectively adds that to the initialization order list and (this is what varies) if you’re not using the loader reentrantly, calls the initializers.

If you are calling into the loader reentrantly, the list of DLLs to initialize is extended but (I should double check the code to make sure I’m not lying; it’s been a couple of years…) only the outer-most invocation of the loader will complete the initialization sequence.

Thus, you call LoadLibrary(“a.dll”).  In a.dll’s DLL_PROCESS_ATTACH, it calls LoadLibrary(“b.dll”).  b.dll’s initializer is added to the init list but it isn’t actually initialized at that time.  Instead, the inner LoadLibrary() just returns and then the DLL_PROCESS_ATTACH returns (successfully for the sake of argument) and then the original stack frame for the original LoadLibrary() initializes b.dll.

Clearly things are a little more complicated since it might not be a.dll that actually calls loadlibrary(); it could be a static import of A which does so and thus the init list may still have uninitialied entries on it at the time that the entries for b.dll (and its static imports!) are appended.

So far so good; assuming that the cycles don’t break you, this was a fine solution to the problem which also avoids arbitrary recursion due to nested dynamic DLL loads.  This was mostly harmless!

Comments (7)

  1. Wound says:

    The question then is: what does LoadLibrary return if the library hasn’t actually been loaded, and what are you calling GetProcAddress on? In your example I would expect GetProcAddress to fail, but DllMain to return TRUE, even though "SomeFunction" was not called.

  2. Graham Harper says:

    I would expect the library has been mapped into the virtual address space of the process and you do get a valid base address back as indicated by the HMODULE. That doesn’t mean the module has been initialized however.

    It’s funny because GetProcAddress’s contract with the caller is only that of providing an address into the module from a symbolic name. It can’t (and won’t) guarantee the module is in a suitable state to run the method pointed to.

    But i could be wrong………

  3. mgrier says:

    Sorry, Graham, you were in fact wrong. I do say "implied" because that’s what it happens to do and also because if it didn’t run the initializers in the "simple" cases, using the pointer you got back from GetProcAddress() wouldn’t be useful anyways.

    Eventually I’m going to get to recommendations of what to do in your DLL_PROCESS_ATTACH (basically nothing!) and how to live with the consequences.

  4. Graham Harper says:

    Sorry I was being really thick!

    What your saying makes an awful lot of sense now.

    It effectivley becomes Just-In-Time initialization of the module as LoadLibrary cannot always guarentee intialization of the target and it’s dependencies?

    If A.DLL calls the LoadLibraryGetProcAddress pair on B.DLL, which in turn has a static dependency on A.DLL (unbeknown to it) the loader isn’t going reinitialize A.DLL again because it’s already in the tables and is currently being initialized. B.DLL will be initialized (because that’s what GetProcAddress implicitly guarantees?) and any subsequent call on the returned pointer into B.DLL could "end in tears" as we’re still sitting in the initialization section of one of the modules on which B.DLL depends. Is this correct?

  5. mgrier says:

    Graham, the loader is smart enough to not try to run A’s initializer again but as you point out that doesn’t mean that B’s initializer will actually work. Now you might think B was smart enough about this, but also all of B’s static imports have to be smart about how they call into A.

    Now you need dynamic global analysis in order to predict whether you have any bugs. Sounds like a recipe for failure to me…

  6. Richard says:

    I note that the DllMain documentation on MSDN (July 05 edition) explicitly says that DllMain must not call LoadLibrary or LoadLibraryEx.