The NT DLL Loader: DLL_PROCESS_ATTACH reentrancy - step 3 - quality requirements

Now we're loaded for bear!  We understand how PEs which are either launched via CreateProcess() or loaded via LoadLibrary() are the roots of directed cyclic graphs.  Each new graph is turned into a linear initialization-order list where nodes further from the root are initialized prior to nodes closer to the root.  Cycles in the graph are resolved based on where you first enter the cycle and thus depend on the entire graph (not just local DLL-to-DLL relationships).  Dynamic loads during initialization are often handled correctly but they themselves can introduce additional cycles.  There is less opportunity to fix this up since the dynamic load results only in new additions to the initialization list.

Great.  Seven impossible things already done (I guess they weren't really impossible, but then again maybe they weren't really done either, eh?) let's see where things start to get really messy.

Dynamic loads during initialization lead to a couple of very interesting things.  First, recall that the initializer that was in progress isn't re-run when GetProcAddress() is called.  That's going to be important.

Let's go back to my sleazy little attempt to call a function here:  (let's assume this is in bar.dll; it's not going to be very important but given all the players it's going to get confusing...)

    case DLL_PROCESS_ATTACH:
SOME_FUNCTION_PTR_T pfn = (SOME_FUNCTION_PTR_T) GetProcAddress(LoadLibraryW(L"SomeOther.DLL"), "SomeFunction");
if (pfn != NULL) (*pfn)();
break;
}

We didn't check the return status from the LoadLibrary() call.  Since LL() doesn't run initializers let's assume that that's OK in this context.  I'll assume that GetProcAddress() fails with an invalid argument and thus still returns NULL.  We do check the result of GPA() and don't call through the pointer if it's NULL so hey, nobody got hurt, right?

The LL() non-check is dubious.  Do you know why LoadLibrary() failed?  If it failed because of out-of-memory maybe it's the wrong thing to press on.  I digress; this is more of a usual topic for my blog rather than focussing on loader related issues.  But it's about to become a loader related issue.

Let's say that foo.dll was already on the init list, after the current DLL.  Let's assume that its refcount was 1.  Now SomeOther.DLL maybe statically imported foo.dll also.  Now it's (foo.dll's) refcount is 2 after the LoadLibrary() call.  Let's write the overall initialization list now:

ntdll.dll
kernel32.dll
bar.dll
foo.dll
somedllimportedbytheexethatusesfoo.dll
someother.dll

The GetProcAddress() call attempts to run foo.dll's initializer (since it has to be initialized before running someother.dll's initializer).  Let's assume it fails.  These things happen, it's the real world out there.  I/Os fail, memory allocations fail, duplicate file names occur, network glitches when trying to open file handles using the redirector, etc.

So, foo fails initialization and properly reports FALSE back to the loader.  The loader will propagate this failure out to the GetProcAddress() call.  But wait, that darned code assumes that the only reason GetProcAddress() can fail is due to ERROR_PROC_NOT_FOUND!

Care to guess what happens next?