I have seen developers trying to call cryptographic functions/ API’s within DllMain and thereby creating a hang in their application that calls this DLL.
The reason behind this is not the cryptographic API but this is a classical deadlock example. This deadlock will appear even if you call any other API’s within DllMain.
When you run the application it hangs at the call to CertOpenStore API.
If you debug the application from a debugger such as WinDbg it will show you:
If DllMain enters a critical section or waits on a synchronization object, and that critical section or synchronization object is owned by some code that is in turn waiting for the loader lock, you just created a deadlock.
The hang is due to the loader lock. This is because you are calling CertOpenStore() within DllMain. The API tries to load its dependent DLL’s (which is “crypt32.dll”) and doing this it hangs because of the deadlock.
The ideal DllMain should be an empty stub. We should not call CertOpenStore() or any other API within DllMain.
Points to remember while writing a DLL
The three main components of the DLL development model are:
· The library loader. DLLs often have complex interdependencies that implicitly define the order in which they must be loaded. The library loader efficiently analyzes these dependencies, calculates the correct load order, and loads the DLLs in that order.
· The DllMain entry-point function. This function is called by the loader when it loads or unloads a DLL. The loader serializes calls to DllMain so that only a single DllMain function is run at a time. For more information, see http://msdn.microsoft.com/library/en-us/dllproc/base/dllmain.asp.
· The loader lock. This is a process-wide synchronization primitive that the loader uses to ensure serialized loading of DLLs. Any function that must read or modify the per-process library-loader data structures must acquire this lock before performing such an operation. The loader lock is recursive, which means that it can be acquired again by the same thread.
Improper synchronization within DllMain can cause an application to deadlock or access data or code in an uninitialized DLL. Calling certain functions from within DllMain causes such problems.
DllMain is called while the loader-lock is held. Therefore, significant restrictions are imposed on the functions that can be called within DllMain. As such, DllMain is designed to perform minimal initialization tasks, by using a small subset of the Microsoft® Windows® API. You cannot call any function in DllMain that directly or indirectly tries to acquire the loader lock. Otherwise, you will introduce the possibility that your application deadlocks or crashes. An error in a DllMain implementation can jeopardize the entire process and all of its threads.
The ideal DllMain would be just an empty stub. However, given the complexity of many applications, this is generally too restrictive. A good rule of thumb for DllMain is to postpone as much initialization as possible. Lazy initialization increases robustness of the application because this initialization is not performed while the loader lock is held. Also, lazy initialization enables you to safely use much more of the Windows API.
You should never perform the following tasks from within DllMain:
· Call LoadLibrary or LoadLibraryEx (either directly or indirectly). This can cause a deadlock or a crash.
· Synchronize with other threads. This can cause a deadlock.
· Acquire a synchronization object that is owned by code that is waiting to acquire the loader lock. This can cause a deadlock.
· Initialize COM threads by using CoInitializeEx. Under certain conditions, this function can call LoadLibraryEx.
· Call the registry functions. These functions are implemented in Advapi32.dll. If Advapi32.dll is not initialized before your DLL, the DLL can access uninitialized memory and cause the process to crash.
· Call CreateProces. Creating a process can load another DLL.
· Call ExitThread. Exiting a thread during DLL detach can cause the loader lock to be acquired again, causing a deadlock or a crash.
· Call CreateThread. Creating a thread can work if you do not synchronize with other threads, but it is risky.
· Create a named pipe or other named object (Windows 2000 only). In Windows 2000, named objects are provided by the Terminal Services DLL. If this DLL is not initialized, calls to the DLL can cause the process to crash.
· Use the memory management function from the dynamic C Run-Time (CRT). If the CRT DLL is not initialized, calls to these functions can cause the process to crash.
· Call functions in User32.dll or Gdi32.dll. Some functions load another DLL, which may not be initialized.
· Use managed code.
Consider a DLL whose main thread contains DllMain. The library loader acquires the loader lock L and then calls into DllMain. The main thread creates synchronization objects A, B, and G to serialize access to its data structures and then tries to acquire lock G. A worker thread that has already successfully acquired lock G then calls a function such as GetModuleHandle that attempts to acquire the loader lock L. Thus, the worker thread is blocked on L and the main thread is blocked on G, resulting in a deadlock.
The above references and example are directly taken from the first reference below.
· “Best Practices for Creating DLLs at download.microsoft.com/download/a/f/7/af7777e5-7dcd-4800-8a0a-b18336565f5b/DLL_bestprac.doc.
– Shamik Misra