I have observed applications that hang on a call to CloseHandle() and thought to myself, "what is the application doing wrong to cause this hang?".
To answer that question, you really need to look at the Windows kernel. The reason is that the Windows Object Manager that takes care of closing handles lives in the kernel. Unfortunately, most user mode programmers don't have experience poking around in kernel mode. There are generally few instances where such a developer would need to know the intimate details of what is happening at the operating system level.
While each case is different, I've found many problems due to third party drivers. In particular, anti-virus drivers like to keep track of what's going on with the system and are not always well behaved. Thus, even though developers may be writing what should be responsive code, drivers that exist beyond their control may end up contributing to a hang.
I don't mean to imply that you should not be calling CloseHandle() on your UI Thread - fixing that would seem like a lot of overhead to solve a very small class of problems. However, if you're closing handles that are likely to be tracked by third party drivers (file handles come to mind), you may want to evaluate why it is being closed on the UI Thread. Device I/O should be performed on a background thread, which would also be a good place to close the handle being used.
Note: LoadLibrary(Ex) performs file I/O and will open and close a file handle in its implementation. This could lead to a hang while closing a handle, even though you didn't explicitly call CloseHandle(). Keep that device I/O activity on a background thread!