I was talking recently with Surupa Biswas (a program manager on the NGen team at Microsoft and the author of the MSDN Magazine article describing NGen at http://msdn.microsoft.com/msdnmag/issues/06/05/CLRInsideOut/). We were discussing a bug that was uncovered in one of the daily builds of the next version of Visual Studio (codenamed Orcas). That bug illustrated a potential issue that any setup could encounter if it includes native image generation (NGen) actions. We decided it would be best to describe this scenario and offer some guidance to setup developers to help them avoid this type of problem in their setups.
The underlying issue is that the NGen service is currently not fully re-entrant, and this can lead to deadlock issues. An example of a deadlock scenario that is possible today is the following:
When NGen attempts to generate a native image for A.dll, and A.dll has a static dependency on some optional component B.dll, NGen will try to load this dependency. That in turn can trigger the installation and native image generation for the optional component. This means that NGen can be triggered for B.dll while the system is still attempting to NGen A.dll, and NGen will end up in a deadlocked state.
How to avoid this problem
In order to avoid this possible deadlock, setup developers should avoid using synchronous NGen during the installation process for optional components. Instead, if NGen is desired for optional components, it can be scheduled to occur asynchronously after the installation process is complete.
If you are using WiX to build an MSI-based installer that includes NGen (by using the instructions I previously posted at http://blogs.msdn.com/astebner/archive/2007/03/03/how-to-ngen-files-in-an-msi-based-setup-package-using-wix.aspx), this means that you need to use an NGen priority value of 3 (for asynchronous) instead of 0 (for synchronous).