Since I originally posted these instructions that describe how to gather callstack information when the .NET Framework setup fails while registering System.EnterpriseServices.dll or running regsvcs.exe /bootstrapi, I have gotten a few problems reported for installations on Windows NT4 SP6a. After looking at the callstack data and looking at past cases logged by our product support team, we found a common pattern that can cause .NET Framework setup failures.
All of these cases shared a common pattern - the NT4 SP6a OS was originally installed on one machine and then imaged with a disk imaging utility, then restored onto another machine that had different hardware (such as a newer processor). There are some instances where imaging and restoring the OS on different hardware will cause .NET Framework setup to fail. The case notes from one support incident indicate the following:
Analysis of the crash dump revealed that the .NET jitter was generating Intel Pentium SSE2 instructions and when such an instruction was executed, it failed with "Illegal instruction". These functions were repeated and eventually a stack overflow crashed the regsvcs process.
Further investigation of the hardware revealed that this CPU supported the SSE2 instruction set, but the OS had not loaded the driver (INTLFXSR.SYS) because this machine was imaged using an image generated from an older CPU (which did not support the SSE2 instruction set).
So far, we have only seen this imaging problem affect NT4 SP6a and not other OS types, but it may be possible for this to happen on other OS types as well. Also, imaging your OS and restoring it on a different machine will not necessarily always cause .NET Framework setup to fail. It is very dependent on the type of hardware on the source and destination machine.
Unfortunately, there is not a good workaround if your machine is hitting this problem other than reinstalling NT4 SP6a directly on the machine (as opposed to restoring a pre-existing image), or installing a newer version of Windows on the machine.