How to prevent the Program Compatibility Assistant from appearing on Windows Vista

While researching the root cause of the issue that can cause a Windows XP SP2 block dialog to appear when trying to install the .NET Framework 3.5 beta 1 on Windows Vista (described in more detail in this blog post), I learned some useful information about the Program Compatibility Assistant (PCA) that I wanted to summarize here.

What kinds of applications are affected by the Program Compatibility Assistant?

The Program Compatibility Assistant monitors setup applications that it detects are not Windows Vista-aware.  The PCA uses a list of well-known names such as setup.exe, install.exe, etc to try to decide whether or not an application is a setup.  The PCA treats all setup applications that do not have an embedded manifest that specifies a requested execution level (asInvoker, highestAvailable or requireAdministrator).

In the case of the .NET Framework 3.5 beta 1, the setup package did not include an embedded manifest specifying a requested execution level, and it was named dotnetfx35setup.exe, which triggers the Windows Vista legacy setup application detection logic.

When does the Program Compatibility Assistant appear?

Once the Program Compatibility Assistant identifies a non-Windows Vista-aware setup application, it will monitor the Programs and Features control panel (previously known as Add/Remove Programs).  If an entry is not created or removed from the Programs and Features control panel, then the PCA treats the process as a failed setup and displays a dialog after the setup process exits that asks if the user wants to attempt to re-install using recommended settings.  In this case, using recommended settings will cause Windows Vista to re-run the setup using Windows XP SP2 compatibility mode.

In the case of the .NET Framework 3.5 beta 1, because the setup application is treated as a non-Windows Vista-aware setup, whenever it returns without creating or removing a Programs and Features control panel entry, the PCA dialog will appear (even if you simply launch setup and cancel on the End User License Agreement page).

How can I opt out of the Program Compatibility Assistant for my setup?

In order to prevent the Program Compatibility Assistant from appearing, you must include an embedded manifest that specifies a requested execution level for your setup executable.  If you wrap the setup executable in a self-extracting package, you must also include an embedded manifest in the self-extracting package too.  Once you do this, Windows Vista will treat your setup as Windows Vista-aware, and it will no longer show the PCA dialog when setup exits after a failure or cancellation.

Where can I find more detailed documentation?

For more detailed information about the Program Compatibility Assistant, I encourage you to check out the following resources: