The first major obstacle to testing an application for compatibility on Windows Vista is to actually get that software installed, and there are still some challenges remaining in achieving this. One of the issues that has popped up is with deferred custom actions.
When you specify the type of a custom action in the MSI database, you are creating a bitmask. This isn’t always easy to see, since the Orca MSI editor provided with the Platform SDK presents this information in decimal rather than hexadecimal format, but it is true. To specify a deferred custom action, you specify msidbCustomActionTypeInScript, or 0x400, in the bitmask. So, if you wanted to do this with a DLL, you would logical or msidbCustomActionTypeInScript and msidbCustomActionTypeDll == 0x400 | 0x1 == 0x401 == 1025.
This is one of the custom action types that we are seeing that are causing problems as a result of UAC. With nothing more than 0x401, this cutom action will be run impersonating the logged in user, not in the system context. It is not using the elevated token, it is using the standard token, which has quite a few privileges and groups removed from it, including the local Administrators group! If your custom action requires elevated permissions to run, then it will fail, as will your installation.
However, you can specify that the custom acton will not use impersonation. By including in the bitmask msidbCustomActionNoImpersonate == 0x800, you will no longer be impersonating the currently logged in user (with their non-elevated token), and instead be running with system privileges just like the rest of the MSI installation. This avoids the issue of an access denied message being triggered by some action in the MSI (at the expense of removing the user impersonation). In the above example, we would use msidbCustomActionTypeInScript | msidbCustomActionTypeDll | msidbCustomActionNoImpersonate == 0x400 | 0x1 | 0x800 == 0xC01 == 3073.
For the setup developer, this means that custom action types that require impersonation of the user should be separated from custom action types that require elevated permissions – the impersonated user no longer has these permissions by default and this is no longer an appropriate assumption. For the administrator, it makes it possible to modify the custom action type (MSI is white box, and you can edit the contents of the MSI database using the Orca SDK tool) to hopefully end up with a successful setup.