Windows Installer Custom Actions, Windows Vista and Terminal Server

  • Article

Using a setup project in Visual Studio is a nice and easy way of creating a deployment solution for your application. In some cases, built-in features such as file copy, GAC registration and registry modification are good enough but there are cases where you need to perform a custom action such as configuring CAS or running a script.

If you have used Visual Studio 2005 to build the setup package (with custom actions) and have tried to run your setup package on Windows Vista with UAC enabled, then you may already know that the setup can fail. This is because Windows Vista is enforcing an architectural intent according to Robert Flaming's blog post. So in order to make sure your custom action runs on Windows Vista without a problem, you need to edit the MSI package and change the custom action type flag to include msidbCustomActionTypeNoImpersonate (see here for the list of options). So for example, if your custom action is defined in a class library, this is how you can calculate the type value for Windows Vista:

msidbCustomActionTypeDll + msidbCustomActionTypeInScript + msidbCustomActionTypeNoImpersonate = 0x00000001 + 0x00000400 + 0x00000800 = 3073

(Note that Visual Studio 2008 sets this value correctly to 3073).

You can then update the type value for this custom action in the Orca tool:

And instead of performing this step manually, you can use a script that does this for you and then run this script as a post-build event of your setup project. Aaron Stebner has a great blog post on this.

So now all is good and we have an automated process for setting the right value for the custom action type. Now what happens if you want to run the same setup package on Terminal Server? If you are performing a per-machine install, the custom action runs with no user impersonation by default and you may need to perform some tasks that rely on the user being the administrator. In order to force the custom action to run with user impersonation, you need to include another flag (msidbCustomActionTypeTSAware), which is ignored if msidbCustomActionTypeNoImpersonate is already present. This makes sense as one is forcing impersonation and the other one is preventing it. But now the challenge is, how can we create a setup package that runs on both platforms?

Windows Vista needs: msidbCustomActionTypeDll + msidbCustomActionTypeInScript + msidbCustomActionTypeNoImpersonate = 0x00000001 + 0x00000400 + 0x00000800 = 3073
Terminal Server needs: msidbCustomActionTypeDll + msidbCustomActionTypeInScript + msidbCustomActionTypeTSAware = 0x00000001 + 0x00000400 + 0x00004000 = 17409

This is where the deployment conditions come to the rescue. First, a bit of background on deployment conditions:

All setup project deployment artifacts such as files, folders, registry entries and custom actions have a Condition property, which will determine whether that item will be installed/run. These properties tell you about the hardware configuration (memory size, CPU type), OS (version, service pack and build number) and other information elements such as user name and computer name. The property reference section on MSDN has a full list of property values you can use. You are not limited to one condition and you can perform logical operations and equality checks too (see Conditional Statement Syntax and Examples of Conditional Statement Syntax).

So let's go back to our scenario. We were looking for a solution that allows a single setup project to be used on Windows Vista and Terminal Server. There is a deployment condition called TerminalServer, which is defined only if we are running on Terminal Server. The deployment condition "TerminalServer" will be true in a Terminal Server environment and the "NOT TerminalServer" will be true in all other environments, including Windows Vista. So we can achieve our objective by creating the following two custom actions:

  • One for Terminal Server environments, Condition: TerminalServer, Type: 17409
  • One for other environments, Condition: NOT TerminalServer, Type: 3073

This doesn't necessarily mean you need to create two installer classes though. You can add the default output of your class library (which contains the installer class) to the setup project twice and then add two custom actions, each of which uses one of those outputs. You can then use the Orca tool to set the types manually (make sure you set the right type for each action) or use the scripting approach as explained by Aaron Stebner (note that you will need to change the script to satisfy the requirements).

Other links:
Operating System property values for deployment conditions

Originally posted Mehran Nikoo on December 23, 2008 here.