In case you haven't noticed, there are a couple of gaps within setup projects regarding how the generated MSIs work on Vista. Allow me to give you a roadmap of where I hope to take this blog to plug some of these holes.
If you thought Windows Installer was complicated, Windows Installer on Vista is even more complicated. Naturally, most of this comes down to how UAC and Windows Installer work together. A lot of people much smarter than me have written quite a bit about this. (Or is it "smarter than I"? I bet those smarter people woud know!) Robert Flaming has an entire blog dedicated to this topic and Aaron Stebner has a great synopsis.
Here is a list of the post build operations I'd like to create to help make setup projects a little bit better (or, you could just wait until Orcas is released: some of these will probably be in there).
Making Custom Actions run without impersonation
All custom actions that a user adds to a setup projects end up as deferred custom actions. It is possible to specify deferred custom actions to run as "No impersonate", meaning the custom action will not be run in the current user's profile (corrected 6/4/2007; see comments). Unforunately, what is not mentioned in the docs but seems to be the general consensus around these parts, is that deferred custom actions should generally be no impersonate. There were no obvious errors around the setup projects neglecting this until Vista came along and started having all sorts of managed custom actions failing. Oops.
Aaron already posted a work-around for this using a jscript and postbuild steps.
Create non-advertised shortcuts
Setup projects always created advertised shortcuts, which causes quite a bit of consternation in the masses: a lot of people want to be able to condition their shortcut, but only non-advertised shortcuts can be conditioned. (More accurately (I think): components can be conditioned. Non-advertised shortcuts are tied to a component while advertised shortcuts are not).
What does conditioning a shortcut have to do with Vista? Nothing. However, you don't get the "Run as..." option for advertised shortcuts like you do for non-advertised shortcuts (or so I've been told). Sometimes on Vista you want the "Run as" capability. So you'll need to create the shortcuts non-advertised.
A Windows Installer package has a way to signal to Vista that the install is purely per-user and there is no need to prompt for elevation (as nearly as I can tell, Windows Installer believes this assertion and doesn't perform any checks to make sure the package isn't lying). Robert Flaming refers to this as the LUAAware bit, but is really an obscure little bit in the Word Count summary property of the MSI.
In case your application install is not non-impactful and doesn't set the LUAAware bit, its considered polite to warn the user that the action they are about to take is going to cause an elevation prompt. In Vista this is done by adding the Shield icon to a button. Yes, Windows Installer buttons can have shields on them, too.
Vista has a set of heuristics to determine whether or not to issue a UAC prompt. Part of that heuristic is "if something is called setup.exe, you probably need administrative priveleges to run". This behavior can be overridden by adding a manifest to your executable.
Unfortunately, the Visual Studio bootstrapper is called setup.exe and doesn't have this manifest embedded within it. So, as soon as the setup.exe is launched on Vista you are prompted to elevate the process. The way the bootstrapper is designed is that everything it does will then have the same token as setup.exe. This is pretty handy for when the bootstrapper installs the .NET Framework and other impactful software packages. But the trailing application (we always refer to it as the "caboose") may not really require elevation: the MSI may be capable of a per-user install (then again, it may not). Of course, if the elevation is done via an "Over the Shoulder" action (that is, using an Administrator account that is not the currently logged on user), the application will not be installed within the original user's profile.
The biggest problem here is that the bootstrapper may not really need elevated priveleges: it's possible all of the prerequisites which could be installed by the bootstrapper are already on the machine. In that case, the bootstrapper just skips ahead and installs the caboose. Of course by then the damage has already been done: setup.exe has been elevated and the install will be, too. This is especially harmful for ClickOnce applications: even if the same user profile was used for the elevation, an install of the ClickOnce application actually runs the application. Suddenly your ClickOnce application is running elevated (and possibly in the profile of someone else)! A similar scenario can exist for MSIs that launch the application after install (which I showed how to do in an earlier blog entry). Essentially, all the work you did to set the LUAAware bit has been wasted if your application gets launched via the bootstrapper.
So, hopefully that's enough to keep your interests piqued over the next month or so. Don't worry, there will likely be new code written for just about all of these things, and I will resume our regularly scheduled boring programming in the next post.