Integration Hurdles for EXE Custom Actions

A while back, two sets of engineers were arguing whether simply calling an EXE custom action would be good enough for Windows Installer based package.  The first team with the EXE didn't want to do the work to move to Windows Installer but they really wanted the second team to take a dependency.  The team based on Windows Installer said the integration problems with calling an EXE was so great they would not integrate without a Windows Installer package. 

As an attempt at arbitrating the disagreement, it was requested the Window Installer team provide an analysis.  A quick paper was written to capture the analysis.  This paper has since been shared in other similar circumstances. 

In a recent instance, Microsoft blogger Aaron Stebner suggested this quick paper would make a good blog.  Here it is.  Hope it's useful ;^)

Integration Hurdles for EXE Custom Actions

Overview and Scope

Windows Installer (MSI) architecture was designed to work best when all the installation behaviors are native to the Windows Installer. When the native behavior (called Standard Actions) are insufficient, there is a way to provide extend behavior (called Custom Actions). Custom Actions come in various base types that are differentiated on the way the Windows Installer service instantiates the custom action in the appropriate sandbox. Among the types of custom actions are executable files and dynamic link libraries.

Bare custom actions are risky to the integrity of a Windows Installer base install and this document will consider the risks.

This document will not dive deep into the architecture or the possibilities to improve the custom action architecture in the future.

Integration Hurdles

The installation integrity risk running EXE custom actions come in a number of varieties








(by consumer of EXE)


Bad EXEs can bring down the custom action sandbox


Binary Dependency

Bad EXEs can require technologies that are not in the system at the time of the install.  Specific instance: MSXML custom actions required MFC 7 but it didn't exist on machines.



Bad EXEs can expect to load DLLs from the path but the path could be customized by the user on the machine.



Bad EXEs can require more rights (e.g. admin) than the package thus breaking scenarios (per-user)



EXEs do not have access to the Windows Installer Log so there is no integrated troubleshooting.

None (best effort, build DLL that provides log path to EXE and puts the log next to or into the Windows Installer log)


EXE do not have access to the UILevel so they do not know whether it's OK to display UI.

None (best effort, build DLL that reads UILevel and alters command line)


EXEs do not have access to the widows installer progress bar.

None (best effort, add entries to the ReserveCost table)


EXE may hang and neither the Windows Installer nor the user has no way of knowing whether the install is hung or just taking a long time.

None (best effort, build DLL that calls CreateProcess and WaitForMultipleObjects)


EXE do not have any way of being integrated into the Windows Installer costing

None (best effort, build DLL that add ticks to the progress bar)


- rollback behavior
- decision when to execute rollback (did failure occur in the FWD case thus calling the Backward case is extraneous?)

Bad EXE do not support rollback.

None (best effort, build DLL that calls EXE uninstall if exists)

Transparency and Predictability

EXEs are not data driven from the contents of the MSI thus are not transparent to users, especially admins.



Bad EXEs do not support uninstall

None (best effort, build DLL that calls EXE uninstall if exists)

Uninstall Rollback

Bad EXEs do not support uninstall rollback

None (best effort, build DLL that calls EXE install)


Bad EXEs do not support repair

None (best effort, build DLL that calls EXE install again)

First Run

Bad EXEs mix per machine installation with per-user installation that should be invoked at first run


Error Codes

Bad EXEs do not provide return codes or have return codes that do not match the custom action return code expectations.

None (best effort, best effort, build DLL that interprets errors returned from the EXE and returns the appropriate value)


EXE failures are difficult to map to Watson buckets (for teams that have implemented Setup Watson)

None (best effort, build DLL that initializes Setup Watson with the needed context to differentiate EXE error)

PSS Costs

When an EXE fails, the supports costs are absorbed by the enclosing product

None (best effort, build DLL that records needed context to differentiate EXE error)


Does installation or manipulation by the EXE cause strings to be written to the system?  If yes, how are the strings differentiated?



Some EXEs have different levels that can be installed (Minimum, Full).

None (best effort, build DLL that maps UI or feature selection to different levels in EXE)

Terminal Server and SMS

Bad EXEs do not run correctly on Terminal Server where there is no user logged in and will not have a user hive


Group Policy (Publishing, assignment)

<missing this context>



There is no way to debug custom actions in EXE from MSI.

None (best effort, build DLL that uses dll debugging then alter EXE call from inside DLL)

Access to Database

EXEs are unable to access the database

None (best effort, build DLL that uses dll method to access the database)

Run From Source or Cache

EXEs are unable to be configured to run from source



EXEs are more difficult to confirm they contain the right security fixes when servicing



EXEs that contain 2.0 User Education integration may not have MSI's namespace parents configured at the time they are called

None (best effort, move the EXE to a different location in the sequence after the html help custom actions)


EXEs that produce events during install will appear outside the MSI context



Numerous third party tools exist to customize an install to the needs of a particular specialized user (usually LORGs)


File In Use

There is no way for a custom action to inform the user that files they want to manipulate are in use


Pending File Renames, File in Use, and Rollback

If a EXE custom action replaces files that are held in use and a MoveFileEx causes those files to end up in Pending File Renames, then the install rolls back it's possible that the files will still be changed on the next boot.


Detection of When to Install (Resiliency v Rollback)

A rollback during repair or reinstall could cause premature removal of a EXE install that existed on the machine before MSI was installed.  If one implements rollback, one has to implement foolproof detection for existing installs of the component and not run the EXE package if the same version of EXE is already on the machine.  Otherwise, a cancel or failure in the MSI could cause a preexisting installation of the EXE component to be removed.



If the EXE requires a reboot, there is no way to communicate the need for the REBOOT to the MSI



As EXEs can not call MsiProcessMessage, they are unable to respond to the cancel button.

None (best effort, build DLL that calls MsiProcessMessage)


EXEs that crash may cause a Watson dialog during the install


Test Scenarios

When testing the MSI that contains this EXE, one needs to test the following

  • Deployment technologies

    • Group Policy Software Distribution

      • Publishing

      • Assignment

    • SMS deployment

    • Install from network share/media

    • Install from local cache

    • Hard drive imaging

  • Customer segments

    • Retail install

    • OEM pre-install

    • Enterprise deployment

  • User types

    • No user (install during winlogon, install by SMS agent) - these will not have a user hive

    • Lockdown user

    • Regular user

    • LUA without registry virtualization

    • Admin user

[Author: Robert Flaming]
This posting is provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at

Comments (11)

  1. The above is very useful and I have linked to it from my documentation but I find it amusing that this page (titled "Specifying the Order of Self Registration") exists in the SDK to tell use to use an EXE CA:

    And msiexec.exe is particularly bad in any case as it doesn’t return a non-zero return code on error (so success can’t be determined)…

  2. Robert Flaming has posted a helpful analysis of the issues that need to be considered when using EXEs

  3. AJ says:


    Well…since we are on the topic of custom actions, how about adding support for managed code custom actions?  I know all the arguments for and against, so spare me the monotony, but this has to be one of the top requested features.  I’d be pleased as pie with real ACL support as well.  The LockPermissions table is laughable at best.

    Remember when script installers became archaic and legacy?  What do you think is happening to Windows Installer?  Go ahead and drop ClickOnce and focus on Windows Installer.  ClickOnce was ill-conceived from the start and from what apparently is the current roadmap will never be a viable replacement for Windows Installer.


  4. managedcoder says:

    This analysis was almost funny, probably unintentionally.

    I agree with AJ 🙂 on the managed code and YES why do we still after all these years need to use thirdparty ACL solutions.

    Funny thing is – for us that makes the applications that sell MS Windows – the WIX people are doing the best work at the moment. -They- are the ones that make it easier to just ‘do things right’.

  5. Windows Installer custom actions that launch executables (base custom action type msidbCustomActionTypeExe

  6. gripper says:

    I don’t consider Custom Action EXEs Bad or Good, I think one simply needs to weight the benefits with the hurdles, and this blog entry is very helpful in that regards. For teams that are simply trying to "push off work", well… that is not a good enough reason. However, there are some cases where benefits outweigh the hurdles, especially in regards to maintenance and packaging options.

    I’ll give you an example where I recently use an MSI EXE Custom Action.

    Originally, we used a Frontend User Interface Chainer to chain an MSI installer (for an application) with a non-MSI Driver Install. But we wanted to get rid of the chainer as some OEMs didn’t want it, they wanted an MSI look and feel. We could rewrite the chainer, or just find a way to add drivers directly to the MSI.

    Then I experimented with the DiFxApp MergeModule. This integrated the drivers behavior directly into the MSI. However this proved to have maintenance drawbacks. Basically, our application is not tied tightly to the driver INF packages, and various customers will get different driver packages. When I was using DiFxApp, we would need to rebuild the MSI projects with the different driver combinations. These needed to be re-qualified by QA, even if the binaries inside the package (other then the drivers, of course) didn’t change.

    So I investigated this Custom Action EXE mechanism, and it seems to work pretty well. Originally I was going to simply use DPInst, but then I created my own Driver Installer that does what DPInst does, plus is “MSI Aware”.

    This allows one to mix and match drivers at will with the current MSI, so QA can have a warm fuzzy that the MSI didn’t change, and only the driver files did.

    We can churn out various variations of the app/drivers for various OEM customers in minutes. Oh, and as another value added feature of this implementation, if an OEM doesn’t want the app, they just get the very same directory with the Drivers files and Custom Action EXE, and use the Custom Action EXE in “standalone” mode to install drivers (akin to DPInst.exe).

    And packaging the launching logic into a Reconfigurable MergeModule (where a developer can quickly add the relative path to where the Custom Action EXE is on their media) makes integration with new install projects and builds a snap.

    I do agree this doesn’t make sense for true application components. I definitely wouldn’t want to use an external EXE to install app-settings, such as more files, registry keys, shortcuts, etc. That is what MSI is for! But I consider the whole DiFxApp mechanism as a half-hearted attempt at not providing a true external EXE for DPInst.exe, anyway.

    Oh, and I have found ways of linking to the MSI progress bar and status text. It would be awesome for one of the MSI team members, or one of you MSI gurus to give an example of how one can find a handle to the ActiveDatabase from an external EXE, to allow a tighter integration with an external EXE Custom Action. I want to do this in my next respin, so I can add full costing support, and support the cancel (right now I simply disable the cancel button when my Custom EXE runs, enabling Cancel after it finishes). I have some ideas on how to do this already, but it would be nice to hear it from the source, so to speak.

    I know why I can’t just get the handle directly, because each custom action runs in its own server, and the handle will change or disappear based on the scope of its owner. However, it would be cool if there was a way that allowed an external EXE to get the handle to its own server from some property or something that can be appended to the command line to the EXE when the server launches the EXE (via Type 2 or Type 18, or Type 50), so you don’t have to always use a CA Type 1 DLL wrapper function just to do this.

    Sometimes I get frustrated, with all the custom action types, why-oh-why does it always eventually return to a Type 1 for everything? It seems whenever I venture to another CA base type, it works for 95% of what I need, but there is always some shortcoming which causes me to go back to good-old Type 1.

  7. Adam says:

    I would love to hear how a VS2005 tool vendor should handle having to run devenv.exe /setup without an EXE custom action.

  8. I&#39;d like to add my thoughts to the recent discussion about how bad custom actions are, and which

  9. Erik says:

    Hey Robert! Great article! I had to make one quick comment though about, "There is no way to debug custom actions in EXE from MSI". True you cannot use MSIBreak but I used to use several methods to get at EXEs through debugging when I was in PSS. You can attach to the custom action server, if this is not out in the field you could do message boxes, or break before the custom action using another custom action, you could do "Image File Execution Options" on MSIExec.exe. But I agree it is much harder to debug.

  10. Adam, you are so right!  Even WiX.MSI calls devenv /setup to install the Votive project and item templates.

    BTW, I’m just now starting down the road of deploying VSTO 3.0 addin’s for MS Office 2007.  I want to use MSI not ClickOnce because I’m also deploying a bunch of other stuff.   Anyone want to tell me how to install the .VSTO without using an EXE CA to call VstoInstaller.exe ?

Skip to main content