Tao of the Windows Installer, Part 5

Testing is a vital part of the package development process and you can avoid many problems in production if your testing strategy is sound. So this time we will look at a couple of pointers on testing.

I must admit I’m less happy with the arrangement of this section than with some of the others. I think Rule 53 is making a bid for its own section and breaks away from the notion of keeping this to some fairly compact bullet points; in any case, it hopefully gives you a starting point on what to think about in your testing. Let me know what you think.

Series links:

Testing and Support

Rule 53: Test thoroughly
It is crucial that you test your packages thoroughly before deployment. A classic mistake is for developers and package authors to test only on their own systems where they have full administrator rights and then discover that normal users cannot use their applications. You may want to divide your testing into these phases:

Installation Testing
Test your package under different installation scenarios, including:

  • Full user input and silent installation
  • Different combinations of selected features
  • Different user roles (e.g. normal user, administrator, etc.)
  • Different deployment methods (e.g. manual, SMS, GPO, etc.)
  • Administrative and advertised installations
Enable installer logging for each test and examine the system event logs for errors after each run. Make sure that all installation initialisation and run-time errors are resolved.

An important, but often overlooked, part of installation testing is to ensure that your setup UI is accessible and can handle different screen resolutions and font sizes.

Patching and Repair
As mentioned in the Patching section, it is very important that you think about patching before deployment. In fact you should not only think about it but test it thoroughly. You should include these scenarios:

  • Small update
  • Minor upgrade
  • Major upgrade
  • Product Repair
  • Multiple simultaneous patch installs and uninstalls
  • Installing and uninstalling patches “out of order”
Uninstall Testing
Many operating system related errors caused by a repackaged application happen during an uninstall session. Therefore it is extremely important to verify that a generated package does not uninstall “too much” and only uninstalls data that belongs to the package itself and not to the operating system or other applications.

Test uninstall as thoroughly as install, including:

  • Uninstall under different user contexts (e.g. normal user and administrator)
  • Uninstall via different methods (e.g. Control Panel, command line, etc.)
  • Make sure that Operating System does not report any integrity problems
  • Reboot a test machine after uninstall (even if the uninstall did not require a reboot)
  • Verify integrity of common system tools and other standard applications
As with install, use Installer logs and event logs to look for errors, and make sure you resolve all those you find.

Product Functionality Test
A successful installation and uninstall sessions can be insufficient for an application to work correctly. Therefore you need to ensure that the application works as expected. Some things to check:

  • Verify each shortcut in the package starts the corresponding application correctly
  • Walk through main menu items in the application
  • Follow the application functionality test provided by the vendor, if any
  • Verify that file associations by launching a file for each registered extension.
  • Make sure that application opens, save and prints the files correctly
It is recommended that corporations provide application functionality tests allowing a repackaging engineer to quickly verify major application functionality right away, and make sure that a developed MSI package correctly installs an application, and an application is ready to work properly.

User Acceptance Test (UAT)
Just as important, but often overlooked, is user input into the testing strategy. User Acceptance Testing (UAT) is especially important when re-packaging legacy applications to MSI format.

Some points to note for UAT:

  • Application should be deployed for UAT in exactly same manner as it will be deployed in the production environment
  • Pilot the deployment with a selection of “typical” users for the application
  • Make it easy for users to report problems and provide feedback
  • For internal applications, the owning Business Unit must ensure each application passes the UAT to be defined by that Business Unit. Where the package is processed externally an amount of time must be agreed upon, after which it is reasonable to assume the package has been accepted into production
  • It is recommended that product functionality tests be provided for all internal applications. Such tests will allow a packaging engineer to verify major application functionality before submitting a migrated package for UAT

Rule 54: Validate Your Packages
Validation is a means by which you can run certain standard checks (called ICEs) against your package to check for inconsistencies and non-compliance with Installer rules.

Many ICEs ship with the Installer SDK in .CUB files and you can write your own. They can be run against your packages using the ORCA or MsiVal2.exe tools from the SDK.

You should always fully validate your packages and, much like when working with application development, correct any reported errors and treat all warnings seriously.

Validation is also a useful troubleshooting technique, particularly if you are working with 3-rd party packages.

Rule 55: Use Installer Logging
As mentioned previously, the Installer supports extensive logging. This is by far the best tool for troubleshooting Installer issues and the time spent understanding the log files will be repaid many times over when problems arise later. At the least logging should be used throughout testing.

You are certain to be asked to provide verbose logs if you contact Microsoft support, so either have these ready or be prepared to reproduce your problem to gather some.

Rule 56: Use Virtual Environments for Testing and Support
Using virtual machines (VMs) for testing rather than real systems has a couple of advantages:

  • You can build a VM for each of the different systems in your environment and store these on a single server or workstation to be brought on-line when needed
  • With “undo”-style features of VMs you can easily revert to a clean, known state after each test
  • The same VM images can be used by testing and helpdesk staff
However, do note that if you think you have discovered a bug in the Installer you may be required to reproduce the problem on real hardware when you contact Microsoft support:

Support policy for Microsoft software running in non-Microsoft hardware virtualization software

[Author: Richard Macdonald]

This posting is provided “AS IS” with no warranties, and confers no rights.

Comments (12)

  1. Phil Wilson says:

    I think the testing rule is probably outside the topic (which I think is how to design an MSI setup that works, is robust, can be serviced, and won’t come back later in some way to bite you).

    Given that there are 50+ rules, I’m wondering if they need categorising by priority rather than by area. Some are more important to follw than others, I think that’s true to say.

  2. zhakim says:

    “I’m wondering if they need categorising by priority rather than by area”
    Placing them in order of importance is likely to lead to a lot of argument – look at the comments from Part 1;  with only a few rules listed there was already disagreement over what was most important.

    As I mentioned before, Rule 1 is the only one that is in order of importance – if you follow that then you’ll avoid a lot of problems anyway.  Apart from that the rules were intended be read in any order.

    However, I agree that there is a need for a better arrangement.

    [Author: Richard Macdonald]
    This posting is provided “AS IS” with no warranties, and confers no rights.
  3. Megh Neupane says:

    I think you can categorize Rule 53 as two parts:

    => Pre-UAT

    This can include the 4 points you have mentioned.

    => UAT

    You can have more categories here instead of scrambling points to 1 topic. For e.g. Test Environment, Prod Environment…This references a direct success from the Pre-UAT functionality testing and above all, user requirements (customization) and application stability.

    These two sections are best suitable for Re-packaging applications from legacy installs. I have been following this procedure for some years now and it eventually leads you to "distinct" package errors and "very close to" repackaging success.

  4. Mike Dimmick says:

    I think you’re mixing together rules for developers, rules for administrators wanting to make use of Windows Installer technology to deploy applications, and rules for administrators wanting to repackage applications built with older technologies. The first and second groups don’t really overlap, while the third overlaps somewhat with both. I think it might be better to have a set of rules for developers, a set for administrators, and point the third group at both sets, with a few extra rules for them.

  5. EdShepley says:

    I like the idea of the rules for everyone being in the same place. At least then there should be less ambiguity in the business about where to get your help and advice from.

    On the subject of "categorising by priority rather than by area", I agree that a by area approach seems sensible and by priority could lead to a ruckus, but how about marking each rule as ‘strict’ or ‘preferred’?

  6. Moses says:

    Is there any special ICE warnings that MAY be ignored? I am working on an msi package and I have a number of ICE 33 warnings that seem to result from some merge modules that I integrate in my msi. I have no errors, but I have a series of warnings such as:


    Reg key Global_Controls_MSADODataOcx_r17.7EBEDD1B_AA66_11D2_B980_006097C4DE24 is used in an unsupported way. This key need not be written as it already exists in the system. ice33.html Registry Registry Global_Controls_MSADODataOcx_r17.7EBEDD1B_AA66_11D2_B980_006097C4DE24

    Evaluation: ICE33


    I am still fairly new to windows installer and I am definitely taking rule# 1 seriously. However, in case someone has a clue how to sort this out, let me know.



  7. Well, here we are with the last part in this series, where we’ll be looking at security in relation to…

  8. DT blogi says:

    Windows Installeriga on küllaltki keerukas algajal otsi lahti teha. Mul endalgi on üks installer käsil ja tänu keerukamale funktsionaalsusele, mida see peab pakkuma, tuli ka rohkem süveneda Windows Installeri hingeellu. Leidsin internetis ringi tu…