Testing is a vital part of the package development process and you can avoid many problems in production if your testing strategy is sound. So this time we will look at a couple of pointers on testing.
I must admit I'm less happy with the arrangement of this section than with some of the others. I think Rule 53 is making a bid for its own section and breaks away from the notion of keeping this to some fairly compact bullet points; in any case, it hopefully gives you a starting point on what to think about in your testing. Let me know what you think.
It is crucial that you test your packages thoroughly before deployment. A classic mistake is for developers and package authors to test only on their own systems where they have full administrator rights and then discover that normal users cannot use their applications. You may want to divide your testing into these phases: Installation TestingTest your package under different installation scenarios, including:
Full user input and silent installation
Different combinations of selected features
Different user roles (e.g. normal user, administrator, etc.)
Different deployment methods (e.g. manual, SMS, GPO, etc.)
Administrative and advertised installations
Enable installer logging for each test and examine the system event logs for errors after each run. Make sure that all installation initialisation and run-time errors are resolved. An important, but often overlooked, part of installation testing is to ensure that your setup UI is accessible and can handle different screen resolutions and font sizes. Patching and RepairAs mentioned in the Patching section, it is very important that you think about patching before deployment. In fact you should not only think about it but test it thoroughly. You should include these scenarios:
Small update
Minor upgrade
Major upgrade
Product Repair
Multiple simultaneous patch installs and uninstalls
Installing and uninstalling patches "out of order"
Uninstall TestingMany operating system related errors caused by a repackaged application happen during an uninstall session. Therefore it is extremely important to verify that a generated package does not uninstall “too much” and only uninstalls data that belongs to the package itself and not to the operating system or other applications. Test uninstall as thoroughly as install, including:
Uninstall under different user contexts (e.g. normal user and administrator)
Uninstall via different methods (e.g. Control Panel, command line, etc.)
Make sure that Operating System does not report any integrity problems
Reboot a test machine after uninstall (even if the uninstall did not require a reboot)
Verify integrity of common system tools and other standard applications
As with install, use Installer logs and event logs to look for errors, and make sure you resolve all those you find. Product Functionality TestA successful installation and uninstall sessions can be insufficient for an application to work correctly. Therefore you need to ensure that the application works as expected. Some things to check:
Verify each shortcut in the package starts the corresponding application correctly
Walk through main menu items in the application
Follow the application functionality test provided by the vendor, if any
Verify that file associations by launching a file for each registered extension.
Make sure that application opens, save and prints the files correctly
It is recommended that corporations provide application functionality tests allowing a repackaging engineer to quickly verify major application functionality right away, and make sure that a developed MSI package correctly installs an application, and an application is ready to work properly. User Acceptance Test (UAT) Just as important, but often overlooked, is user input into the testing strategy. User Acceptance Testing (UAT) is especially important when re-packaging legacy applications to MSI format. Some points to note for UAT:
Application should be deployed for UAT in exactly same manner as it will be deployed in the production environment
Pilot the deployment with a selection of “typical” users for the application
Make it easy for users to report problems and provide feedback
For internal applications, the owning Business Unit must ensure each application passes the UAT to be defined by that Business Unit. Where the package is processed externally an amount of time must be agreed upon, after which it is reasonable to assume the package has been accepted into production
It is recommended that product functionality tests be provided for all internal applications. Such tests will allow a packaging engineer to verify major application functionality before submitting a migrated package for UAT
Rule 54: Validate Your Packages
Validation is a means by which you can run certain standard checks (called ICEs) against your package to check for inconsistencies and non-compliance with Installer rules. Many ICEs ship with the Installer SDK in .CUB files and you can write your own. They can be run against your packages using the ORCA or MsiVal2.exe tools from the SDK. You should always fully validate your packages and, much like when working with application development, correct any reported errors and treat all warnings seriously. Validation is also a useful troubleshooting technique, particularly if you are working with 3-rd party packages.
Rule 55: Use Installer Logging
As mentioned previously, the Installer supports extensive logging. This is by far the best tool for troubleshooting Installer issues and the time spent understanding the log files will be repaid many times over when problems arise later. At the least logging should be used throughout testing. You are certain to be asked to provide verbose logs if you contact Microsoft support, so either have these ready or be prepared to reproduce your problem to gather some.
Rule 56: Use Virtual Environments for Testing and Support
Using virtual machines (VMs) for testing rather than real systems has a couple of advantages:
You can build a VM for each of the different systems in your environment and store these on a single server or workstation to be brought on-line when needed
With “undo”-style features of VMs you can easily revert to a clean, known state after each test
The same VM images can be used by testing and helpdesk staff