Reverse engineering a setup - lesson 1 - .NET Framework 1.1

  • Article

Hey all, as I promised a while back, I wanted to walk through some examples of how I approach reverse engineering a setup package to learn how it works, make any necessary changes, figure out command line switches, etc. I apologize for not posting anything in a while, I was spending some time with my family who was in town from Texas and then I managed to get sick right after they left. But I'm back now :-)

I'm going to start with a relatively straight-forward reverse engineering example - the .NET Framework 1.1 package. I chose this first because that setup is very simple - no setup data files, no configurable setup UI options, etc. Yet it is complicated enough to be interesting and it is a setup that a lot of people want to include in their setup packages and/or have had issues trying to install. I'm going to try to approach this from the perspective of someone who is familiar with different types of setup technologies in a broad sense, but who has not yet seen this particular setup before (I may get a little off track on this part because I'm so familiar with this setup from my previous work, but I'll try.....)

Step 1 - figure out what packaging technology is being used

When I download the .NET Framework 1.1 I see that it is a single EXE named dotnetfx.exe. When I double-click on it, setup asks me if I want to install and when I say Yes it proceeds to extract files to the %temp% folder on my machine. Then a EULA appears. When I go to %temp% after the EULA appears, I see a folder named ixp000.tmp. This tells me that the package in question is an IExpress self-extracting EXE. Since I know it is IExpress, I can now use some well-known IExpress command lines to extract the package to a folder of my choosing. I will run dotnetfx.exe /t:c:\aaron /c to create a folder named c:\aaron on my machine and unpack the .NET Framework files to this folder. The /t flag specifies the folder to extract to, and the /c flag specifies that I want to extract the files only and not run the setup after extraction.

Side note for setup developers - I found some IExpress docs on the web here, and you can also find iexpress.exe in %windir%\system32 on an XP Professional machine. It is a useful tool for packaging up multiple files into a single package but it is also a fairly old technology and has some limitations that may not make it practical to use, depending on your scenario.

Step 2 - figure out what setup/installation technology is being used

Now that I have extracted the files, I will go to the c:\aaron folder that I created above.

  • The first thing I see in the folder is a file named netfx.msi - this tells me that the setup is a Windows Installer-based setup.
  • I see a file named netfx1.cab, and opening it in a file extraction utility such as WinZip, or in Windows Explorer on Windows XP or higher shows me that there are 204 files with long funny looking names inside. My knowledge of Windows Installer tells me that these are the files that will be installed by netfx.msi because I know that if you install files from a CAB file using Windows Installer, you are required to name files in the cab using the File token from the File table of the MSI - hence the funny names.
  • There are 2 self-extracting EXE files named instmsi.exe and instmsiw.exe, and I recognize those as the setup packages for Windows Installer itself. This means to me that this setup will install the required version of Windows Installer for the user if it is not already present on the machine.
  • Finally, I see a file named install.exe. Initially I am not sure what this is for, but I guess that it is a bootstrapper EXE that will launch netfx.msi and will also launch instmsi.exe or instmsiw.exe if it is needed. Out of curiousity, I run install.exe /? and observe the usage information that appears and it does indeed appear to be a .NET Framework setup bootstrapper.

Step 3 - figure out the basics of how the setup works

Most of the information I need to figure out how the setup works will be gained by installing Orca and then looking at the MSI, but first I want to look at a basic overview of what the setup is doing. I will start by double-clicking install.exe - when I do so, the .NET Framework setup starts and the EULA appears. Now I will take a look in the %temp% directory and see if the setup happens to be creating a log file since most setups will create some kind of log for debugging and troubleshooting. I can sort the %temp% directory by Date Modified, and I see a log named dotnetfx.log there. Opening this up and reading it gives me some ideas of what the install.exe process is doing - it is checking several system requirements such as OS type, OS service pack, Internet Explorer version, MDAC version. It is also loading msi.dll to determine the version of Windows Installer on my system - this must be how it decides whether or not to run instmsi.exe/instmsiw.exe. There is also a call to a function called StopDarwinService. I'm not sure why this is here, but I know that Darwin is a code word for Windows Installer, and I know that on Windows NT-based OS's Windows Installer is actually a service instead of just an application, so I am guessing that install.exe is trying to stop the Windows Installer service before starting installation of the .NET Framework.

After I look at all of this, I go ahead and step through the UI and install the .NET Framework 1.1, and then re-open the log file dotnetfx.log. I see that it lists the command line that is passed to the MSIInstallProduct function - and I know that this is an API call for an application to install a product using Windows Installer. Then I see a return code for that API call - fortunately it is 0 in this case to tell me that the .NET Framework 1.1 installed correctly. Then I see another call to StopDarwinService, so this install.exe application is stopping the Windows Installer service again after installation is complete - kind of strange.

Step 4 - figure out details about how the setup works

Now that I have a basic understanding of how the .NET Framework 1.1 setup works, I'm going to use my Windows Installer expertise to dig a little deeper. The first thing I will do is install Orca, then right-click on netfx.msi and choose Edit with Orca to view the contents of the MSI. There is a ton of data to wade through in an MSI and it is overwhelming at first, so I will only look at a few key things here:

  • Go to the property table and look at the product code, upgrade code, and any other interesting properties that are set. Most of the properties in this table will be standard data documented in msi.chm, but sometimes there may be interesting custom properties set here.
  • Go to the Tools menu and see if the Dialog Preview item is enabled. If it is, that means that this setup package uses UI that is contained in the MSI itself. For the .NET Framework 1.1, it is enabled and we can look at all of the possible UI screens. If you look closely at the Dialog Preview and cross reference it with data in the UI tables of the MSI (the Control, ControlEvent, and Dialog tables among others), you can find a bug in this setup. The ActionDialog is shown during installation, and it has a Control_Default set to cancel which means that if you press Enter with this dialog on-screen it will trigger the cancel action. However, an MSI-based setup should not allow you to cancel if it reaches the commit phase of setup (which is normally distinguished by the cancel button being greyed out or disappearing from the setup UI progress screen). However, for the .NET Framework 1.1, you can press Enter even when the cancel button is gone and it will trigger the cancel action.
  • Go to the InstallUISequence table and observe the conditions and order of actions when running the setup with full setup UI
  • Go to the AdminUISequence table and observe the conditions and order of actions when running the setup in administrator mode. This is often the source of some bugs if someone forgets to author an action or condition on the AdminUISequence table but it exists in the InstallUISequence - this will cause behavior differences for setup depending on whether or not it is run in admin mode.
  • Go to the InstallExecuteSequence table and observe the conditions and order of actions when setup is actually installing/uninstalling
  • Go to the LaunchCondition table and see if there are any actions that will block setup from installing. In the case of the .NET Framework 1.1, it will block installation if the OS being installed to is 64-bit.
  • Go to the Component table and observe the attributes and conditions for components to be installed. In the case of the .NET Framework 1.1 - some files will not install on certain OS types (for example, ASP.NET is not supported on OS's before Windows 2000). Also, all files have at least an attribute of 8 meaning that Windows Installer will respect and increment/decrement the legacy shared DLL refcounting scheme for these files. In addition, files that have OS conditions have the attribute of 64 (= transitive), which means that Windows Installer will re-evaluate component conditions when a repair is attempted. This is nearly always needed for files with OS conditions because we want to have the files added or removed as appropriate when performing a repair after an OS upgrade or downgrade. For example - if you install the .NET Framework 1.1 on Windows 98/ME then upgrade to Windows XP and then perform a repair, we want the ASP.NET files to be installed for the user.

There are many more things that we could look at in an MSI, but the above are good starting points. Often for other setups you will see items in tables that will lead you on trails through other tables. Most commonly, if there are launch conditions that depend on the existence of other applications, you will be led to the AppSearch table to figure out exactly what file or registry data the setup is looking for when deciding whether or not to block.

Step 5 - install with verbose logging and look at the MSI log file

Reading an MSI log file is more of an art than a science but can often be useful when reverse engineering a setup. For the .NET Framework 1.1, we know that because it is an MSI we can set the verbose logging flags before running setup - HKLM\Software\Policies\Microsoft\Windows\Installer, Debug (REG_DWORD = 7) and Logging (REG_SZ = voicewarmup!). These will give us a verbose log named msi*.log in %temp% where * is a randomly generated set of characters.

I also noticed when running install.exe /? it says that if you pass a /l flag it will generate netfx.log in %temp%. Since I also noticed that the log created by running install.exe was named dotnetfx.log and not netfx.log, I might be curious what the difference is and run install.exe /l from c:\aaron to see what it does. When I install this way I see that the resulting file %temp%\netfx.log is essentially the equivalent of a Windows Installer verbose log file without needing those registry keys to be set. So now I can take a look at netfx.log (or msi*.log if I choose) and view the step-by-step process of this setup.

With that, the process of reverse engineering the .NET Framework 1.1 setup is done for the common cases. There is of course more detail that may be needed depending on your scenarios, but much of that will require more in-depth analysis of the MSI tables and/or the verbose MSI log files. I will leave that for a later lesson.

Side note for the curious - the reason that the .NET Framework 1.1 setup stops the Windows Installer service is because Windows Installer uses pieces of the .NET Framework to install assemblies to the GAC (the MSIAssembly and MSIAssemblyName tables in the MSI) in Windows Installer 2.0 and higher. This creates a classic chicken-n-egg problem - Windows Installer is going to install the .NET Framework but Windows Installer needs the .NET Framework in order to do so. Because of that, Windows Installer has some special logic to bootstrap the part of the .NET Framework it needs to install assemblies, but since Windows Installer is run as a service, it could already be running from a previous setup and we want to stop the service to unload any older versions of the .NET Framework from memory so that the most recent version will be used.

I hope this post is at least a little bit useful to give some insight into how I approach the process of taking apart a setup and figuring out how it works. I will post more later with a more complicated example of a setup that uses information from data files, etc. Let me know via comments or emails if you have any questions about any of the above.....