Windows Resource Protection (WRP) and ActiveX Control Installation on Windows Vista

I came across this problem today with a customer - Windows Resource Protection preventing ActiveX control installation on Windows Vista.

First, some background. Windows Resource Protection is the feature we put in place to help keep Windows in a known state. For example, if we make a change to user32.dll, we may also need to make a change to shell32.dll to accommodate this change, so the update will contain both files. If an application then comes along and writes an older version of shell32.dll, it may no longer work with the newer version of user32.dll. The operating system is no longer in a known and tested state.

Of course, a lot of applications try to drop them anyway, and for application compatibility reasons we don't want to just break them. So, we had conflicting goals. On XP, we would address this with System File Protection (SFP). Essentially, we would watch these files, and if you changed them, we would go ahead and let you do that, but then we'd come along a few seconds later and replace them with the original version. The application doesn't fail, but Windows still returns to a known and tested state. It seemed like a good solution at the time. However, there are a couple of issues with this.

First, since we are copying the original files back, people figured out that the files had to come from somewhere, and they figured out where that somewhere was. They would then copy a file to that somewhere first, then copy the Windows file in its location, and when we triggered that it was changed we would copy it over ... from the location you just put your stuff in! So you could get around it.

Second, the amount of time it would take to replace files might be too long. I had one game application that had 4 CDs for the install. About 1 CD worth of content was most every file from Windows. When they build the installer, apparently they weren't sure that Windows would be there. Fair enough, we would notice that the installer did that, and then copy the original files back, and no harm done. However, at uninstall, it removed all of these Windows files, because it thought it had dropped them. Then, before the few seconds had passed, it would prompt for a reboot. If you immediately say "yes", then it reboots - without those Windows files. Which makes it really hard to boot, because critical files are gone. I had to do a repair install of Windows. At 7pm. On a Sunday. With a flight the next morning. Not my best night.

Clearly, there was room for improvement. So, in Windows Vista, we just don't let you write to Windows operating system files at all. They are ACL'd to only allow the TrustedInstaller ACE to write to them, and this is handled by the TrustedInstaller service, which Windows Update calls into. But we knew that apps will still break, and compatibility is hugely important, so we took another compatibility approach: we just lie. With the WRPMitigation shim, you can try to write to a protected file, and we'll tell you that you did. We won't actually let you, but we'll let you think you did. It turns out that that's enough just about all of the time.

Now, you can apply that shim manually, or we'll apply it for you if we detect that you are a setup using the UAC setup-detection heuristics. We also apply the shim to regsvr32.exe.

Now that we have that out of the way, we were investigating an issue where an ActiveX control wouldn't install. We would navigate to a page, it would prompt for control installation, we would approve it, but then we'd just see the placeholder. It never installed. What was going on?

So, we opened the source view and searched for the object tag. We then pointed our browser to the cab file it was looking for, so it would download and we could save it to the desktop. We then extracted the CAB, and took a look. Here's the ini file that we found in the cab (file names elided to protect the guilty):

; Sample INF file for clientautomation.dll
[version]
; version signature (same for both NT and Win95) do not remove
signature="$CHICAGO$"
AdvancedINF=2.0

[Add.Code]
<the application's code>.dll=<the application's code>.dll
msscript.ocx=msscript.ocx

; needed DLL
[<the application's code>.dll]
file-win32-x86=thiscab
clsid={<the applicaton's GUID>}
FileVersion=1,0,0,14
RegisterServer=yes

[msscript.ocx]
file-win32-x86=thiscab
clsid={0E59F1D5-1FBE-11D0-8FF2-00A0D10038BC}
FileVersion=1,0,0,7615
DestDir=11
RegisterServer=yes

;end of INF file

The DestDir entry for msscript.ocx is interesting - it wants to drop the file in the system directory. We took a look to see if we could find it, and indeed msscript.ocx is already in the system32 directory, and it happens to be a WRP protected file. So it can't drop it! (What they were actually trying to drop was the Windows 2000 version of this file.) And, since IE doesn't have WRPMitigation applied, the installer never succeeded.

So, we just went ahead and created a new CAB file to test out. We removed msscript.ocx, and modified the ini file to remove the references to msscript, and gave it a go - the ActiveX control worked just fine, and functionality was restored.

Another compatibility bug closed.