Why does my software keep prompting me for a reboot on Windows Vista?

I ran into an interesting situation with a customer's application the other day. You would try to install the application, and it would prompt for a reboot. OK, so it couldn't grab that file handle yet, I'll go ahead and let it. After a reboot, the installation would continue, and then prompt for a reboot again. And this cycle would pretty much continue indefinitely. What was going on?

Since the installation was an MSI, getting to the bottom of this one was a bit easier. Just pop open the Orca SDK tool, and take a peek. In an MSI database, you have a FeatureComponent table. This links to the Component table in the MSI. So, one by one, you can just see which components it is trying to install. And, since it was failing, you can always use the trusty EventLog, which conveniently provides the ComponentId which it was unable to write.

And here is where I found the culprit. But first, some background...

In Windows 2000 and Windows XP, we wanted to protect critical Windows sytem files. However, if we locked them down, then a lot of existing software would break, since the installers would try to install Windows components. You know, just in case they weren't there. So we had to be more clever about it. What we ended up doing was implementing a detection mechanism. We would detect that you modified one of the Windows system files, allow you to do this, and then come along later and just put the original copy back.

This was imperfect for a couple of reasons.

First, the copy was not immediate. I know the pain of this one first hand. I had an application that believed that not only did it need to install critical Windows system files, but it needed to install pretty much all of them. No problem, the original ones were dropped back in place and everything was fine. Then, I went to uninstall the application. It assumed that, since it dropped those critical Windows system files, it would be polite enough to go ahead and remove them for me. Just to be thorough. The problem was, it prompted for a reboot immediately afterwards. And, inconveniently, before the system file checker had a chance to lay them back down for me. So, upon reboot, Windows was unable to start because those files were never replaced. Yuck.

The other reason is that you could always get around this. We implemented this by laying down two copies of the system files. One in the normal location, and the other in the dllcache directory. Which means you could circumvent the protection if you were smart about it. If you lay down your copy in the dllcache directory first, and then lay down in the original location, then the system file checker would replace the copy in the original location with the copy in dllcache, which would be your modified copy.

And that is exactly what this application was trying to do. It wanted to replace the keyboard handlers from Windows with its own, so it was laying them down in dllcache first, and then laying them down in the original location. This little trick used to work on Windows 2000 and XP.

It doesn't work so great any more.

Because our previous method of approaching the problem was imperfect, we came up with a better one. We now ACL critical Windows system files to only allow the TrustedInstaller user to modify them. Not even System can modify these files, and certainly not Administrator (elevated or not). This is called Windows Resource Protection. How do we keep existing applications from breaking, since this is the reason we didn't implement the old technique in the first place? We lie to them. We detect if you are an installation program (using the same set of heuristics that we use to automatically mark setup applications to elevate), and if you are, then we accept your request to modify a protected system resource, we return success, but we never actually do anything to that file. If you are not detected as a setup program, then we don't lie - we return the access denied message. (Of course, you can apply the WRPMitigation shim using Compatibility Administrator if you need that behavior elsewhere, and we disable this check once you manifest your application with a Windows Vista manifest.)

As a result, we dont have dllcache any more. So, this system was trying to drop a new file into dllcache, it wasn't able to because that folder isn't there any more, so it figured that a reboot would help. And every single reboot, it wasn't there, so it figured yet another reboot might help next time. And on it went...

Comments (12)

  1. thats very interesting, good to hear that we have a new implementation in windows vista.

  2. matt says:

    This is great information. However, the "installation program heuristics" sounds a little bit fishy. I have to assume that if I write my own installation routines that you won’t accurately mark my program to elevate.

  3. cjacks says:

    We detect installation programs by looking for the strings "setup," "install," and "update" in the program name and the string table. We also look for characteristics that indicate the exe was authored by a number of known authoring tools. This "auto detection" is designed for legacy applications. If you are writing one now, just add an application manifest specifying that you requireAdministrator, and we will elevate (and disable these heuristics, which are designed to help out software written before this manifest schema existed). Maybe I’ll put together a complete post on this…

  4. JonDW says:

    You mention the TrustedInstaller ‘protection’ process and it’s great that you protect system files from install programs (nothing else should run with admin privileges so is not an issue), but you need to provide a (sensible) mechanism to bypass this if the need arises.

    In system support we come across this type of need all the time: A complicated situation would be if a user tries to run an app but it fails to work properly, we examine the system and the app dependencies and find that the only difference between this and a working one is a certain file (DLL?). So we need to replace the DLL temporarily to prove that is the issue before going down the route of finding out why it is there (maybe it is a new version which has a bug?) – however we can’t do that anymore so we have no way of finding out the cause – big problem (Especially if it’s the CEOs machine you’re trying to get working).

    A simpler situation would be that we use the hosts file (eg for a home network or a secure standalone network, or just to avoid a dependancy on DNS), which every system has used since IP was created, but not in Vista because it’s in a directory which no one has write permissions to! Not only that but you can’t even take ownership. I’m all for security, but this is a step too far.

  5. cjacks says:

    What is the scenario for which you want to replace system files? A debugger can tell you if one particular file happens to be causing an issue for you much easier than trial and error, and keeps your system in a state that is comprehensible and tested. We provide side by side installation of many system dlls where versioning can be an issue, and you can select the implementation via manifest.

    Also, the hosts file is not protected by WRP. Because it affects system-wide settings, the ACLs require admin rights as they always have. The difference is that more people are running as standard users because of UAC, not that we have changed the underlying approach to securing this file appropriately.

  6. Guid says:

    There goes my inetcomm.dll hack .. *sigh*.

  7. Mark says:

    >a (sensible) mechanism to bypass this

    >the scenario for which you want to replace system files?

    I continue to use code in FoxPro 2.6 DOS.  On multiple processors, ntvdm.exe can hang, causing data corruption.  Setting processor affinity using imagecfg on ntvdm.exe can fix this.  But I can’t figure out how to replace ntvdm.exe.



  8. Mark says:

    Aha, re. last post:  I spent hours looking for the Windows Vista Windows System Resource Manager to set processor affinity.  Found the guide on TechNet, but not how to start the WSRM, or download it for Vista (there’s an ISO on TechNet for W2k3 Enterprise, which I haven’t tried mounting on Vista yet).  Where is it?

    But: I found the Vista START command, and used the /AFFINITY 0x1 switch in a batch file to run FoxPro DOS.  Works now.

  9. Jamie says:

    Hi, I’ve encountered the same problem in Vista where my application attempts to replace the keyboard handlers (kbdusr.dll) from Windows to the dllcache folder. I’m installing using MSI. Can you please advice how should I implement the fix? I’m a beginner and not sure how do I implement the WRPMitigation shim in the MSI. Thanks!

  10. cjacks says:

    If the problem is that it’s trying to drop into dllcache (which doesn’t exist), you could create a dllcache folder (so the file has someplace to land) and just have it drop there and be ignored (the attempt to overwrite the actual system file will have WRPMitigation applied already since it’s a setup) potentially.

    If you want to truly fix it, you can just edit the MSI. The Orca SDK tool is pretty bare-bones, but it has the advantage of being free. Open the MSI in Orca, and remove references to both the system file and the dllcache file so the installer no longer tries to drop it. And, of course, there are many full-featured setup creation packages, you could re-package, and so forth.

  11. Cris Mooney says:

    Evolving. Yeah, sure.

    Some may find it comforting to know God is alive, and here on earth. I, however, have lived long enough to know better.

    I guess I should not be surprised. Not even the IRS has learned that the more convoluted rules you put in place, the more holes you generate. But, I guess MS has the as much money as God, and the IRS, so I should not be surprised at the similarity.

    If there is anyone over 21 at MS that has any inclination to tutor these young overly zealous innocents that have no idea what they are up to, I would appreciate you stepping in to guide them.

    I know all you guys don’t get paid for doing things the simple way, or if you don’t reinvent the wheel. But I really wish you would get out of the way of those of us who would like to get things done instead of spending all day trying to figure out why we can’t get anything done (or working around your "we know best" logic that is often wrong).

    In specific, how about just protecting files and letting folks know with straight forward alerts, and providing us simple overrides. You are quite welcome to detect and report infractions, and offer tools to clean up. But, you are not as smart as you think you are, and every day I get closer to throwing this OS out as you screw it up more and more.

    I am tired of the further corruption on the once attempted slogan of MS "where do you want to go?"… since all to often my answer is met with "why would you want to go there?". Get out of my way, and once I am there you might just be able to understand.

    My current Oracle install is unable to write my own TEMP directory now, though it was yesterday, and 5 hours later I am still unable to figure out why despite the fact I’ve used MS operating systems as a programmer for about 20 years. Very useful boys. Real evolution.

    Cris Mooney

  12. Cris Mooney says:

    As my original post last week started "Evolving…", I am of the belief that this BLOG had references to "evolving the MS platform" and such, which I feel may have been changed. However, it is possible that it was just the simple "Software Evolution" I see paired with "UAC" in the upper right of the page today that got my goat.

    Furthermore, the "Evolving…" context I perceived paired with User Access Control (UAC) "implementation possessive" comments like "Because our previous method of approaching the problem was imperfect, we came up with a better one…" led me to believe Chris was a key part of the UAC design and development team.

    My point is that certain aspects of my post only make contextual sense if seen in the light that Chris was part of the User Access Control team, and that he believed they deserved an "evolving" title.

    In today’s light, I am inclined to believe that Chris is not part of the UAC team, but instead as much a victim as I am. Given this, my post is a bit harsh and misdirected, though I stand by everything I wrote – I just need to find out where to direct it (he has provided some direction at "http://tinyurl.com/5s8ro4"). I believe Chris should be as upset as I am, and I suspect that is all I should have commented about in this forum.

    I stand by my condemnation that UAC, while having reasonable goals as one who castrates to help control world population, is poorly implemented, its release premature, and its impact very detrimental to multitudes of users – but Chris does not appear blameworthy.

    Sorry for the misdirected response prompted by UAC’s assault on me,

    Cris Mooney

Skip to main content