IIS6: "Supposed" Metabase Corruption
Prologue (required boredom)
I recently worked on an issue where the IIS Admin service would not start after a reboot. The customer had just finished installing a set of patches, and thought that might be related (we'll soon see that it wasn't related). The error was:
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7024
Date: xx/xx/2005
Time: xx:xx:xx PM
User: N/A
Computer: xxxxx
Description:
The IIS Admin Service service terminated with service-specific error 2148073478
0x80090006).
After much initial troubleshooting by more than enough people, metabase XML files were backed up, renamed, re-copied, scrubbed, analyzed, scrutinized, and investigated. Machinekeys were then targeted, checked for proper ACLs, etc. and after nothing worked, the conclusion was metabase corruption, and IIS should be uninstalled and reinstalled. This solved the issue, but for all the wrong (and unknown) reasons.
This had occurred, however, on 3-4 other occasions in the customer's environment, so it was time to find out why. Like clockwork, the problem occurred again, so thanks to a friend in Exchange (you know who you are) asking me to give my .02 cents, I got my chance to look at it.
Epilogue (the good part)
In my experience, the reason IISADMIN fails to start in this scenario is because of a failure to decrypt secure data in the metabase, which, of course, brings us back to our old friend Machinekeys. The machine key is a file that IIS uses to decrypt secure data in the metabase. There can be many machine key files in the MachineKeys directory, but the IIS specific one starts with "c2319", which I remember thanks to the memorable quote in Monsters, Inc. (and now you will too).
There's a little known fact that the machine key file IIS uses can be stored in one of two locations. The typical one everyone seems to know is:
%ALLUSERSPROFILE%\Application Data\Microsoft\Crypto\RSA\MachineKeys
The other location that is not so well-known is:
%windir%\Profiles\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys
This is almost always due to the fact that the machine had been originally upgraded from NT4. We looked in the registry under the following key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders
and confirmed the Common AppData value was, indeed, pointing to the C:\Windows\Profiles directory! Now we're getting somewhere!
Next we made copies of the multiple sets of machine key files, deleted the "c2319" machine key file in the \Windows\Profiles\...\MachineKeys folder and then copied the one from \Documents and Settings\...\MachineKeys into the \Windows\Profiles\...\MachineKeys folder and voilà this was a match! IISAdmin started just fine.
The keen reader should notice this means IIS WAS using the \Documents and Settings\...\MachineKeys folder initially and something changed it after the fact. This is the reason for my almost comment above. Upon further inspection, it appeared the customer had installed a piece of older software (NT4 version, perhaps?) that actually modified the registry to point to the \Windows\Profiles\...\MachineKeys directory. OUCH!
One other tidbit I noticed during the course of working on this issue is if you delete the machine key IIS uses, then attempt to start IISADMIN, another "c2319" machine key file will be created using the same filename. The problem with this is the data/private key inside will not match what was used to encrypt the metabase data initially, therefore, it's completely useless and IISADMIN will not start. This leads me to ask why it's created in the first place. If you know, I would love to hear...