William here. I know I promised to continue the discussion of Group Policy preferences. Next up was supposed to be a look at configuring special preferences by using editing states, but trust me you’re going to appreciate this interruption just as much. So here’s what happened: A friend of mine who works in IT (and is a very senior administrator for a large enterprise) called me up to help diagnose a problem they were having with their servers. Here’s the scenario and the problem they were having:
They were pilot testing Exchange Server 2010. The test was going well—almost perfectly. Everyone was ecstatic until they found that they could no longer make basic changes to the operating system and Exchange Management Console failed to start.
Thinking something was wrong with the Exchange configuration, they tried to modify the server configuration. However, they could not successfully complete the installation or removal of roles, role services or features. Server Manager had a perpetual warning: Restart required to complete configuration. The only problem is the restart never resolved the problem.
Let the troubleshooting commence? Oh yeah… First stop: the application event logs. Exchange sources had generated over a thousand errors. Exchange was generating about 20 or so errors a minute. What in the world was happening? Here is an example of an error they were getting:
(Microsoft.Exchange.Search.MailboxCrawlFailedException: Failed to get entry id or create folders to index ---> Microsoft.Exchange.Data.Storage.ObjectNotFoundException: The store ID provided isn’t an ID of an item. ---> Microsoft.Mapi.MapiExceptionInvalidEntryId: MapiExceptionInvalidEntryId: Unable to open entry ID.
Seems like a problem with Search, right? Well, there were similar events for the Windows Search Service:
The Windows Search Service has failed to create the new search index. Internal error, Failed to create an application directory.
Okay, so troubleshooting seems to lead to a problem with Search indexing or such. As you might imagine, troubleshooting led them all over the place. They tried all sorts of things. Nothing worked. They dug into the event logs a bit further, spotted intermittent ESENT errors mixed in with the multitude of other Exchange errors. Literally, there were thousands of Exchange errors to sift through by now. The ESENT errors said:
An attempt to create the file tmp.edb failed with the system error 112 “There is not enough space on the disk.” The create file operation will fail with the error -1808.
These new errors spotted seemed to point to a problem with disk space—and a disk space problem should be easy enough to resolve right? Well, not really. Troubleshooting a disk space problem took them nowhere as there wasn’t a disk space problem. As they still could not resolve their problem, it was back to the logs and diagnostics. There were all sorts of errors in the logs, but none seemed to be related to the problem at hand and also could lead in all sorts of different directions.
Another error that seemed interesting was from Application Management Group Policy:
Failed to apply changes to software installation settings. The Active Directory path could not be updated on the client. The error was: %%112
They were thinking how in the world could all these errors be related? Were these errors even related?
The final solution: Hey these are test servers anyway. Let’s just reinstall and call it a day. Reinstall didn’t work either.
So about 40 or so hours into the problem and with the weekend fast approaching, I get a phone call. “It’s an emergency, we’re having a problem and can’t resolve it. We’ve tried everything we can think of. I don’t want to be here all weekend. Can you help us out?”
Intrigued, I accepted and asked them to send me detailed information on the problem. My friend was rather incredulous and insisted this was one problem I had to be on site to solve. I asked him to just humor me and send me the details. Before getting into my troubleshooting process, do you have any idea what was wrong? Is this an Exchange problem? Is this a disk problem? Is this an Active Directory problem?
My biggest clue to what was wrong had nothing to do with Exchange and everything to do with what was happening on the server. The server had disk errors and write failures. Server Manager was prompting them to restart to complete configuration changes, but restart did not resolve the problem. The problem? The solution? Well, the solution was as unexpected to them as the problem. The problem had nothing to do with Exchange, nothing to do with Active Directory, nothing to do with disk space, and yet at the same time everything to do with the same. The problem I said was disk quotas. The solution was to remove the disk quotas.
My friend had a big laugh. He thought I was joking but decided to humor me by checking Group Policy. He checked Group Policy, determined which GPOs were being applied and said no disk quotas were being applied. I told him to check again. He laughed a bit more. I said I was serious he checked and sent me a screenshot of Administrative Template\System\Disk Quotas in the only applicable GPO. I told him to check RSoP to ensure that was the only GPO applicable. He did and sent me the results. No Disk Quotas were being applied. I told him to do me a favor and right-click the system disk, select Properties, click Show Quota Settings on the Quota tab. I asked if the status read: Disk quotas are enabled. He said yes. I asked if the Disk quota settings were dimmed (meaning he could not set them). He said yes. I told him to see if local policy had disk quotas but local policy didn’t have disk quotas.
So was there a disk quota problem or not? Was I just going out on a crazy limb?
Got any idea what the problem is? Write to me with your answers at williamstanek at aol dot com. Or just tune in on Thursday for the sequel: “Horror story from the trenches & a few answers.”
William R. Stanek
williamstanek at aol dot com
Twitter at williamstanek