William here. Okay, so back to our “story from the trenches,” which started here and continued here. By now you’ve probably figured out what happened or you have a few guesses. Or, if you’re like me, you created a test network and set up a similar environment to see this issue first hand. I know I have and it’s a fun issue to see up close, especially as it appears to be several different things on the surface. So here’s what happened exactly...
The administration team has a development environment where servers, apps, and technologies are tested and a staging environment where configurations are proven prior to integration and moving solutions to production. This type of staged approach moves technology through the enterprise, works well, and generally ensures all things that move into production are fully vetted.
What happened was that a Group Policy Object was applied to the servers that strictly enforced disk quotas. The enforced limit was 100 megabytes, and the enforced limit was also applied to the System, Network Service, and Local Service accounts and those accounts had exceeded their enforced limits. Because those particular service accounts were exceeding their enforced quota limit, the services were prevented from writing to the system disk. This in turn prevented Group Policy changes from being applied. Being unable to change Group Policy had all sorts of unexpected consequences, because the servers were essentially stuck with some previously configured settings. And, most important, this meant administrators were unable to disable or modify the quota settings through Group Policy.
In this scenario where service contexts had reached an enforced quota limit, any other configuration settings that use these service contexts and required making changes to files on disk would likely also fail. As an example, this is why the administrators were unable to complete role, role service, and feature installation or removal. This also was why the server was in a state where Server Manager always had a warning that a restart was needed to complete configuration tasks and why restarting the computer did not resolve anything. Incidentally, the servers were running Windows Server 2008 R2 and Exchange Server 2010, and again, the problem had nothing to do with either.
The way I resolved the problem was to have my friend edit the disk quota entries for the system disk, raise the enforced limits on the service accounts, and then restart the computer. By restarting the computer, they triggered the finalization tasks and allowed the computer to complete any configuration tasks, such as role/role service/feature changes, stuck in a pending status. As the Group Policy Client service could process changes again and write them to the system disk, changes to Group Policy were then applied as well.
With the solution before you, it might seem an easy enough problem to solve—or it might not. Hard to say. But either way you may be wondering about the guys who spent 40 or so hours on the problem. Well, not a one of them is an idiot, I can assure you. They’re all certified to the 9s. Their problem then? Tunnel vision. They were focused on Exchange Server. They were so busy with Exchange Server and so focused on what they were doing that they couldn’t see the forest for the trees.
Know the classic rock song that says “I did a bad, bad thing, a real bad thing”? Well, Tunnel Vision is a bad, bad thing and it can make otherwise smart people do dumb things—like overlooking a solution they likely otherwise would have found.
Thanks for reading and I’m still looking for your emails about what you want me to write about next! Next up is “Postmortem: He’d dead, Jim, so why continue to flog him?” I’ll look at what this issue and similar issues (like disk space and disk write failures) look like with regard to other services. Then I’ll get back to “Configuring special preferences using editing states.”
Oh and yeah, you may be wondering what my reward was for solving my friend’s problem. Well, I’m promised a “nice dinner” and “someday soon.” Gee thanks! Note to self: get new friends. Just kidding, guys! 😉
William R. Stanek
williamstanek at aol dot com
Twitter at https://twitter.com/williamstanek