The Shadow in the Machine

There's some ethereal guy called "system" wandering around inside my servers stealing stuff. It's a bit like when you were a kid and your parents hid things from you. When my hamster died, my Dad told me it had gone to live on a farm. Of course, when I got a bit older and my Grandmother passed away I realized he was telling fibs because she suffered from hay fever and was afraid of cows, so there's no way she would go and live on a farm. Yet, even though I've now reached the age where people generally feel they can tell me the truth (often, worryingly, to may face), I discover that Windows Server 2008 is still hiding stuff from me.

I suppose it's all related to the poor decisions I made when ordering my servers. Ever since I set them up with Hyper-V, and virtualized all the machines I find I need for my diminutive network here at chez Derbyshire, I've been struggling for disk space. It seems that 300GB is just an aperitif when you get serious about virtualization. OK, so the Server 2008 docs do say you need a minimum of 40GB for a standard installation, but I made the VMs only 30GB. My Windows 2003 Server VM that runs ISA is 30GB and has 22GB free. Though the Windows 2008 VMs that don't have very much at all installed are both showing only 8GB free of 30GB so maybe they were right...

Anyway, although my math skills may have waned since leaving school, I managed to calculate that I could run four 30GB VMs on a 150GB disk (yes, I know you're supposed to put them on separate disks, but my network loading is somewhat less than heavy - none of the machines goes above about 3% CPU utilization). Yet I could never get all of them onto the disk. OK, so Hyper-V does use some extra space for each VM when it's running (about 2GB for a 30GB VM), but I should still have space for four of them. In fact, as one of the VMs is a tightly locked down copy of Windows XP used for browsing and troubleshooting while I'm pretending to be a system administrator, and its only 10GB, I should have space left to swing several cats round simultaneously. But I could only ever fit the three 30GB VMs onto the disk.

I did try reducing the size of the VM with the 20+GB of free space using the Hyper-V tools, but (as they say in several blog posts I found) it's not a trivial exercise. You can convert the VM to a dynamic disk and compact it (it went down to 5.6GB), but when you convert it back to a fixed size disk there is no option to specify the size because it automatically grows to the partition size specified in its boot sector. You need to edit the partition size to reduce the physical disk size, and I didn't fancy playing round with that on a Sunday afternoon. Please, Hyper-V guys, can we have a tool to do this (and better docs that explain why you are wasting good gardening time playing with the existing tools).

So I've put off dealing with this issue for the last few months since setting everything up, but now that we are suddenly experiencing tropical conditions here in Little Olde England I decided I needed to find a way to get this sorted so I could shut down the "spare" server and reduce the searing temperatures in my server cabinet (see last week's ramblings for details). So out comes the calculator: three times 32 (the three VMs on the disk) equals 96. Check the disk properties and it says 133GB used, 14GB free. So where did all the spare disk space go? Maybe it's got some lost clusters, so I schedule a disk check and reboot. After restarting, look in the bootlog.txt file and - lo and behold - around 40GB is described as "in use by the system". What on earth for? Is it hiding secret documents from me? Does it need some spare disk space for playing Mahjong when nobody is watching? Is it full of dead hamsters that never made it to the farm?

So I did the usual, check the properties of each folder and add the total sizes together. 96GB. Then turn on "view operating system files" and do the same. Still 96GB. See what I mean? Most things made of metal expand when they get hot, so my disk drives should be getting bigger not smaller. I even considered looking underneath to see if there was a pool of congealed clusters that had leaked out of the bottom (OK, so not really). But then - "Aha!" - I remember seeing the occasional error message in Event Log about something to do with "Not sufficient disk space to create shadow copies". One of those messages that I've conveniently been ignoring.

So after furkling through the properties of the disk, I find that Windows has allocated 41GB to shadow copies. I suppose the fact that you can see this in the Shadow Copies tab of the Properties dialog means that it's not technically "hidden", but where is the file? You can't see it in Windows Explorer, even with "show operating system files" and "show hidden files" turned on. And how do you stop it happening? After reading some online docs and blog posts, it became clear that the shadow copies are there because the disk has a share set up, and it allows connected users to get at the previous deleted or updated data that was on the disk. I have the disk with the VMs on shared at admin level to be able to do backups, so I can't really just turn off sharing. And according to the Shadow Copies dialog, they are disabled on the disk anyway.

I had a go with the vssadmin command line tool that is part of Server 2008, but that said it couldn't find any shadow copies (that system guy obviously hides stuff from Windows as well). It seems that vssadmin can only delete shadow copies you create manually. And to make it worse, the more I tried enabling and disabling shadow copies, the larger the shadow copy got. After ten minutes it had grown to 55GB! In the end, more by luck that any administrative capability on my part, I found that by clicking the Schedule button and deleting the two existing scheduled shadow copy tasks, and setting the size to 300MB (the minimum you can specify), the shadow copies magically just disappeared. Suddenly I've got tons of spare disk space on all of my drives!

Of course, now I can't sleep at night worrying that shadow copies aren't occurring for my shares, but seeing as how: a) I didn't know there were there before, b) I've never had reason to use them, and c) I can't see why I'd need to get a previous copy of a VM when they are all exported and backed up in multiple places regularly (and don't actually change that much anyway), maybe I'm just being as paranoid as usual.

I suppose I'll find out one day when the sky does fall in, and I can't get to the Internet to update my blog. You'll probably be able to tell when this happens because the post will suddenly end in mid