Hyper-Ventilation, Act III

After approximately two weeks of intermittent network upgrades, I seem to still have a working network. I guess at least that's something to be thankful for. But it's still not fulfilled the original plan. And much hyper-ventilation has occurred during the process, particularly when watching those little green caterpillars crawl across the endless "Please wait..." dialogs, and wondering what the next error dialog will say...

Scene I: "Virtual Notworks"

One of the hardest parts of the configuration process for Hyper-V (at least for me) seems to be understanding virtual networking, and applying appropriate network settings. Despite reading up on it beforehand and thinking I grasped how it worked, I encountered endless error messages about multiple gateways and duplicated connections while trying to configure the network connections for the VMs and the host machine. It turns out that I was probably being as dense as usual in that I missed the obvious point about what the virtual switches that Hyper-V creates actually do.

Stepping back, the scenario is a machine with three physical network cards. I want to use one to connect the host (physical) server O/S to the internal network, one to connect specific virtual machines to the internal network, and one to connect specific virtual machines to the outside world. One of the virtual machines, hosting the firewall and mail server, will connect to both.

So step one is to use the Virtual Network Manager to create two virtual switches and connect these to the two physical NICs. When you look in the Manage Network Connections dialog on the host, you see - as expected - the three physical connections and two additional connections. What's confusing is that they are all "Connections". It's only when you examine the properties of each one that you realize two of them are bound to the new Microsoft Virtual Switch protocol and nothing else. At this point, it's a good idea to rename these connections so the name contains the word "Switch" to help you easily identify them.

So, now you can use the Hyper-V Settings dialog for each virtual machine to add the appropriate network connection(s) to that VM. What this actually does is create a connection within the virtual machine and "plug it into" the virtual switch you specify. You can, of course, plug the connections within more than one VM into each virtual switch. It really does help to think of the "switch" connections as "real" network switches like the 4 and 8 port ones you can buy from your local computer store. Ben Armstrong has some nice pictures in his blog post that illustrate this.

What's confusing in almost every post and document I've read is the use of the word "host" or "parent" to refer to the physical machine and its O/S. It implies that the VMs somehow run "inside" the O/S that is running directly on the hardware. I've started to refer to it in my head as the "base machine" and "base O/S" instead. While the base O/S and the Hyper-V runtime implement the virtual switches, these switches are not "inside" the base O/S. The Virtual Network Manager effectively moves them out of the base O/S. So the confusing part (at least for me) was what do I do with the two new "Connections" that are visible in the Manage Network Connections dialog of the base O/S. I know that I must configure the non-virtual connection that the base O/S will use to talk to the network. And I know that I have to configure, within each VM, the connections that I add to these VMs using the Hyper-V Settings dialog.

Unable to find any guidance on the matter, I assumed that the two "Connections" visible in the base O/S were being used to link the physical NICs to the virtual switches, hence the quandary over how to configure them. As it was, I followed the "know nowt, do nothing" approach and left them set to the default of "Obtain an IP address automatically". It was only after a day or so I noticed that file copy speed was erratic, and that the physical servers each had two different IP addresses in the domain DNS list.

Probably you are already hopping up and down, and waving your arms to try and attract my attention, with the answer. My error is obvious now, but wasn't at the time. What the Virtual Network manager does is steal the physical NIC and plug it into a virtual switch. However, this would cause a problem if the machine only had one physical NIC, so it tries to be helpful by automatically creating a new connection in the base O/S for each virtual switch it creates, and then plugs these new connections into the appropriate virtual switch. This means that the base O/S still has access to the physical NIC.

 However, this also means that, on a multi-NIC machine, you can easily get duplicate connections in the base O/S. For example, in my case I already have a connection in the base O/S that's nailed to one of the physical NICs, and that's all I need. But when I dig a bit of CAT6 out of the junk box and plug one of the other physical NICs in the machine into the network, the virtual switch links it to one of the un-configured "Connections" in the base O/S. This means I've got two connections from the base O/S to the network for the same machine, but with different IP addresses.

If you managed to follow that rambling description, you'll be pleased to know that it finally dawned on me what was going on, and I confirmed it when I finally came across this advice in the last comment to a long blog post on the subject: "...if you have multiple physical NICs, disable the duplicated connections in the base O/S that the Virtual Network Manager creates". In other words, in the Manage Network Connections list in the base O/S, unplug all the "Connections" (not the "Switches") that Hyper-V so helpfully created (and, coincidently, you don't know what to do with). Unless, of course, you need the base O/S to talk to more than one network, but that probably negates the whole point of having a vanilla and minimum base O/S install that runs multiple VMs containing all the complicated stuff.

Note: In Windows Server 2008 R2 you can untick the Allow management operating system to share this network adapter option in Virtual Network Manager to remove these duplicated connections from the base O/S so that updates and patches applied in the future do not re-enable them.

By the way, if you get odd messages about duplicate connection names, gateways, or other stuff while configuring network connections within a VM, it's worth checking for any "orphan" unconnected connections that the Virtual Network Manager may have created. In fact, it's worth doing this anyway to avoid "connections problems" when you try to import an exported VM if the roof falls in. Use the process described in https://support.microsoft.com/kb/269155 to find these and uninstall them.

Scene II: "An Exchange of Plan"

All that remains now is to get one more VM up and running to host my firewall, public DNS, and Exchange Server. One more day's work and it will all be done. All the hardware is in place, all the infrastructure and networks installed, and most of it is performing without filling the Event Logs with those nasty "red cross" messages. Maybe I can phone the lad down the road who is finding a home for my old boxes and get rid of the last one...

Or maybe not. I just read the "ReadMe" file for ISA 2006 and discovered that I can't run it on a 64-bit machine. Yet Exchange Server really wants 64-bit to work properly (according to the docs). And why should I run 32-bit software on my gleaming new 64-bit boxes anyway? So I check out the replacement, Forefront, but it's still in Beta. Do I want to chance that on my only connection to the outside world? Probably not.

And after reading How to Transition (or Migrate) to Microsoft Exchange Server 2007 I begin to wonder how migration will go when I'm coming from a box that was originally upgraded from Exchange 5.5 to Exchange Server 2000. Do I really need an Exchange Server? Yes, it's useful for experimenting and researching stuff I work on, but the administrative overhead - never mind the upgrade hassle I can see lurking in the wings - probably far outweighs the gains.

In fact, if it's comparable to the struggle with Windows 2000 Server, I'll probably have to book a week's vacation. Or hire someone who knows what they're doing. Maybe I should just have done that in the first place, but then I wouldn't have learned all this valuable stuff about how it all works.

For example, after a couple of days, the old server, which is still the main domain controller for the external network, started filling the Event Log with a message every five minutes telling me that there was a domain error. According to Microsoft, the message you usually get is

"Windows cannot access the file gpt.ini for GPO CN={31B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=domainname,DC=com. The file must be present at the location <\\domainname.com\sysvol\domainname.com\Policies\{31B2F340-016D-11D2-945F-00C04FB984 F9}\gpt.ini>. Group Policy processing aborted."

Well, that would be useful. What I got was:

"Windows cannot access the file gpt.ini for {}. The file must be present at the location (). Group Policy processing aborted."

However, after implementing the process described in Event ID 1000, 1001 is logged every five minutes in the Application event log and rebooting, it seems to be fixed. The problem was incorrect permissions on the Winnt\SysVol folder and rights assignment for "Bypass traverse checking". Probably another left-over from the original NT4 installation. Thank heavens for Technet...

And, increasingly, I find I'm struggling for disk space. I need 120GB just to back up the three VMs I'm running, and the servers only have a pair of 160GB disks. If you are ordering hardware to do Hyper-V, buy boxes with four times the space you think you'll need. And make sure you get Gigabit NICs in them and use quality CAT6 cable and a Gigabit network switch 'cos you're going to be spending a lot of time copying very large files...

Ultimately, I took the decision to outsource my Exchange Server to a well-known and reputable company here in the UK. The cost is less than I pay now just for outsouorced email filtering services, so it looks like a bargain. And that meant that I could create a virtual 32-bit Windows 2003 instance (ISA will not install on Win 2008) on Hyper-V to run just ISA 2006 and the external DNS server for my public domains. Less stuff to worry about in the long term I hope, though I'll probably have to upgrade that to Forefront on Win 2008 some time in the future. But at least there's no need now for an external domain! 

Scene III: "DNS = Decidedly Negative Scenario"

Of course, everyone knows that DNS is a black art, and that you should never expect a DNS server to do what you expect. Well, unless you know about this stuff anyway. Up to now, my old DNS setup seemed to be working fine, though probably more through luck and old shoelaces than any real expertise on my part. So I decided this time to read up on how I should do DNS for ISA and an external DNS server to see if I could get it right. And, having got it all set up and running fine on a spare IP address, all seemed hunky-dory.

Until the "big switch-over day" arrived and I pulled the old ISA box out of the network. Everything stopped working. Every machine began to spew its excess event log messages all over the garage floor. My wife was shouting that she couldn't get her email. And it was only 9 o'clock on a Sunday morning. Maybe I should just put the old ISA box back and go back to bed...? However, after calming down and topping up with coffee, I started to investigate. A couple of wrong gateway entries in the domain controller network connections obviously weren't helping, but fixing these didn't cure it. So I went back to the docs to see what I missed the first time round.

The guidance I'd used was Configuring DNS Servers for ISA Server 2004 (there is no ISA 2006 version), which shows the setup for "Domain Member ISA Server computer with full internal resolution". However, the doc is a bit confusing in that it covers several different scenarios. In the end, it was grasping that the ISA box needs to use the internal DNS server and that the internal DNS server will do all forwarding to other DNS servers. These forward lookups go out to the Internet through the ISA server, but do not go to the DNS server on the ISA box. Read "Why can’t I point to the Windows 2000 DNS first, and then to the ISP DNS?" in the "Common Questions" section of that document to understand why. Plus, the internal domain machines must not include the external DNS server in their list of DNS servers, but should instead reference only the internal DNS and allow that to forward lookups (I use DHCP to set these options). Maybe the following more detailed version of the schematic in the Technet doc will help...

Note: If your public DNS server is only answering queries for zones for which it is authoratative (which is most likely the case) make sure you set the Disable Recursion option in the Advanced tab of the Properties dialog for the DNS server. See Can I Plug My Guitar Into My DNS Server? for more details.  

I set the zone TTL for the external DNS server zones to one day, but you may want to increase that if you don't plan moving IP addresses around or updating records very often. Keep the internal TTL at about an hour to cope with DHCP and dynamic address updates. One thing I noticed is that, if you don't specify a DNS server for an interface (i.e. the external network connection), Windows uses the local 127.0.0.1 address "because DNS is installed on this machine". But it doesn't seem to break anything that I've noticed yet...

Scene IV: "Time Passes..."

They say that the show ain't over till the fat lady sings. I sincerely hope she's in the wings tuning up and ready to let rip, because the tidying up after my virtual Yuletide seems to go on and on. Obviously I broke most of the connections and batch files on the network by changing the machine names and IP addresses. But other things about Hyper-V are still catching me out.

For example, I've always used the primary domain controller as a reliable time source for each domain by configuring it to talk to an external time server pool. I even know the NET TIME command line options off by heart. But it all gets silly on Hyper-V because you have multiple servers trying to set the time. The solution, I read, is to get the base O/S to act as a reliable time source, and target the VMs (and other machines if required) to it. You have to use use the more complex syntax of the W32TM command, but it all seemed to work fine until I installed the ISA box. ISA 2006 is clever in that it automatically allows a domain-joined machine to talk to "trusted servers" (which, you'd assume, includes its domain controller). But I had tons of messages saying it couldn't contact or get a valid time through the internal or external connection.

Well, I have to say that I wouldn't expect it to work with the external connection as that is blocked for the ISA box. But why not over the internal connection? Should I just disable the w32time service on the grounds that Hyper-V automatically syncs time for the VMs it hosts (unless you disable this in the Hyper-V Settings dialog for the VM)? Or should I allow external NTP (UDP) access from the ISA box to an external time server? In the end, after some help from other bloggers, I just used NET TIME to remove any registered time servers from the ISA box, restarted the w32time service, and it automatically picked up time from both the "VM IC Time Synchronization Provider" and the domain controller. Perhaps, like me, it just needed a rest before starting again.

Another interesting (?) issue that crawled out of the woodwork after a few days was the error "The Key Distribution Center (KDC) cannot find a suitable certificate to use for smart card logons, or the KDC certificate could not be verified." As I don't use smart cards, I ignored the error until I found the article Event ID 29 — KDC Certificate Availability on Technet. Another example of problems bought on by domain migration from Windows 2003 perhaps. As with several other issues, the solution is less than useful because I get to the bit where it says "...click Request New Certificate and complete the appropriate information in the Certificate Enrollment Wizard for a domain controller certificate", but the Wizard tells me that "certificate types are not available" and I should "contact the Administrator".

Not a lot of use when I am pretending to be the Administrator. Unable to find any other useful guidance, I took a chance and installed the Active Directory Certificate Services role, which created a root certificate in the Personal store and allowed me to create the domain controller certificate I needed. I have no idea if this is the correct approach, but time will no doubt tell...

One thing I would recommend is putting the machine name in big letters on the screen background. I used to get lost just working four machines through a KVM. Now there are multiple machines for some of the KVM buttons. And if you are executing command line options, use the version that contains the machine name as a parameter in case you aren't actually on the machine you think you are...

Finale: "Was It Worth It?"

So, after three weeks, was it actually worth it? I'm not referring to the time you've wasted reading all this administrative junk and doubtful meanderings. I mean, what do I think about the process and the result? Here's my opinions:

• Installing Windows Server 2008 on suitable hardware is a breeze. And it doesn't keep asking for the original disk when you add or remove roles and features.
• Hyper-V is great. On my basic Dell 2.4 GHz Xeon 4-core boxes it runs well, as do the three VMs. Though in reality they aren't under a great deal of load. The biggest bonus for me is a reasonable chance of disaster recovery without (additional) weeks of work.
• Exporting, copying, and managing VMs seems easy and reasonably quick. Note that snapshots are not recommended for VMs running Active Directory. And you'll need lots of additional disk space for this anyway.
• The new servers are green (in environmental and performance terms, not actual color), quiet, and emit very little heat. So I should save on electricity, earplugs, and cabinet cooling fans.
• Moving or migrating stuff from Windows 2003 Server to Windows Server 2008 is not desperately painful, but doing the same from Windows 2000 Server is tough and I nearly gave up (via FDISK) more than once.
• Ultimately, overall, it was well worth the effort. At least until I find the lurking problems that haven't yet raised their ugly heads...

And the good news for any remaining readers of my blog is that I can maybe find something more interesting to ramble on about next week...