Recently I had a customer hit an issue that was hard to resolve.....until we stopped looking at the data and reconsidered our design.
What We Had
- Virtual DC (Hyper-V guest running in Azure, but the location and virtualization doesn't matter!)
- C$ OS, E$ Sysvol/NTDS
- Bitlocker enabled for all drives
The customer installed a relatively small and innocent piece of software, rebooted, and then we entered the BSOD loop -- hard to see since it was on a guest in Azure. After working through the night on the Azure aspect and out of ideas, we asked an AD guru to take a look. Within minutes he had figured it out!
So what was going on?
- When the VM boots up, it tries to unencrypts the OS drive first. This OS disk key is stored in AD (accessible on another DC) or in 3rd party tool, like CloudLink
- Once the OS is unencrypted, Bitlocker frantically tries to unlock any data drives.....
- Meanwhile AD services are starting up (among the first services), but they can't get to the AD database (it's sitting on that locked E$.....)
- AD determines it can't get to it's database, crashes, throws the BSOD (with a nice pause so you can frantically try to write down the error message), and then reboots the server.
- The cycle repeats....
Basically our DC is so secure, even we can't use it! Luckily a DC is easily rebuilt and we all know better than to only have a single DC in the domain, right?
If you want to Bitlocker your DCs, put all those critical DC bits on the C$!
A big thanks to John Bay, our AD guru!