How Hyper-V Backup Got Better in 2012 R2 (but now requires a SCSI controller)

Hyper-V has always provided the ability to backup all running virtual machines from the host operating system, without incurring any downtime for the virtual machines being backed up.  To do this it utilizes the fantastic VSS (Volume Shadow Copy Services) functionality in Windows.  When a user wants to backup a Hyper-V server, the following process is followed for Windows Server 2008, 2008 R2 and 2012:

  1. The backup application requests that Hyper-V backup all virtual machines
  2. Hyper-V goes through each virtual machine and sends a request to the Hyper-V Backup integration service inside of the virtual machine
  3. The Hyper-V Backup integration service inside of the virtual machine then asks the guest instance of VSS to create a data-consistent VSS snapshot inside of the virtual machine
  4. Once all the virtual machines have confirmed that they have taken a VSS snapshot, Hyper-V tells the host instance of VSS that it can take a host VSS snapshot
  5. Now, at this stage – some time has elapsed between steps 3 and 4.  This means that the virtual machines were continuing to write data to their virtual hard disks, and that the virtual hard disks host VSS snapshot are not actually data consistent.  To handle this, Hyper-V then:
    1. Locates the copies of the virtual hard disks that are stored in the newly created host side VSS snapshot
    2. Mounts each of these virtual hard disks in the host environment
    3. Uses VSS to revert each virtual hard disks state to the VSS snapshot that was taken in step 3
    4. Dismounts each virtual hard disk
    5. Informs VSS that it is complete – and that it can return the files in the host VSS snapshot to the backup application
This is all fine and good, but it does have its drawbacks.  Particularly, it involves mounting a copy of every virtual hard disk on the system in the host environment.  So, for a computer with 100 virtual machines – each with 2 virtual hard disks – that means 200 virtual hard disks are mounted and dismounted every time a backup is taken.  Needless to say, this can cause interesting problems (like this one, for example).

 

To make the system more reliable, especially at scale, we have made a change to this process.  In Windows 2012 R2 backing up Hyper-V now follows this process:
  1. The backup application requests that Hyper-V backup all virtual machines
  2. Hyper-V goes through each virtual machine and sends a request to the Hyper-V Backup integration service inside of the virtual machine
  3. The Hyper-V Backup integration service inside of the virtual machine then asks the guest instance of VSS to create a data-consistent VSS snapshot inside of the virtual machine
  4. Once all the virtual machines have confirmed that they have taken a VSS snapshot, Hyper-V tells the host instance of VSS that it can take a host VSS snapshot
  5. Now, at this stage – some time has elapsed between steps 3 and 4.  This means that the virtual machines were continuing to write data to their virtual hard disks, and that the virtual hard disks host VSS snapshot are not actually data consistent.  To handle this, Hyper-V then:
    1. Locates the copies of the virtual hard disks that are stored in the newly created host side VSS snapshot
    2. Connects each virtual hard disk copy to the still running virtual machine that it was copied from
    3. Sends a message to the Hyper-V Backup integration service inside the virtual machine that instructs it to revert the virtual hard disk copy to the VSS snapshot that was taken in step 3
    4. Disconnects each virtual hard disk
    5. Informs VSS that it is complete – and that it can return the files in the host VSS snapshot to the backup application
As you can see – the big change here is that the action of reverting the virtual hard disk to the data consistent VSS snapshot now takes place inside the virtual machine instead of in the host operating system.  This has many benefits, one of which being that it scales excellently!  It does, however, have one (minor) drawback.  In order for this method to work – we need to be able to hot add and remove virtual hard disks to and from the virtual machine.  And this is something that is only supported on our SCSI controller (not our IDE controller).

 

Fortunately, we have always created virtual machines with a SCSI controller connected (even if there were no disks attached).  If, however, you have manually removed all SCSI controllers from a virtual machine – Hyper-V backup will now fail on that virtual machine.

 

Cheers,
Ben