Warnings sent to customers when Azure is about to be updated

I sometimes get customers asking me about the warnings they’ll get when updates are rolled out across Azure. Well – at the bottom of this post is an example email sent to me. Notice the emphasis placed on:

• Putting multiple VMs in to availability sets
• Creating multiple instances of each role in Cloud Services

I can’t remember exactly but Igal Figlin from Microsoft did some background research in to this and found that 40% (it might be higher I can’t remember exactly – you can watch the video here) of deployments are not in availability sets. Have a read of the email below and you’ll start to realise how much risk you are putting yourself to if you don’t use multiple VMs in availability sets.

When you put VMs in to availability sets they are also distributed across up to 5 update domains. When Microsoft updates Azure, they’ll walk from one update-domain to the next. You can see what they are saying in this email – they’ll leave 30 minutes between updating each update domain. Let’s say you have 2 machines in an availability set. They’ll be spread across 2 fault domains and 2 update domains. That means if an infrastructure fault occurs (like say power or a network segment), only one of your VMs will be affected. It also means if Microsoft has to do an update, it will take one of your machines out of the configuration at a time.

If you want to be super-cautious, you could protect against the scenario that while Microsoft is walking the update domains in your availability set, you also get an infrastructure failure – that could take out a further machine. The table below shows how.

Imagine the update process had done the update on the instance in Update Domain 0, it had then walked on to Update Domain 1 and was in the middle of updating that instance. Instance 1 is now offline. At the same time a power failure occurs to the rack on Fault Domain 0. That would cause Instance 0 and instance 2 to also be taken offline. You’d now have an availability set with no running machines. You can counter this by adding a VM to the availability set. Because there can only ever be one Update Domain in an availability set undergoing an update – you are protected. Let’s say you are in the middle of updating one of the services yourself. Your update will be stalled, the Microsoft update will complete and then your update will continue. In other words updates are applied to an update domain synchronously. And if you are in the middle of updating one Update Domain, Microsoft won’t start simultaneously updating a different Update Domain. So the following table will remove all risk from simultaneous Update Domain and Fault Domain operations.

The failure of any Fault Domain will take out 2 instances and a simultaneous update can take out only one Update Domain. This means a maximum of 3 instances can be offline because of simultaneous Update Domain/Fault Domain operations. That would leave you with one running instance.

You’d have to be very unlucky to get an infrastructure failure occur while an update is going on. The availability SLA takes the above scenarios in to consideration – you only have to have 2 instances in your availability sets to enjoy the uptime guarantee. If you are unlucky enough to suffer a double problem and the availability drops below the guarantee then Microsoft compensates you.

I made a post about Update Domains and Fault Domains a couple of weeks ago. Interesting stuff if you’re going to take the Azure Infrastructure exam.

Anyway – here’s the email:

---- cut here -----

Have fun – Planky == @plankytronixx