Kirk Evans Blog

.NET From a Markup Perspective

Autoscaling Azure–Virtual Machines

This post will demonstrate autoscaling in Azure virtual machines.

image

Background

While I spend most of my time working with PaaS (platform as a service) components of Azure such as Cloud Services and Websites, I frequently need to help customers with solutions that require IaaS (infrastructure as a service) virtual machines.  A topic that comes up very regularly in both of those conversations is how autoscaling works. 

The short version is that you have to pre-provision virtual machines, and autoscale turns them on or off according to the rules you specify.  One of those rules might be queue length, enabling you to build a highly scalable solution that provides cloud elasticity.

Autoscaling Virtual Machines

Let’s look at autoscaling virtual machines.  To autoscale a virtual machine, you need to pre-provision the number of VMs and add them to an availability set.  Using the Azure Management Portal to create the VM, I choose a Windows Server 2012 R2 Datacenter image and provide the name, size, and credentials.

image

The next page allows me to specify the cloud service, region or VNet, storage account, and an availability set.  If I don’t already have an availability set, I can create one.  I already created one called “AVSet”, so I add the new VM to the existing availability set.

image

Finally add the extensions required for your VM and click OK to create the VM.  Make sure to enable the VM Agent, we’ll use that later.

image

You can see that I’ve created 5 virtual machines. 

image

I’ve forgotten to place VM2 in the availability set.  No problem, I can go to it’s configuration and add it to the availability set.

image

This is the benefit of autoscaling and the cloud.  I might use the virtual machine for a stateless web application where it’s unlikely that I need all 5 virtual machines running constantly.  If I were running this on-premises, I would typically just leave them running, consuming resources that I don’t actually utilize (overprovisioned).  I can reduce my cost by running them in the cloud and only utilize the resources that I need when I need them.  Autoscale for virtual machines simply turns some of the VMs on or off depending on rules that I specify.

To show this. let’s configure autoscale for my availability set.  Once VM5 is in the Running state, I go to the Cloud Services tool in the portal and then navigate to my cloud service’s dashboard.  On the dashboard I will see a section for autoscale status:

image

It says that it can save up to 60% by configuring autoscale.  Click the link to configure autoscale.  This is the most typical demo that you’ll see, scaling by CPU.  In this screenshot, I’ve configured autoscale to start at 1 instance.  The target CPU range is between 60% and 80%.  If it exceeds that range, then we’ll scale up 2 more instances and then wait 20 minutes for the next action.  If the target is less than that range, we’ll scale down by 1 instance and wait 20 minutes.

image

Easy enough to understand.  A lesser known but incredibly cool pattern is scaling by queues.  In a previous post, I wrote about Solving a throttling problem with Azure where I used a queue-centric work pattern.  Notice the Scale by Metric option provides Queue as an option:

image

That means we can scale based on how many messages are waiting in the queue.  If the messages are increasing, then our application is not able to process them fast enough, thus we need more capacity.  Once the number of messages levels off, we don’t need the additional capacity, so we can turn the VMs off until they are needed again.

I changed my autoscale settings to use a Service Bus queue, scaling up by 1 instance every 5 minutes and down by 1 instance every 5 minutes.

image

After we let the virtual machines run for awhile, we can see that all but one of them were turned off due to our autoscale rules.

image

Just a Little Code

Virtual machines need something to do, so we’ll create a simple solution that sends and receives messages on a Service Bus queue.  On my laptop, I have an application called “Sender.exe” that sends messages to a queue.  Each virtual machine has an application on it that I’ve written called Receiver.exe that simply receive messages from a queue.  We will have up to 5 receivers working simultaneously as a competing consumer of the queue.

image

The Sender application sends a message to the queue once every second.

Sender
  1. using Microsoft.ServiceBus.Messaging;
  2. using Microsoft.WindowsAzure;
  3. using System;
  4.  
  5. namespace Sender
  6. {
  7.     class Program
  8.     {
  9.         static void Main(string[] args)
  10.         {
  11.             string connectionString =
  12.                 CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
  13.  
  14.             QueueClient client =
  15.                 QueueClient.CreateFromConnectionString(connectionString, "myqueue");
  16.  
  17.             int i = 0;
  18.             while(true)
  19.             {                
  20.                 var message = new BrokeredMessage("Test " + i);
  21.                 client.Send(message);
  22.                 
  23.                 Console.WriteLine("Sent message: {0} {1}",
  24.                     message.MessageId,
  25.                     message.GetBody<string>());
  26.                 
  27.                 //Sleep for 1 second
  28.                 System.Threading.Thread.Sleep(TimeSpan.FromSeconds(1));
  29.                 i++;
  30.             }
  31.             
  32.         }
  33.     }
  34. }

The Receiver application reads messages from the queue once every 3 seconds.  The idea is that the sender will send the messages faster than 1 machine can handle, which will let us observe how autoscale works.

Receiver
  1. using Microsoft.ServiceBus.Messaging;
  2. using Microsoft.WindowsAzure;
  3. using System;
  4. using System.Collections.Generic;
  5. using System.Linq;
  6. using System.Text;
  7.  
  8. namespace Receiver
  9. {
  10.     class Program
  11.     {
  12.         static void Main(string[] args)
  13.         {
  14.             string connectionString =
  15.                 CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
  16.  
  17.             QueueClient client =
  18.                 QueueClient.CreateFromConnectionString(connectionString, "myqueue");
  19.             
  20.  
  21.             while (true)
  22.             {
  23.                 var message = client.Receive();
  24.                 if(null != message)
  25.                 {
  26.                     Console.WriteLine("Received {0} : {1}",
  27.                         message.MessageId,
  28.                         message.GetBody<string>());
  29.                     
  30.                     message.Complete();
  31.                 }
  32.                 
  33.                 //Sleep for 3 seconds
  34.                 System.Threading.Thread.Sleep(TimeSpan.FromSeconds(3));
  35.             }                                        
  36.         }
  37.     }
  38. }

I built the Receiver.exe application in Visual Studio then copied all of the files in the bin/debug folder to the c:\temp folder on each virtual machine. 

image

Running Startup Tasks with Autoscale

As each virtual machine is started, I want the Receiver.exe code to execute upon startup.  I could go into each machine and set a group policy to assign computer startup scripts, but since we are working with Azure, we have the ability to use the custom script extension which will run each time the machine is started.  When I created the virtual machine earlier, I enabled the Azure VM Agent on each virtual machine, so we can use the custom script extension.

We need to upload a PowerShell script to be used as a startup task to execute the Receiver.exe code that is already sitting on the computer.  The code for the script is stupid simple:

Startup.ps1
  1. Set-Location "C:\temp"
  2. .\Receiver.exe

This script is uploaded from my local machine to Azure blob storage as a block blob using the following commands:

Upload block blob
  1. $context = New-AzureStorageContext -StorageAccountName "kirkestorage" -StorageAccountKey "QiCZBIREDACTEDuYcqemWtwhTLlw=="
  2. Set-AzureStorageBlobContent -Blob "startup.ps1" -Container "myscripts" -File "c:\temp\startup.ps1" -Context $context -Force

I then set the custom script extension on each virtual machine.

AzureVMCustomScriptExtension
  1. $vms = Get-AzureVM -ServiceName "kirkeautoscaledemo"
  2. foreach($vm in $vms)
  3. {
  4.     Set-AzureVMCustomScriptExtension -VM $vm -StorageAccountName "kirkestorage" -StorageAccountKey "QiCZBIREDACTEDuYcqemWtwhTLlw=="ContainerName "myscripts"FileName "startup.ps1"
  5.     $vm | Update-AzureVM
  6. }

Once updated, I can see that the Receiver.exe is running on the one running virtual machine:

image

Testing It Out

The next step is to fire up my Sender and start sending messages to it.  The only problem is that I haven’t done a good job in providing any way to see what is going on, how many messages are in the queue.  One simple way to do this is to use the Service Bus Explorer tool, a free download.  Simply enter the connection string for your Service Bus queue and you will be able to connect to see how many messages are in the queue.  I can send a few messages, then stop the sender.  Refresh the queue, and the number of messages decreases once every 3 seconds. 

image

OK, so our queue receiver is working.  Now let’s see if it makes Autoscale work.  I’ll fire up the Sender and let it run for awhile.

image

The number of messages in the queue continues to grow…

image

And after a few minutes one of the virtual machines is automatically started.

image

I check and make sure that Receiver.exe is executing:

image

Waiting for awhile (I lost track of time, guessing 30 minutes or so) you can see that all of the VMs are now running because the number of incoming messages outpaced the ability for our virtual machines to process the messages.

image

Once there are around 650 messages in queue, I turn the queue sender off.  The number of messages starts to drop quickly.  Since we are draining messages out of the queue, we should be able to observe autoscale shutting things down.  About 5 minutes after the number of queue messages drained to zero, I saw the following:

image

Go back to the dashboard for the cloud service, and once autoscale shuts down the remaining virtual machines (all but one, just like we defined) you see the following:

image

Monitoring

I just showed how to execute code when the machine is started, but is there any way to see in the logs when an autoscale operation occurs?  You bet!  Go to the Management Services tool in the management portal:

image

Go to the Operation Logs tab, and take a look at the various ExecuteRoleSetOperation entries. 

image

Click on the details for one.

Operation Log Entry
  1. <SubscriptionOperation xmlns="http://schemas.microsoft.com/windowsazure"
  2.                        xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
  3.   <OperationId>REDACTED</OperationId>
  4.   <OperationObjectId>/REDACTED/services/hostedservices/KirkEAutoscaleDemo/deployments/VM1/Roles/Operations</OperationObjectId>
  5.   <OperationName>ExecuteRoleSetOperation</OperationName>
  6.   <OperationParameters xmlns:d2p1="http://schemas.datacontract.org/2004/07/Microsoft.WindowsAzure.ServiceManagement">
  7.     <OperationParameter>
  8.       <d2p1:Name>subscriptionID</d2p1:Name>
  9.       <d2p1:Value>REDACTED</d2p1:Value>
  10.     </OperationParameter>
  11.     <OperationParameter>
  12.       <d2p1:Name>serviceName</d2p1:Name>
  13.       <d2p1:Value>KirkEAutoscaleDemo</d2p1:Value>
  14.     </OperationParameter>
  15.     <OperationParameter>
  16.       <d2p1:Name>deploymentName</d2p1:Name>
  17.       <d2p1:Value>VM1</d2p1:Value>
  18.     </OperationParameter>
  19.     <OperationParameter>
  20.       <d2p1:Name>roleSetOperation</d2p1:Name>
  21.       <d2p1:Value><?xml version="1.0" encoding="utf-16"?>
  22.         <z:anyType xmlns:i="http://www.w3.org/2001/XMLSchema-instance"
  23.                    xmlns:d1p1="http://schemas.microsoft.com/windowsazure"
  24.                    i:type="d1p1:ShutdownRolesOperation"
  25.                    xmlns:z="http://schemas.microsoft.com/2003/10/Serialization/">
  26.           <d1p1:OperationType>ShutdownRolesOperation</d1p1:OperationType>
  27.           <d1p1:Roles>
  28.             <d1p1:Name>VM3</d1p1:Name>
  29.           </d1p1:Roles>
  30.           <d1p1:PostShutdownAction>StoppedDeallocated</d1p1:PostShutdownAction>
  31.         </z:anyType>
  32.       </d2p1:Value>
  33.     </OperationParameter>
  34.   </OperationParameters>
  35.   <OperationCaller>
  36.     <UsedServiceManagementApi>true</UsedServiceManagementApi>
  37.     <UserEmailAddress>Unknown</UserEmailAddress>
  38.     <ClientIP>REDACTED</ClientIP>
  39.   </OperationCaller>
  40.   <OperationStatus>
  41.     <ID>REDACTED</ID>
  42.     <Status>Succeeded</Status>
  43.     <HttpStatusCode>200</HttpStatusCode>
  44.   </OperationStatus>
  45.   <OperationStartedTime>2015-02-20T20:47:54Z</OperationStartedTime>
  46.   <OperationCompletedTime>2015-02-20T20:48:38Z</OperationCompletedTime>
  47.   <OperationKind>ShutdownRolesOperation</OperationKind>
  48. </SubscriptionOperation>

Notice on line 26 that the operation type is “ShutdownRolesOperation”, and on line 28 the role name is VM3.  That entry occurred after VM3 was automatically shut down.

Summary

This post showed a demonstration of Autoscale in Azure turning virtual machines on and off according to the number of messages in a queue.  This pattern can be hugely valuable to build scalable solutions while taking advantage of the elasticity of cloud resources.  You only pay for what you use, and it’s in your best interest to design solutions to take advantage of that and avoid over-provisioning resources.

For More Information

Solving a throttling problem with Azure

Automating VM Customization tasks using Custom Script Extension

Service Bus Explorer