Scaling Windows Azure Roles using Ruby

While Brian started discussing gathering Windows Azure diagnostics from PHP on Monday, I’m going to jump straight into scaling Windows Azure from Ruby and discuss gathering diagnostic information in a later post. Diagnostic information is great if you want to determine when to scale, but I think first we need to talk about how to accomplish scaling.

Scaling out an application hosted on Windows Azure is pretty easy; it’s just updating a field in a configuration file that tells Windows Azure how many instances to create for the role that hosts your application, and then uploading the new configuration file so it takes effect. The service configuration file is documented at https://msdn.microsoft.com/en-us/library/windowsazure/ee758710.aspx, and the following is an example of what one looks like:

 <?xml version="1.0" encoding="utf-16"?>
 <ServiceConfiguration xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="https://www.w3.org/2001/XMLSchema" serviceName="" osFamily="1" osVersion="*" xmlns="https://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">
 <Role name="HelloRole">
 <ConfigurationSettings>
 <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="DefaultEndpointsProtocol=https;AccountName=storageaccountname;AccountKey=storageaccountkey />
 </ConfigurationSettings>
 <Instances count="1" />
 <Certificates />
 </Role>
 <Role name="ByeRole">
 <ConfigurationSettings>
 <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true" />
 </ConfigurationSettings>
 <Instances count="1" />
 <Certificates />
 </Role>
 </ServiceConfiguration>

See the count attribute on the Instances element? That’s what we want to change to control how many instances Windows Azure creates for a given role. There are a couple things we need to know in order to accomplish this though. We need to know the role name, the hosted service name, and the deployment slot that the role is deployed into. You can get this information through the Windows Azure portal, or using a tool such as waz-cmd (https://github.com/smarx/waz-cmd/blob/master/README.md.) Once you have this information, you need to perform the following steps:

1. Use the Get Deployment operation to return the configuration for the role you want to modify (https://msdn.microsoft.com/en-us/library/windowsazure/ee460804.aspx discusses Get Deployment)
2. Extract the service configuration from the response. It’s base64 encoded, so you have to decode that to obtain the XML file.
3. Find the section for the role you want to modify the instance count for and change the value.
4. Use the Change Deployment Configuration operation to upload the modified configuration (https://msdn.microsoft.com/en-us/library/windowsazure/ee460809.aspx discusses Change Deployment Configuration)
5. At this point, Windows Azure will increase/decrease the number of running instances to match the number you’ve specified.

 You might also want to look at the role status to check that all your roles are in a 'Ready' state after changing the instance count, since it may take a few minutes for Windows Azure to spin up a new role.

Here’s an example of what you might do to accomplish this using Ruby:

 require 'httparty'
require 'openssl'
require 'base64'
require 'nokogiri'

class Hash
  def seek(*_keys_)
    last_level    = self
    sought_value  = nil
 
    _keys_.each_with_index do |_key_, _idx_|
      if last_level.is_a?(Hash) && last_level.has_key?(_key_)
        if _idx_ + 1 == _keys_.length
          sought_value = last_level[_key_]
        else                   
          last_level = last_level[_key_]
        end
      else 
        break
      end
    end
 
    sought_value
  end 
end

class AzureRole
  
  def initialize(params)
    @subscription = params[:subscription]
    @service_name = params[:service_name]
    @role_name = params[:role_name]
    @deployment_slot = params[:deployment_slot]
    @pem_file = File.read(params[:pem_path])
  end
  
  # return the status
  def status
    role_state = query_azure.seek('Deployment','RoleInstanceList','RoleInstance').reject do |entry|
      entry['InstanceStatus']=='Ready'
    end
    if role_state.empty? then
      'Ready'
    else
      'Not Ready'
    end
  end
  
  def instances
    current_config=Nokogiri::XML(Base64.decode64(query_azure["Deployment"]["Configuration"]))
    current_config.xpath(role_xpath).attribute('count').value.to_i
  end
  
  def instances=(other)
    current_config = get_configuration
    # set the new value
    current_config.xpath(role_xpath).attribute('count').value = other.to_s
    new_configuration = Nokogiri::XML::Builder.new(:encoding => 'utf-8') { |xml|
        xml.ChangeConfiguration('xmlns' => 'https://schemas.microsoft.com/windowsazure') {
          xml.Configuration Base64.encode64(current_config.to_xml(:encoding=>'utf-8')).rstrip
        }
      }.to_xml
    # post the new value
     query_azure('post', new_configuration)
  end
  
  private
  
  def role_xpath
    "//xmlns:Role[@name='#{@role_name}']/xmlns:Instances"
  end
  
  # query the azure REST APIs
  def query_azure(verb = 'get', body = '')
    request_url = "https://management.core.windows.net/#{@subscription}/services/hostedservices/#{@service_name}/deploymentslots/#{@deployment_slot}"
    request_options = {:headers => {'x-ms-version' => '2011-08-01',
                                    'Content-Type' => 'application/xml'},
                       :format => :xml,
                       :body => body,
                       :pem => @pem_file
                      }
    request_url = "#{request_url}/?comp=config" if verb == 'post'
    req = verb=='get' ? HTTParty.get(request_url, request_options) : HTTParty.post(request_url, request_options)
    resp = req.parsed_response
    # error or result?
    if (200..299).include?(req.code) then
      resp
    else
      raise "#{resp['Error']['Code']}: #{resp['Error']['Message']}"
    end
  end
end

Here's an explanation of how this works.

Expected Initialization Values

There are a few values that need to be passed in; your subscription ID, the service name. the role name, the deployment name, and the file containing your private key, which is used to authenticate to Windows Azure. The AzureAdmin class expects these parameters as a hash, so creating a new object would look like this: foo=AzureAdmin.new(:subscription=>'subscription guid', :service_name=>'myawesomeservice', :role_name=>'winning', :deployment_slot=>'staging', :pem_path=>'c:\temp\certificate.pem').

You might be asking "what's a pem file?" It's the file that contains the private key for the certificate I'm using to authenticate to Windows Azure. I generated it using the following OpenSSL commands:

 openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem

 openssl x509 -inform pem -in mycert.pem -outform der -out mycert.cer

The first line generates the private key .pem file, which you use to authenticate to Windows Azure REST APIs, and the second generates the .cer file that is uploaded to the Windows Azure portal so that it will recognize you when you authenticate.

query_azure

This method builds the URL, queries the service, and returns the result parsed into a hash.

 def query_azure(verb = 'get', body = '')
    request_url = "https://management.core.windows.net/#{@subscription}/services/hostedservices/#{@service_name}/deploymentslots/#{@deployment_slot}"
    request_options = {:headers => {'x-ms-version' => '2011-08-01',
                                    'Content-Type' => 'application/xml'},
                       :format => :xml,
                       :body => body,
                       :pem => @pem_file
                      }
    request_url = "#{request_url}/?comp=config" if verb == 'post'
    req = verb=='get' ? HTTParty.get(request_url, request_options) : HTTParty.post(request_url, request_options)
    resp = req.parsed_response
    # error or result?
    if (200..299).include?(req.code) then
      resp
    else
      raise "#{resp['Error']['Code']}: #{resp['Error']['Message']}"
    end
  end

status

This method pulls out the status from the role instance(s). If all instances are 'Ready' then it returns ready, otherwise 'Not Ready'. I didn't really want to dip into returning an array containing the status of all possible results for the user to deal with, but that might be something to investigate if you need something that gives you more granular information.

 def status
    role_state = query_azure.seek('Deployment','RoleInstanceList','RoleInstance').reject do |entry|
      entry['InstanceStatus']=='Ready'
    end
    if role_state.empty? then
      'Ready'
    else
      'Not Ready'
    end
  end

Instances and Instances=

Here I'm just returning and setting the instance value. Returning it is trivial, and just pulls in the value from the decoded XML. Setting it is a bit more involved as you have to build the XML response message, base64 encode the config, and pass it to query_azure as a post.

 def instances
    current_config=Nokogiri::XML(Base64.decode64(query_azure["Deployment"]["Configuration"]))
    current_config.xpath(role_xpath).attribute('count').value.to_i
  end
  
  def instances=(other)
    current_config = get_configuration
    # set the new value
    current_config.xpath(role_xpath).attribute('count').value = other.to_s
    new_configuration = Nokogiri::XML::Builder.new(:encoding => 'utf-8') { |xml|
        xml.ChangeConfiguration('xmlns' => 'https://schemas.microsoft.com/windowsazure') {
          xml.Configuration Base64.encode64(current_config.to_xml(:encoding=>'utf-8')).rstrip
        }
      }.to_xml
    # post the new value
     query_azure('post', new_configuration)
  end

Patch the Hash class

This little piece of code patches the Hash class to add a seek method. This makes it easier to deal with the nested structures returned by HTTParty. For example, instead of doing query_azure['Deployment']['RoleInstanceList']['RoleInstance'].... I can instead do query_azure.seek('Deployment','RoleInstanceList','RoleInstance'). Looks a little cleaner. Much thanks to Corey O'Daniel, who's blog I found this on (https://coryodaniel.com/index.php/2009/12/30/ruby-getting-deeply-nested-values-from-a-hash-in-one-line-of-code/).

 class Hash
  def seek(*_keys_)
    last_level    = self
    sought_value  = nil
 
    _keys_.each_with_index do |_key_, _idx_|
      if last_level.is_a?(Hash) && last_level.has_key?(_key_)
        if _idx_ + 1 == _keys_.length
          sought_value = last_level[_key_]
        else                   
          last_level = last_level[_key_]
        end
      else 
        break
      end
    end
 
    sought_value
  end 
end

Typical use

An example of using this would be:

 test=AzureRole.new(:subscription => 'subscription id GUID',
                   :service_name => 'myservice', 
                   :role_name=> 'myrole',
                   :deployment_slot => 'staging', 
                   :pem_path=>'c:\temp\mycert.pem')

 test.instance=test.instance+1

 puts test.status

Summary

As the example code above demonstrates, it's not hard to tell Windows Azure to scale a hosted application; just one entry in a config file. And it's pretty easy to accomplish from a Ruby application using HTTParty for the REST calls, Nokogiri for XML, and OpenSSL for certificates. There are other gems that you can use to accomplish this, and there are probably cleaner ways to accomplish it using Ruby. If you have suggestions on how I might improve this code, or if there's already a better example of how to accomplish this that I've missed, let me know.