Accommodating Your Batch Window in Windows Azure

  • Article

Certain functions of a system need to run once per day in a typical “batch window” scenario.

You might already be aware that Azure compute is paid by the CPU-hour.  If you’re occupying the CPU, whether or not actively using it, you’re still paying for it.  You occupy the CPU ‘slot’ by having a role deployed, regardless of state - it doesn’t have to be in the ‘Ready’ state.

So how do you get your batch of work done without paying for the slot 24 hours per day?

The answer is found in the Windows Azure Service Management API.  The CreateDeployment API can be used to deploy an app into an existing Hosted Service.  You can do this in various ways including client code, a PowerShell script, or a web or worker role that’s running in Windows Azure.

There are a couple of versions of the Service Management API: managed code and REST.  At the time of this writing, the REST version of the API is the most complete and feature rich.  The managed version doesn’t currently support the ability to CreateDeployment.

For my purposes, I looked at how to use an Azure role to deploy an app.  The PowerShell script already exists and is easy to use – as depicted in one of the Hands on Labs in the Windows Azure Platform Training Kit.  By creating code that runs in a worker role you can then replicate (multi-instance) it in a management pack for your service, thus giving it high availability and fault tolerance.

The sample code that I provide is in a web role.  I used the web page to display diagnostics as I went along.  The project is just a new cloud solution with a web role as provided by Visual Studio 2010.  I’m using the 1.3 version of the Windows Azure SDK, but everything I’ve done here can also be done with the 1.2 version.  In my code, I:

  1. Load the certificate
  2. Initialize storage
  3. (user clicks “test” button)
  4. Get ServiceConfiguration.cscfg from blob storage
  5. Build request and payload
  6. Submit the request
  7. Display the ID of the request as returned from the platform.
  1. The
  2. source code
  3. is posted on CodePlex.

Following are signposts to help you avoid some of the bumps on the road:

Certificate Handling

When you are accessing the Service Management API from the client, you must use a certificate to authenticate the program to the portal.  You must upload the management certificate to the Azure portal and have the original certificate on the client.  The PowerShell script (or windows client) references the local version and matches it to the version that’s in the portal.

When you’re using a web or worker role, the cert has to be loaded into the role as well as the management certificate store so the role can match its copy to the one in the management certificate store.  Hence, you actually have to export the certificate twice – once to the DER version (to a file with .CER extension) as well as export the private key version (.PFX extension).  The source code that I’ve provided has additional details.

Request URI

One of the parameters in the request URI is <service-name>.  The value that you need to use is the one in the ‘DNS Prefix’ field in the management portal.  It’s case sensitive.

Creating the Payload

The web request for CreateDeployment requires a payload.  I tried a couple of ways to create the payload and decided I like using XDocument and XElement (System.XML.Linq) to manage this.  The other way I tried is plain old embedded text.

If you like the plain old text method, there’s a wrinkle to avoid.  Don’t precede your XML with a newline.  I tried this at first for readability, but the request fails with InvalidXMLRequest.

GOOD

payload = string.Format(@"<?xml version=""1.0"" encoding=""utf-8"" standalone=""yes""?>
<CreateDeployment xmlns=""https://schemas.microsoft.com/windowsazure""> <Name>{0}</Name>
…

BAD

payload = string.Format(@"
<?xml version=""1.0"" encoding=""utf-8"" standalone=""yes""?>
<CreateDeployment xmlns=""https://schemas.microsoft.com/windowsazure""> <Name>{0}</Name>
…

If you prefer the System.XML.Linq approach, there is another wrinkle to avoid.

XNamespace xn = "https://schemas.microsoft.com/windowsazure";
XDocument doc = new XDocument(
new XElement(xn + "CreateDeployment",
new XElement(xn + "Name", "gotest-20101231"),
…

After building up the XDocument, you need to get it into a string before transmitting it.  This string must be UTF-8 encoded.  XDocument.Save wants a StringWriter, and StringWriter.ToString() outputs UTF-16.  (If you forget and do this, you’ll get ProtocolError.)  The solution is to subclass StringWriter, adding an encoder that outputs UTF-8.  Again, details are in the provided source.

Referencing REST Resources

In general, REST resource designators are case sensitive in the Windows Azure REST API.  The portal will enforce lower case on some things, but your deployment package might be upper and lower case, so be careful with that.

Multiple Versions

There have been several releases of the API, and older versions are still supported.  Hence, when you build your request, you need to specify x-ms-version in a request header.  Check carefully for the correct version; the value given for CreateDeployment is 2009-10-01 (or later) in the Request Headers section of the documentation.  If you read further you’ll find that it must be 2010-04-01 if you include StartDeployment or TreatWarningsAsError in your payload.

I hope these tips come in handy.  Please comment freely and let me know if there are other aspects of this topic that you’d like to hear more about.

Cheers!

Attribution:

Thanks to Steve Marx for his very valuable assistance on several fronts.
Thanks to Ricardo Villalobos for his assistance with the StringWriter subclass.