Using Azure block blob storage to establish master/slave roles

When building an application in Azure one of the common scenarios is the need to establish master/slave roles.   When we have many instances of a worker role which scale horizontally it is often useful for one of these nodes to establish itself as a "master".   This allows it to do various housekeeping tasks while ensuring that its actions aren't in conflict with any other instances.

There is no native way in Azure to denote a worker role as "master".   We can accomplish this however by leveraging the lease capability of Azure block blob storage.   In this way we're using Azure block blob storage as the cloud equivalent of a file share witness.   Each worker role instance attempts to obtain an exclusive lease on a blob.   Only the instance which holds the lease can consider itself "master".   The rest of the instances are all "slaves". 

We start up a new background task using TPL which is constantly trying to obtain or renew a lease.  This controls a Boolean which is at a higher scope denoting the currently running instance as master or not.  Your work continues running in the main thread.

Here is a sample implementation of this as a console application:

//Connection information for storage account string storageConnectionString ``= "DefaultEndpointsProtocol=https;AccountName=XXXXX"; CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString); //Default state of not the master bool isMaster = false; //Any value between 15 and 60 int leaseLengthInSeconds = 15; //Dispatch background task which will renew or attempt to obtain the lease every N seconds Task t = Task.Factory.StartNew(() => { //Create or obtain a reference to a blob called "lease.txt" in the container "locks" string leaseid = string.Empty; CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient(); CloudBlobContainer container = cloudBlobClient.GetContainerReference("locks"); string blobName = "lease.txt"; CloudBlockBlob blob = container.GetBlockBlobReference(blobName); if (!blob.Exists()) { blob.UploadFromStream(new MemoryStream()); } while (true) { //No lease so lets get one if (string.IsNullOrEmpty(leaseid)) { //Create lease for 15 seconds TimeSpan? leaseTime = TimeSpan.FromSeconds(leaseLengthInSeconds); //Acquire a lease on the blob. string proposedLeaseId = Guid.NewGuid().ToString(); //proposed lease id (leave it null for storage service to return you one). try { leaseid = blob.AcquireLease(leaseTime, proposedLeaseId); //Got a lease so this is the master role isMaster = true; } catch (StorageException ex) { //Something happened and we were unable to obtain the lease isMaster = false; leaseid = string.Empty; //For reference you can interrogate the response from Azure blob storage this way if you want more details string err = ex.RequestInformation.ExtendedErrorInformation.ErrorCode; if (err == "LeaseAlreadyPresent") { //Do something useful? } } } //Lease already exists so lets try to renew else { try { //Specify the lease ID that we're trying to renew and attempt AccessCondition ac = new AccessCondition(); ac.LeaseId = leaseid; blob.RenewLease(ac); //It must have worked or an exception would have been thrown so we're keeping the master role isMaster = true; } catch (StorageException ex) { //Failure renewing -- relinquish master role -- again you could interrogate the error code here if you had other logic to apply isMaster = false; leaseid = string.Empty; } } //Sleep for lease duration minus two seconds for just in time renewal Thread.Sleep((leaseLengthInSeconds * 1000)-2000); } }, TaskCreationOptions.LongRunning); while (true) { if (isMaster) { Console.WriteLine("I'm the boss"); } else { Console.WriteLine("I'm just a slave"); } Thread.Sleep(1000); } }