Using Azure block blob storage to establish master/slave roles


When building an application in Azure one of the common scenarios is the need to establish master/slave roles.   When we have many instances of a worker role which scale horizontally it is often useful for one of these nodes to establish itself as a "master".   This allows it to do various housekeeping tasks while ensuring that its actions aren't in conflict with any other instances.

There is no native way in Azure to denote a worker role as "master".   We can accomplish this however by leveraging the lease capability of Azure block blob storage.   In this way we're using Azure block blob storage as the cloud equivalent of a file share witness.   Each worker role instance attempts to obtain an exclusive lease on a blob.   Only the instance which holds the lease can consider itself "master".   The rest of the instances are all "slaves". 

We start up a new background task using TPL which is constantly trying to obtain or renew a lease.  This controls a Boolean which is at a higher scope denoting the currently running instance as master or not.  Your work continues running in the main thread.

Here is a sample implementation of this as a console application:

//Connection information for storage account
string storageConnectionString
= "DefaultEndpointsProtocol=https;AccountName=XXXXX";
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);

//Default state of not the master
bool isMaster = false;

//Any value between 15 and 60
int leaseLengthInSeconds = 15;


//Dispatch background task which will renew or attempt to obtain the lease every N seconds
Task t = Task.Factory.StartNew(() =>
{

//Create or obtain a reference to a blob called "lease.txt" in the container "locks"
string leaseid = string.Empty;
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = cloudBlobClient.GetContainerReference("locks");
string blobName = "lease.txt";
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);

if (!blob.Exists())
{
blob.UploadFromStream(new MemoryStream());
}


while (true)
{
//No lease so lets get one
if (string.IsNullOrEmpty(leaseid))
{
//Create lease for 15 seconds
TimeSpan? leaseTime = TimeSpan.FromSeconds(leaseLengthInSeconds); //Acquire a lease on the blob.
string proposedLeaseId = Guid.NewGuid().ToString(); //proposed lease id (leave it null for storage service to return you one).

try
{
leaseid = blob.AcquireLease(leaseTime, proposedLeaseId);

//Got a lease so this is the master role
isMaster = true;
}
catch (StorageException ex)
{
//Something happened and we were unable to obtain the lease
isMaster = false;
leaseid = string.Empty;

//For reference you can interrogate the response from Azure blob storage this way if you want more details
string err = ex.RequestInformation.ExtendedErrorInformation.ErrorCode;
if (err == "LeaseAlreadyPresent")
{
//Do something useful?
}
}
}

//Lease already exists so lets try to renew
else
{
try
{
//Specify the lease ID that we're trying to renew and attempt
AccessCondition ac = new AccessCondition();
ac.LeaseId = leaseid;
blob.RenewLease(ac);

//It must have worked or an exception would have been thrown so we're keeping the master role
isMaster = true;
}
catch (StorageException ex)
{
//Failure renewing -- relinquish master role -- again you could interrogate the error code here if you had other logic to apply
isMaster = false;
leaseid = string.Empty;
}
}

//Sleep for lease duration minus two seconds for just in time renewal
Thread.Sleep((leaseLengthInSeconds * 1000)-2000);
}
}, TaskCreationOptions.LongRunning);


while (true)
{

if (isMaster)
{
Console.WriteLine("I'm the boss");
}
else
{
Console.WriteLine("I'm just a slave");
}

Thread.Sleep(1000);
}
}

Comments (1)

  1. Alexandre.Brisebois says:

    Hi Jason,

    A little while back I implemented the leader election pattern ( alexandrebrisebois.wordpress.com/…/preventing-jobs-from-running-simultaneously-on-multiple-role-instances ) and it has come in handy. Let me know what you think of this implementation.

Skip to main content