Azure@home Part 11: Worker Role Run Method (concluded)
This post is part of a series diving into the implementation of the @home With Windows Azure project, which formed the basis of a webcast series by Developer Evangelists Brian Hitney and Jim O’Neil. Be sure to read the introductory post for the context of this and subsequent articles in the series.
Well, it looks like eleven may be the magic number for this blog series. It’s been a couple of weeks, so to review: my last post focused on most of the processing inside of the Run method of the Azure@home WorkerRole – primarily the LaunchFoldingClientProcess implementation, which is responsible for steps 4 through 8 of the architecture diagram to the right. Steps 7 and 8 were touched upon, but not fully explored, and that’s the topic of this post, namely looking at how progress on a given Folding@home simulation is reported:
- to the local Azure table named workunit, and
- to the ‘overseer application,’ distributed.cloudapp.net, which keeps track of each of the individual Azure@home deployments.
The snippet of code from LaunchFoldingClientProcess we’re focused on is recreated below, and it’s specifically the pair of invocations at Lines 46-47 and Lines 61-62 that do the reporting. The first set handles reporting on a configurable interval of time (pollingInterval – by default every 15 minutes), and the second set handles the final report for a work unit immediately after its associated Folding@home client process has successfully completed.
38: while (!exeProcess.HasExited)
39: {
40: // get current status
41: FoldingClientStatus status = ReadStatusFile();
42:
43: // update local status table (workunit table in Azure storage)
44: if (!status.HasParseError)
45: {
46: UpdateLocalStatus(status);
47: UpdateServerStatus(status, clientInfo);
48: }
49:
50: Thread.Sleep(TimeSpan.FromMinutes(pollingInterval));
51: }
52:
53: // when work unit completes successfully
54: if (exeProcess.ExitCode == 0)
55: {
56: // make last update for completed role
57: FoldingClientStatus status = ReadStatusFile();
58:
59: if (!status.HasParseError)
60: {
61: UpdateLocalStatus(status);
62: UpdateServerStatus(status, clientInfo);
63: }
64:
65: // re-poll table (if empty, this provide means to exit loop)
66: clientInfo = GetFoldingClientData();
67: }
UpdateLocalStatus
UpdateLocalStatus has an implementation that should appear fairly familiar if you’ve followed along with this blog series. Its role is to update the workunit table for each distinct deployment of Azure@home thus providing the status information for the WebRole’s status.aspx page (cf. Part 4 of this series).
1: internal static void UpdateLocalStatus(FoldingClientStatus statusInfo)
2: {
3: var cloudStorageAccount =
4: CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
5:
6: // ensure workunit table exists
7: var cloudClient = new CloudTableClient(
8: cloudStorageAccount.TableEndpoint.ToString(),
9: cloudStorageAccount.Credentials);
10: cloudClient.CreateTableIfNotExist("workunit");
11:
12: // select info for given workunit
13: var ctx = new ClientDataContext(
14: cloudStorageAccount.TableEndpoint.ToString(),
15: cloudStorageAccount.Credentials);
16: var workUnit = (from w in ctx.WorkUnits
17: where w.PartitionKey == RoleEnvironment.CurrentRoleInstance.Id &&
18: w.RowKey ==
w.MakeKey(statusInfo.Name, statusInfo.Tag, statusInfo.DownloadTime)
19: select w).FirstOrDefault<WorkUnit>();
20:
21: // if it's a new one, add it
22: if (workUnit == null)
23: {
24: workUnit = new WorkUnit(statusInfo.Name, statusInfo.Tag,
statusInfo.DownloadTime, RoleEnvironment.CurrentRoleInstance.Id)
25: {Progress = statusInfo.Progress, StartTime = DateTime.UtcNow };
26: ctx.AddObject("workunit", workUnit);
27: }
28:
29: // otherwise, update it
30: else
31: {
32: workUnit.Progress = statusInfo.Progress;
33: if (workUnit.Progress == 100)
34: workUnit.CompleteTime = DateTime.UtcNow;
35: ctx.UpdateObject(workUnit);
36: }
37: ctx.SaveChanges();
38: }
Passed into this method (and into UpdateServerStatus as well) is a simple class – FoldingClientStatus – that encompasses the information extracted from the unitinfo.txt file (Step 6 in the architecture diagram above).
The method ReadStatusFile in FoldingClientCore.cs contains the code that parses unitinfo.txt (we didn’t cover its implementation explicitly in this series). This Folding@home file format isn’t documented, so as a bit of defensive programming, the HasParseError flag was added to the status class. When a parsing error is detected, a message is written (via Azure Diagnostics) to the Azure log, and a default value is provided for the element that failed parsing. Granted, that could result in some inconsistent entries in the workunit table .
Per the pattern for this series, let’s dissect the code above section by section.
Lines 3-4: get the Windows Azure Storage account information from the service configuration file.
Lines 6-10: check for existence of workunit table, and if it’s not there, create it.
Lines 13-15: create a data context to issue storage requests against the table. Recall that the TableServiceContext, as well as the WorkUnit and ClientInformation entity classes, are defined in the AzureAtHomeEntities assembly that’s part of the Azure@home project.
Lines 16-19: select the single entity (if it exists) from the workunit table that corresponds to the current Folding@home instance being run by the WorkerRole. Note, because both PartitionKey and RowKey are part of the filter, at most a single entity will be returned, and therefore continuation token logic (that we covered in Parts 6 and 7 of this series) isn’t necessary.
Lines 22-27: are executed when there isn’t yet a progress record for the given work unit in the workunit table. A new entity is created based on the status information passed into the method, and then that entity is added to the context.
Lines 31-36: are executed when the progress record exists and needs to be updated with the percentage of completion (which is parsed from the unitinfo.txt file). Once the work unit is complete, the CompleteTime is also updated (Line 34) and used as a flag (in status.aspx of the WebRole) to differentiate completed work units from in-progress ones.
Line 37: commits the change to the context – either creating a new entity or updating an existing one – to the workunit table, via a REST call of course!
UpdateServerStatus
This method passes on the progress information from every WorkerRole in the deployed Azure@home project back to a single endpoint – a web service hosted by distributed.cloudapp.net. The information is tallied by the distributed.cloudapp.net project (which is out of the scope of this series) to provide a number of statistics on the impact of Azure@home on the overall Folding@home effort (see snapshots below).
 |
 |
 |
The endpoint in distributed.cloudapp.net is a simple ASMX web service called ping that accepts a straightforward name/value pair payload including the client data (name, latitude, longitude, etc.) and the latest FoldingClientStatus data (progress, work unit name, work unit tag, etc.). The code that sets up the web service call appears below:
1: internal static void UpdateServerStatus(FoldingClientStatus status,
ClientInformation clientInfo)
2: {
3:
4: string url = string.Format("{0}/ping",
5: RoleEnvironment.GetConfigurationSettingValue("AzureAtHome_PingServer"));
6:
7: string data = string.Format("username={0}&passkey={1}&instanceid={2}&lat={3}&long={4}
&foldingname={5}&foldingtag={6}&foldingprogress={7}
&deploymentid={8}&servername={9}&downloadtime={10}",
8: HttpUtility.UrlEncode(clientInfo.UserName),
9: clientInfo.PassKey,
10: RoleEnvironment.CurrentRoleInstance.Id,
11: clientInfo.Latitude,
12: clientInfo.Longitude,
13: HttpUtility.UrlEncode(status.Name),
14: HttpUtility.UrlEncode(status.Tag),
15: status.Progress,
16: RoleEnvironment.DeploymentId,
17: clientInfo.ServerName,
18: HttpUtility.UrlEncode(status.DownloadTime)
19: );
20:
21: try
22: {
23: System.Net.WebRequest req = System.Net.WebRequest.Create(url);
24: req.ContentType = "application/x-www-form-urlencoded";
25: req.Method = "POST";
26:
27: byte[] bytes = System.Text.Encoding.ASCII.GetBytes(data);
28: req.ContentLength = bytes.Length;
29:
30: using (System.IO.Stream stream = req.GetRequestStream())
31: {
32: stream.Write(bytes, 0, bytes.Length);
33: stream.Close();
34: }
35: }
36: catch (Exception ex)
37: {
38: Trace.TraceWarning(String.Format("Error sending ping: {0} | {1} | {2}",
ex.Message, url, ex.StackTrace));
39: }
40: }
Here each WorkerRole is essentially taking on the role of a client to a web service that also happens to be running in Windows Azure, but there’s really nothing about this WebRequest implementation that’s specific to the cloud.
Lines 4-5: set up the web service URL based on a configuration parameter in the ServiceConfiguration.cscfg file,
Lines 7-19: set up the payload for the HTTP POST request that will be issued,
Lines 23-28: set up the HTTP request, and
Lines 30-34: issue the request.
This call is a ‘fire-and-forget’ request, so there’s no attempt (or need) to process any type of response. The exception handling code in Lines 36ff does, however, emit an entry to the Azure log file if there was some issue in issuing the web request.
Here’s an example of one of the web service requests (issued from the local development fabric) captured via Fiddler:
POST https://distributed.cloudapp.net/ping HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: distributed.cloudapp.net
Content-Length: 257
username=Jack&passkey=&instanceid=deployment(499).AzureAtHome.WorkerRole.0&lat=42.8115&
long=-76.2891&foldingname=Azure&foldingtag=-&foldingprogress=0&
deploymentid=deployment(499)&servername=azuresyracuse.cloudapp.net&
downloadtime=November+10+21%3a18%3a31
Final(?) Words
Whether you’ve made it through the entire series front-to-back or just popped in on an article of interest or two, I hope the relatively deep coverage of the Azure@home project and relevant Azure topics has been helpful. While I honestly can’t think of anything significant that I’ve left untouched in this series, I’m certainly open to feedback if there’s some aspect of the project that you think deserves more attention in a blog post.
Given the recent announcements at PDC, the gears are already turning in my head in terms of how I might update this project to take advantage of some of the new features – like administrative access, the VM Role, and AppFabric caching. So don’t be surprised to see the series resurrected to explore these new aspects of the Windows Azure Platform!