Windows Azure Diagnostics–From the Ground Up

(The information in this post was created using Visual Studio 2010 and the Windows Azure SDK 1.6 in April of 2012.  All tests were done in the compute emulator.)

I think of Azure diagnostics in two broad categories: the stuff that you get more-or-less for free and the stuff that you log with statements similar to Trace.WriteLine.  There is more – quite a lot actually – but with these two forming the basis the rest should flow more easily by referencing the documentation in MSDN.

Starting with Visual Studio 2010, create a new cloud project and add a worker role.  Delete the Diagnostics plug in from your CSDEF file and the Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString from all CSCFG files (they have to match).  Run the project.  Take a look at the contents of the new GUID/directory/DiagnosticStore directory as depicted below.  Put the lines back in the config files and run it again, examining the DiagnosticStore directory.  Explore the Monitor directory deeply.  You should see some pretty interesting things.  This page gives the details:
https://msdn.microsoft.com/en-us/library/windowsazure/hh411546.aspx

What we’ve just demonstrated (with zero code) is the level of diagnostics information that you get more-or-less for free with the worker role and web role templates.  This MSDN page details what we just did and what it means:
https://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.diagnostics.diagnosticmonitor.aspx

Before:

After:

The information that’s being gathered is in a binary format, so you can’t view it easily.  To do that, you need to set up a periodic transfer of the information to Azure table storage.  This not only converts the information into a readable form, but also puts it into a location that is easy to access.

There is a singleton DiagnosticsMonitor associated with each role (web or worker) in your service.  This object defines how much, how often and what is pushed out to table storage.  There is a factory default configuration embedded into the object that says “push nothing to table storage”.  There are three straightforward ways of changing this.  One of them is to add code to your role.  OnStart is the typical location of this code.  Putting it here affects all role instances.  Here’s an example (#1):  (information logged with this example appears in Azure table WADDiagnosticInfrastructure)

Another way (#2) is to edit a rendition of the default configuration file that is put into blob storage when your instances start up.  There is one of these files for each instance.  If you edit this file, the DiagnosticsMonitor will pick up the change and change its behavior accordingly.  So where are you going to find these files?  In a container called “wad-control-container”.  Which storage account?  The one designated by the Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString setting in your CSCFG file.  Seriously, try it!  Don’t put the code above into OnStart, run your role, then edit the configuration file.  Wait a couple minutes and voila!  Diagnostics information will start dropping into table storage.

This page gives another code sample and some explanation of this:
https://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.diagnostics.diagnosticmonitorconfiguration.aspx

This page talks more about the various methods of altering DiagnosticMonitor’s configuration:
https://msdn.microsoft.com/en-us/library/windowsazure/hh411551.aspx

So much for the more-or-less free stuff. What about sending trace output to the Azure logs?

This is almost as free.  You have to write the Trace.WriteLine statements.  And you have to tinker with the DiagnosticMonitorConfiguration a bit as well, but no more than you did for the previous stuff.  Instead of DiagnosticMonitorConfiguration.DiagnosticInformationLogs as in the example above, set values for the properties of DiagnosticMonitorConfiguration.Logs instead.  You can do this in either of the three ways noted above (I only gave examples of two, but you get the point.)  The information shows up in WADLogsTable. 

An important aspect to diagnostics is Trace Listeners.  Listeners basically hook up information sources (IIS, your Trace.WriteLine code, etc.) with output mechanisms (text file, output window in Visual Studio, etc.)  The web and worker role templates also include a listener for Azure diagnostics: Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener.  This is configured in <system.diagnostics>in web.config or app.config.  If you remove the listener, nothing breaks but you won’t get any output, either.  Try it!  Just remove the listener from app.config, run your project, then take a look in the Resources directory (detailed above).  It’s pretty empty. 

Cost

Logging isn’t free!  It’s using Windows Azure Storage, both blobs and tables.  There are about 43k minutes per month.  If you push trace information once per minute and only use one storage transaction per push, that’s about $0.43 / month.  Now multiply that by the number of things you’re logging.  We are careful about writing multiple records at a time to minimize costs, but we can only do so much.  Since you can configure logging so easily, be judicious about the amount and frequency of logging.

Tools

I used a couple of tools to assist with these tests.  I used Azure Storage Explorer to access my developer storage directly – blobs/tables/queues.  Cerebrata Azure Diagnostics Manager v2 did a really nice job of formatting the diagnostic information that was pushed into table storage.  It allows you to look at it from offline, historical and live sources.

Cheers!

[UPDATE on 4/26/2012] Thanks to Fernando for pointing out an arithmetic error, which has been corrected.