Azure OpInsights: Enabling Collection of Azure Diagnostics Data from Storage Account

This post demonstrates how specific diagnostic data in storage accounts of an Azure Cloud Service (consisting of web and worker roles) can be collected by Microsoft Azure Operational Insights (OpInsights) and how Windows events from custom application specific event logs can be collected as well.
     
At current (March 2015), the only way to send diagnostic data of type Windows EventLog and IIS Logs of Microsoft Azure Worker or Web role instances to OpInsights is via the Azure Management Portal.

You can create and add custom Windows Eventlogs to log application specific Windows events in web role or worker role instances of Azure Cloud Service imperatively, and store the events in the Cloud Service’s storage account in the WADWindowsEventLogTable by the Diagnostic Monitor.     
To do this, here are the high-level steps:
– Add a task to create the custom eventlog as a startup task under the ServiceDefinition.csdef file.
Modify the code of the Web Role or Worker Role to update the custom event log at runtime.

As a demo, a custom Windows EventLog – Guestbook Logging -  is created for the Cloud Service. An event is log whenever a button is click on the it website.

To subscribe to the OpInsights service in the Azure Management Portal, click NEW, and create a new workspace or link to an existing OpInsights workspace:
(In this test, a new workspace is created.)

image


When the service is active, clicking the –> arrow will take you to a page with the option to enable OpInsights to query from storage accounts of web and worker roles:

image

image

Select and the storage account of the cloud service of interest and the data types to collect:

image      image

Finally, here is the summary page on the storage account + data type added for OpInsight and the status of the connection:

image


In the OpInsights preview portal, in the corresponding workspace, add the custom eventlog name under EVENT LOGs and check the IIS LOGS box to collect IIS Logs events from the Cloud Service, under the Log Management IP:

image

Within an hour or two, the events from the custom event log (Guestbook Logging) can be seen in OpInsights, as a result of the above configuration:

image

image

Additional info:

Collect Logging Data by Using Azure Diagnostics:
https://msdn.microsoft.com/en-us/library/azure/gg433048.aspx 

Disclaimer:
All information on this blog is provided on an as-is basis with no warranties and for informational purposes only. Use at your own risk. The opinions and views expressed in this blog are those of the author and do not necessarily state or reflect those of my employer.