Microsoft Azure log integration – Preview


Both PaaS and IaaS services hosted in Azure generate a large amount of data in security logs. These logs contain vital information that can provide intelligence and powerful insights into policy violations, internal and external threats, regulatory compliance issues, and anomalies in network, host, and user activity.

This ability to get raw logs from your Azure resources into your Security Information and Event Management (SIEM) systems provides a unified dashboard for all your assets, on-premises or in the cloud, so that you can aggregate, correlate, analyze and alert for security events associated with your applications. Azure Log Integration enables you to integrate these logs from assets deployed in Azure to on-premises Security Information Event Management (SIEM) systems . 

High level architecture: 

highlevelarchAzlog

 

What logs can I integrate?

Azure produces extensive logging for every service. These logs are categorized by two main types:

  • Control/Management logs – Control/Management logs give visibility into the create, update and delete operations that goes through Azure Resource Manager. Azure Audit logs contain these logs

Data Plane logs – Data plane logs give visibility into the events raised using the usage of the Azure resource. An example of this are the Windows event system, security and application logs in a Virtual machine

Get Started with Azure Log Integration

Download the package from the Microsoft Download Center  and install Azure Log integration

Note: The Azure Log integration service collects telemetry data from the machine on which it is installed. Please uncheck the option if you would not like to allow Microsoft to collect the telemetry data.

Telemetry data collected –

  • Exception information that happens during execution of Azure log integration
  • Metrics about # of queries made and # of events processed
  • Usage statistics about which Azlog.exe command line option is being used

Integrate Azure VM Logs from your WAD (Windows Azure Diagnostics) Storage accounts

  1. Ensure that your WAD storage account is collecting the logs before continuing on the Azure log integration
  2. Open command prompt, and cd into c:\Program Files\Microsoft Azure Log Integration
  3. Run the command:  azlog source add <FriendlyNameForTheSource> WAD <StorageAccountName> <StorageKey>
    <StorageAccountName> – This is the Azure storage account configured to receive Diagnostics events from your Virtual Machine
     Example: azlog source add azlogtest WAD azlog9414 fxxxFxxxxxxxxywoEJK2xxxxxxxxxixxxJ+xVJx6m/X5SQDYc4Wpjpli9S9Mm+vXS2RVYtp1mes0t9H5cuqXEw==
    Optionally, you can append the subscription ID to the friendly name if you would like the subscription id to show up in the event XML. 
    azlog source add <FriendlyNameForTheSource>.<SubscriptionID> WAD <StorageAccountName> <StorageKey>
  4. To view the events that are pulled from the storage account, Open Event Viewer –>Windows Event log–> Forwarded Events on the Azlog Integrator
  5. Make sure your standard SIEM connector (e.g. Splunk Universal Forwarder or ArcSight Windows Event Smart Collector or QRadar WinCollect) installed on the machine is configured to pick events from forwarded events folder and pipe them to SIEM instance. Review the SIEM specific information to ensure that you are integrating the Azure VM logs.

 

Integrate Azure Audit logs and Azure Security Center Alerts

  1. Open command prompt, and cd into c:\Program Files\Microsoft Azure Log Integration
  2. Run the command:  azlog createazureid  This command will prompt for your Azure Login and creates an Azure Active Directory Service Principal in the Azure AD Tenants that host the Azure subscriptions in which the logged in user is a Co-Administrator or owner. The command will fail if the logged in user is only a Guest user in the Azure AD Tenant.Authentication to azure is done through Azure AD.  Creating a service principal for Azlog Integration will create the Azure AD identity that will be given access to read from Azure subscriptions.
  3. Run the command:  azlog authorize <SubscriptionID>
    The azlog authorize command assigns reader access on the subscription to the service principal created in step # If you don’t specify a SubscriptionID , then the service principal will be assigned the reader role to all subscriptions to which you have any access.(Note: You may see some warnings if you run the authorize command immediately after createazureid. The reason for this is that there is some latency between the Azure Active Directory account creation and the account being available for use. If you wait about 10 seconds after running createazureid and then run authorize, then you should not see these warnings)
  4. Check the following folders to confirm Audit log JSON files exist in them:
    C:\Users\azlog\AzureResourceManagerJson
    C:\Users\azlog\AzureResourceManagerJsonLD
    The tool generates both pretty printed and line delimited JSON.
  5. Check the following folders to confirm that Azure Security Center alerts exist in them:
    C:\Users\azlog\ AzureSecurityCenterJson
    C:\Users\azlog\AzureSecurityCenterJsonLD
  6. Point the standard SIEM file forwarder connector to the appropriate folder to pipe the data to SIEM instance. You may need some field mappings based on SIEM product you are using.
    To learn more about Azure Audit logs and property definitions, please see:
    https://msdn.microsoft.com/library/azure/dn931934.aspx
    https://azure.microsoft.com/en-us/documentation/articles/resource-group-audit/
    To learn about Azure security center alerts, please visit
    https://azure.microsoft.com/en-us/documentation/articles/security-center-managing-and-responding-alerts/

 


Comments (6)

  1. Nitin Pande says:

    Does it integrate logs from Azure Diagnostics to HP Arcsight? can some share procedure for integration configuration?

    1. Hi Nitin – we’re preparing the “how-to” documentation for integration with Arcsight. We expect that doc to be up sometime this week. Thanks! -Tom.

    2. Rick says:

      Got an error while installing the AzureLogIntegration.msi on Windows 7 and Windows 2012 R2, “There is a problem with this Windows Installer Package. A program runs as part of the setup did not finish as expected. Contact your support personnel or package vendor.”

      1. Hi Rick – send an email to AzSIEMteam@microsoft.com and they should be able to help you.

  2. Ronald says:

    When will Azure Log Integration GA?

    1. GA dates are always dependent on service quality – so hard to say at this time. Keep an eye on this blog so that you’ll know when it happens!

Skip to main content