Injecting Simulated Records for the Change Tracking Solution into an OMS Workspace

Article
10/17/2016

In this post, we look at how we can leverage the OMS HTTP Collector API to introduce a Custom Log Type that simulates the record type of the Change Tracking Solution – ConfigurationChange, and injecting sample records for it into an OMS Workspace. With the right set of sample data and the right search queries adapted from the ones used to power the visualization of the Change Tracking Solution, a simulated search and visualization result can be reconstructed in any OMS Workspace for exploratory purposes. However, there are some views and dashboards with rich data visualization and contextual search capabilities used in some OMS Solutions that cannot be recreated with the View Designer at this time. These capabilities are only available and packaged with specific OMS Solutions in order to provide deeper insights pivoted around the problem areas they address.

Important Note:
This simulation method should only be used in test environments for demo or POC purposes as sample data SHOULD NOT be injected into a production environment and be mixed with live production data.

The OMS Change Tracking Solution can be used to identify software and Windows Services changes and Linux daemon changes that occur in your environment and help pinpoint operational issues. Changes to installed software and Windows services on the monitored servers are read and then the data is sent to an OMS Workspace by the OMS Agent for processing. Logic is applied to the received data and the cloud service records the data of type ConfigurationChange. When changes are found, servers with changes are shown in the Change Tracking dashboard. Here is an example of an ConfigurationChange type record:

There are 2 types of fields for this record type:

Type 1 Fields: Common fields:
- SourceSystem
- MG
- id
- Type
- ManagementGroupName
Type 2 Fields: Solution level fields:
- TimeGenerated
- Computer
- SourceComputerId
- ConfigChangeType
- ChangeCategory
- SvcChangeType
- SvcDisplayName
- SvcName
- SvcState
- SvcPreviousState
- SvcStartupType
- SvcAccount
- SvcPath
- SoftwareType
- SoftwareName
- Previous
- Current
- Publisher
- SvcPreviousStartupType
- SvcPreviousPath

Lets name the Custom Log Type that we are creating to simulate the ConfigurationChange, record type as MyConfigurationChange.
Since the values for the Type 1 fields are automatically assigned to ALL records upon ingestion by the OMS Service, the sample records of type MyConfigurationChange will have most of these fields populated as well. Hence only the Type 2 fields and their corresponding sample values are required to be added into the JSON payload for MyConfigurationChange records to be injected into an OMS Workspace of interest.

Here is a sample PowerShell script that converts a CSV file with sample ConfigurationChange type data into JSON data and injecting it into an OMS Workspace with the Log Analytics HTTP Data Collector API in a JSON payload. The sample data will appear as records of type MyConfigurationChange_CL in the targeted OMS Workspace.

This PowerShell script consist of 4 main parts:

Part 1: Specify the key information for authentication, custom log type name and record time stamp.
Note: The MyTimeGenerated field holds the timestamp for the records in the CSV file and is assigned to $TimeStampField variable so that the records injected will have the same create time as their timestamp value in the MyTimeGenerated field once in the OMS Workspace.

 # Replace with your Workspace ID
$CustomerId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" 
 
# Replace with your Primary Key
$SharedKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
 
#Specify the name of the record type that we'll be creating.
$LogType = "MyConfigurationChange"
 
#Specify a time in the format YYYY-MM-DDThh:mm:ssZ to specify a created time for the records.
$TimeStampField = "MyTimeGenerated"

Part 2: Prepare the standard functions to create the authorization signature, and to create and post the request (Copied from the snippets provided in the standard API Documentation):

 # Function to create the authorization signature.
Function Build-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
    $xHeaders = "x-ms-date:" + $date
    $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource

    $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
    $keyBytes = [Convert]::FromBase64String($sharedKey)

    $sha256 = New-Object System.Security.Cryptography.HMACSHA256
    $sha256.Key = $keyBytes
    $calculatedHash = $sha256.ComputeHash($bytesToHash)
    $encodedHash = [Convert]::ToBase64String($calculatedHash)
    $authorization = 'SharedKey {0}:{1}' -f $customerId,$encodedHash
    return $authorization
}

# Function to create and post the request
Function Post-OMSData($customerId, $sharedKey, $body, $logType) 
{
    $method = "POST"
    $contentType = "application/json"
    $resource = "/api/logs"
    $rfc1123date = [DateTime]::UtcNow.ToString("r")
    $contentLength = $body.Length
     $signature = Build-Signature `
        -customerId $customerId `
        -sharedKey $sharedKey `
        -date $rfc1123date `
        -contentLength $contentLength `
        -fileName $fileName `
        -method $method `
        -contentType $contentType `
        -resource $resource
    $uri = "https://" + $customerId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01"

    $headers = @{
        "Authorization" = $signature;
        "Log-Type" = $logType;
        "x-ms-date" = $rfc1123date;
        "time-generated-field" = $TimeStampField;
    }

    $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
    return $response.StatusCode
}

Part 3: Convert CSV data to JSON payload and convert data to correct type.
Note that the HTTP Data Collector API automatically determines the data type of the custom fields for the Custom Log Type based on their values in the JSON payload. For more information on record types and properties, refer to the API Documentation.

 $ConfigurationChangeData = Import-Csv -Path <<Path\SampleConfigurationChangeData.csv>> | select MyTimeGenerated,Computer,SourceComputerId,ConfigChangeType,ChangeCategory,SvcChangeType,SvcDisplayName,SvcName,SvcState,SvcPreviousState,SvcStartupType,SvcAccount,SvcPath,SoftwareType,SoftwareName,Previous,Current,Publisher,SvcPreviousStartupType,SvcPreviousPath

The fields selected for the custom MyConfigurationChange record type in order to simulate Change Tracking Solution records in an OMS Workspace are:
MyTimeGenerated , Computer, SourceComputerId, ConfigChangeType, ChangeCategory, SvcChangeType, SvcDisplayName, SvcName, SvcState, SvcPreviousState, SvcStartupType, SvcAccount, SvcPath, SoftwareType, SoftwareName, Previous, Current, Publisher, SvcPreviousStartupType,SvcPreviousPath

Here is an example of a sample record imported from the CSV file and converted to JSON format to be submitted to the HTTP Data Collector API:

 {
        "MyTimeGenerated": "2016-10-18T20:05:12.07Z",
        "Computer":  "MyComputer.contoso.com",
        "SourceComputerId":  "805ca53d-1a02-be85-e10c-4b4f32f2f2ac",
        "ConfigChangeType":  "WindowsServices",
        "ChangeCategory":  "Modified",
        "SvcChangeType":  "State",
        "SvcDisplayName":  "Microsoft Storage Spaces SMP",
        "SvcName":  "smphost",
        "SvcState":  "Running",
        "SvcPreviousState":  "Stopped",
        "SvcStartupType":  "Manual",
        "SvcAccount":  "NT AUTHORITY\\NetworkService",
        "SvcPath":  "C:\\Windows\\System32\\svchost.exe -k smphost",
        "SoftwareType":  "",
        "SoftwareName":  "",
        "Previous":  "",
        "Current":  "",
        "Publisher":  "",
        "SvcPreviousStartupType":  "",
        "SvcPreviousPath": ""
},

Part 4: Submit the data to the API endpoint.

 ForEach($ConfigurationChangeSingleRecord in $ConfigurationChangeData)
{
  $ConfigurationChangeSingleRecordFixed = ConvertTo-Json -InputObject @($ConfigurationChangeSingleRecord) 
  # Submit the data to the API endpoint
Post-OMSData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($ConfigurationChangeSingleRecordFixed)) -logType $logType
}

So, if Part 1 + Part 2 + Part 3 + Part 4 of the sample PowerShell script returns Status Code of 202 for each record, it means that the JSON Payload has been accepted by the HTTP Data Collector API and the simulated records should appear in the targeted OMS Workspace within a few minutes time.
To list all records of the MyConfigurationChange type that were successfully injected into a targeted workspace, run the Type=MyConfigurationChange_CL search query. The records returned from the search result should appear as follows with the field data types identified accordingly as indicated by their suffix:

With these sample records, we can view all recommendations using the OMS log search, correlate them with other existing record types, or create custom views and dashboards of a simulated Change Tracking Solution. Here is an example of a dashboard created with the View Designer using log search queries adapted from the actual Change Tracking Solution :

Here are the queries for each visualization and navigation component of each View in the dashboard above:

 ##Configuration Changes Summary
#WindowsServices Chart Query
Type=MyConfigurationChange_CL ConfigChangeType_s="WindowsServices" | Measure count() by ConfigChangeType_s
#Software Chart Query
Type=MyConfigurationChange_CL ConfigChangeType_s="Software" | Measure count() by ConfigChangeType_s
#Other Configuration Changes (Files or Daemons) Chart Query
Type=MyConfigurationChange_CL ConfigChangeType_s!="Software" ConfigChangeType_s!="WindowsServices" | Measure count() by ConfigChangeType_s

##Changes by Condiguration Type
#Trend1 Query
Type=MyConfigurationChange_CL (ConfigChangeType_s=WindowsServices AND SvcChangeType_s!=State AND NOT (SvcName_s=BITS AND SvcChangeType_s=StartupType))
#Trend2 Query
Type=MyConfigurationChange_CL (ConfigChangeType_s=Software AND NOT (SoftwareName_s=*KB2461484* OR SoftwareName_s=*KB2267602*))
#List Query
Type=MyConfigurationChange_CL ((ConfigChangeType_s=WindowsServices AND SvcChangeType_s!=State AND NOT (SvcName_s=BITS AND SvcChangeType_s=StartupType)) OR (ConfigChangeType_s=Software AND NOT (SoftwareName_s=*KB2461484* OR SoftwareName_s=*KB2267602*)) OR (ConfigChangeType_s=Daemons AND (SvcChangeType_s=StartupType OR SvcChangeType_s=Path OR SvcChangeType_s=Runlevels)) OR ConfigChangeType_s=Files) | measure count() as Count by ConfigChangeType_s
#Navigation Query
Type=MyConfigurationChange_CL ((ConfigChangeType_s=WindowsServices AND SvcChangeType_s!=State AND NOT (SvcName_s=BITS AND SvcChangeType_s=StartupType)) OR (ConfigChangeType_s=Software AND NOT (SoftwareName_s=*KB2461484* OR SoftwareName_s=*KB2267602*)) OR (ConfigChangeType_s=Daemons AND (SvcChangeType_s=StartupType OR SvcChangeType_s=Path OR SvcChangeType_s=Runlevels)) OR ConfigChangeType_s=Files) | measure count() as Count by ConfigChangeType_s

##Total Application and Package Changes
#First Number Query
Type=MyConfigurationChange_CL ConfigChangeType_s="Software" SoftwareType_s="Application" | Measure count() by SoftwareName_s
#Second Number Query
Type=MyConfigurationChange_CL ConfigChangeType_s="Software" SoftwareType_s="Package" | Measure count() by SoftwareName_s
#List Query
Type=MyConfigurationChange_CL ConfigChangeType_s="Software" (SoftwareType_s="Package" OR SoftwareType_s="Application") | Measure count() by SoftwareType_s"

##Software, Windows Services, LINUX DAEMON Changes
#Donut Query
Type=MyConfigurationChange_CL (ConfigChangeType_s=Software AND NOT (SoftwareName_s=*KB2461484* OR SoftwareName_s=*KB2267602*)) | measure count() by ChangeCategory_s
Type=MyConfigurationChange_CL ConfigChangeType_s=WindowsServices SvcChangeType_s!=State And Not (SvcName_s=BITS And SvcChangeType_s=StartupType) | measure count() by SvcChangeType_s
Type=MyConfigurationChange_CL (ConfigChangeType_s=Daemons AND (SvcChangeType_s=StartupType OR SvcChangeType_s=Path OR SvcChangeType_s=Runlevels)) | measure count() by SvcChangeType_s
#List Query
Type=MyConfigurationChange_CL (ConfigChangeType_s=Software AND NOT (SoftwareName_s=*KB2461484* OR SoftwareName_s=*KB2267602*)) | measure count() as Count by Computer | top 500000
Type=MyConfigurationChange_CL ConfigChangeType_s=WindowsServices SvcChangeType_s!=State And Not (SvcName_s=BITS And SvcChangeType_s=StartupType) | measure count() as Count by Computer | top 500000
Type=MyConfigurationChange_CL (ConfigChangeType_s=Daemons AND (SvcChangeType_s=StartupType OR SvcChangeType_s=Path OR SvcChangeType_s=Runlevels)) | measure count() as Count by Computer | top 500000
#Navigation Query
Type=MyConfigurationChange_CL (ConfigChangeType_s=Software AND NOT (SoftwareName_s=*KB2461484* OR SoftwareName_s=*KB2267602*)) | measure count() as Count by Computer
Type=MyConfigurationChange_CL ConfigChangeType_s=WindowsServices SvcChangeType_s!=State And Not (SvcName_s=BITS And SvcChangeType_s=StartupType) | measure count() as Count by Computer 
Type=MyConfigurationChange_CL (ConfigChangeType_s=Daemons AND (SvcChangeType_s=StartupType OR SvcChangeType_s=Path OR SvcChangeType_s=Runlevels))

**Cleaning Up:** To clean up the **Custom Fields** of this Custom Log Type from an OMS Workspace, go to **Settings –\> Data –\> Custom Fields –\> Manage custom fields**, and click **Remove** to remove them: ![image](https://msdntnarchive.blob.core.windows.net/media/2016/11/image84.png "image")

A shout out and word of thanks to my colleague Ben Shy (OMS PM) for helping me figure out how to allow the $TimeStampField variable to specify a timestamp field to use from a record’s data.

Additional Resources:

Log Analytics HTTP Data Collector API by Brian Wren
https://azure.microsoft.com/en-us/documentation/articles/log-analytics-data-collector-api/

HTTP Data Collector API: Send us data from space… or anywhere! by Evan Hissey
https://blogs.technet.microsoft.com/msoms/2016/08/30/http-data-collector-api-send-us-data-from-space-or-anywhere/

Change Tracking solution in Log Analytics by Bill Anderson
https://azure.microsoft.com/en-us/documentation/articles/log-analytics-change-tracking/

Disclaimer:
All information on this blog is provided on an as-is basis with no warranties and for informational purposes only. Use at your own risk. The opinions and views expressed in this blog are those of the author and do not necessarily state or reflect those of my employer.

Injecting Simulated Records for the Change Tracking Solution into an OMS Workspace

Additional resources