OMS Data Collector API and Twitter Sentiment Demo (Part 1)

With OMS Data Collector API you can now input any data into your OMS workspace, and not just computer data. The API allows programmatically insertion of json documents into OMS Log Analytics, so you can search, correlate, visualize and analyse it. In this tutorial, you’ll learn how I did my twitter demo for Ignite NZ 2016. Where I collected all tweets with #MSIgniteNZ, and filtered by session code to build a dashboard in OMS.

There are 2 parts to this blog, this being the first part:

Part 1 – Create OMS Data Collector Azure App API Part 2 – Create Azure Logic App

What you’ll need:
• Visual Studio (I used Visual Studio 2015 with Update 1) and ;
• Azure SDK (I used Azure SDK 2.8.2, which you can install with the “Web Platform Installer”)
• A twitter account
• An Azure Subscription
• OMS workspace ID and key;
• Cognitive Services Text Analytics subscription. You can sign up for a free one of up to 5000 transactions per month here: https://www.microsoft.com/cognitive-services/en-us/text-analytics-api
https://azure.microsoft.com/en-us/documentation/articles/cognitive-services-text-analytics-quick-start/

Part 1: Build your API

Create a new Project in Visual Studio, and select “ASP.NET Web Application”, (bonus points for also creating an Application Insights to monitor your API)

Choose  ‘Azure API App’, as your template:

I enabled Swagger (https://swagger.io/) so I could test my API before publishing it. In the project go to App_Start, SwaggerConfig class and enable the UI by uncommenting this:

}) .EnableSwaggerUi(c => {

The new project will have ValuesController under Controllers folder, rename the controller to OMSAPIController, make sure you change the controller name and the file name too to OMSAPIController.cs. Here is the code for my controller, you’ll need to update 2 values: customerId and sharedKey using System; using System.Collections.Generic; using System.Linq; using System.Net; using System.Net.Http; using System.Web.Http; using System.Security.Cryptography; using System.Web.Script.Serialization; using System.Runtime.Serialization; using System.Text; namespace OMSAPI.Controllers { public class OMSAPIController : ApiController { static string customerId = ""; //For shared key use either primary or secondary Connected Sources client authentication key static string sharedKey = ""; //LogName is name of the event type that is being submitted to Operational Insights static string LogName = "TwitterLogs";//Optional field used to specify time stamp from the data. If time field not specified, assumes message ingestion time static string TimeStampField = "";public class tweet { public string TwitterUser { get; set; } public string TwitterText { get; set; } public Sentiment Sentiment { get; set; } } public class Sentiment { public Document[] documents { get; set; } public object[] errors { get; set; } } public class Document { public float score { get; set; } public string id { get; set; } } // POST api/values public void Post([FromBody]tweet jsonInput) {//call function to determine Sentiment based on score string SentimentName = GetSentiment(jsonInput.Sentiment.documents[0].score);//build json out of the data for injection in OMS string json= "{\"TwitterUser\":\"" + jsonInput.TwitterUser + "\",\"TwitterText\": \"" + jsonInput.TwitterText + "\",\"Sentiment\": \"" + SentimentName + "\"}"; var datestring = DateTime.UtcNow.ToString("r"); string stringToHash = "POST\n" + json.Length + "\napplication/json\n" + "x-ms-date:" + datestring + "\n/api/logs"; //build signture for encryption string hashedString = BuildSignature(stringToHash, sharedKey); string signature = "SharedKey " + customerId + ":" + hashedString;PostOMSData(signature, datestring, json); }//function to build signature for encryption public static string BuildSignature(string message, string secret){ var encoding = new System.Text.ASCIIEncoding(); byte[] keyByte = Convert.FromBase64String(secret); byte[] messageBytes = encoding.GetBytes(message); using (var hmacsha256 = new HMACSHA256(keyByte)) { byte[] hash = hmacsha256.ComputeHash(messageBytes); return Convert.ToBase64String(hash); } }//determining sentiment as a string based on float score public static string GetSentiment(float SentimentScore) { string SentimentString; if (SentimentScore < 0.35) { SentimentString = "Negative"; } else if (SentimentScore > 0.65) { SentimentString = "Positive"; } else { SentimentString = "Neutral"; }return SentimentString;

}
//posting json into OMS using OMS ingestion API public static void PostOMSData(string signature, string date, string json) { string url = "https://" + customerId + ".ods.opinsights.azure.com/api/logs?api-version=2016-04-01"; using (var client = new WebClient()) { client.Headers.Add(HttpRequestHeader.ContentType, "application/json"); client.Headers.Add("Log-Type", LogName); client.Headers.Add("Authorization", signature); client.Headers.Add("x-ms-date", date); client.Headers.Add("time-generated-field", TimeStampField); client.UploadString(new Uri(url), "POST", json); } } }

}

Now test your application, click on Run, that will launch your internet browser with your application: https://localhost:[port]. Add “/swagger” to your URL, it will look like something like this: https://localhost:1486/swagger/

Click on “Expand Operations” and you’ll see your application like this:

The model on the side explains the json format expected, you can copy and paste the text below to test your API, click on “Try it out!”
{ "Sentiment": { "documents": [ { "score": 0.99, "id": "1" } ], "errors": [] }, "TwitterText": "Test twitter text", "TwitterUser": "test user" }div>

You should get a 204 response code. Also, check in your OMS workspace, that the data arrived.

Once you’re happy with your API, you can publish it to Azure.
Right-click on your solution and publish it!

Next blog post we will put it all together using a Logic App.