Export telemetry from Application Insights
December 11, 2014
At last! Everyone’s been asking for this, so we’re very pleased to announce it.
You can set up a continuous export of your telemetry from the Application Insights portal in JSON format. We put the data into blob storage in your Microsoft Azure account. From there, you can pick it up and write whatever code you need to process it.
For most recent updates about Continuous export go to our documentation page.
Set up continuous export
Open Continuous export from your Application Insights overview blade:
Add a new continuous export. Choose an Azure Storage account and the destination container where you want to put the data:
Once you’ve created your export, it starts going. There’s already telemetry hosing into there. (If you aren’t seeing any data, that’s probably because Application Insights isn’t seeing any coming from your app. You only get data that arrives after you switch on export.)
You can delete the export to stop the stream. Doing so doesn’t delete your data.
Get your telemetry
When you open your blob store with a tool such as CloudXplorer, you’ll see a container with a set of blob files. The URI of each file is application-id/telemetry-type/date/time.
The date and time are the time the telemetry was deposited in the store – not the time it was generated. This means that you can move linearly through the stored data.
If you want to download this data to your own machines, you could use the blob store REST API.
Or you could use Data Factory, in which you can set up pipelines to manage data at scale.
Analyze your stuff
OK, so now you’ve got your data – what’s in there?
What do you get?
The exported data is the raw telemetry we receive from your application, except:
- Web test results aren’t currently included.
- We add location data which we calculate from the client IP address.
Calculated metrics are not included. For example, we don’t export average CPU utilisation, but we do export the raw telemetry from which the average is computed.
What does it look like?
Unformatted JSON. If you want to sit and stare at it, we like Notepad++ with the JSON plug-in, so that you can view it in a nicer layout like this:
How to process it?
On a small scale, you can write some code to pull apart your data, suck it into a spreadsheet, and so on.
If there’s going to be a lot of it, consider HDInsight – Hadoop clusters in the cloud. HDInsight provides a variety of technologies for managing and analyzing big data.
Delete your old data
Please note that you are responsible for managing your storage capacity and deleting the old data if necessary.
Recent updates
- You can now choose which telemetry types to export.
- Support edit export configuration.
What do you think?
We’d love to get your feedback. How can we make this better? How would you like to get your telemetry? Please let us know!
Q & A
-
But all I want is a one-time download of a chart while I’m looking at it.
We’re working on that one separately.
-
Why didn’t you create a REST API to interrogate data in the portal?
Well, you can programmatically get the data from the blob. We figure this solution is very flexible.
-
Can I export straight to my on-premises store?
No, sorry. Our export engine needs to rely on a big open throat to push the data into.
-
Can I export to an Azure SQL database?
No. It’s on the backlog.
-
Is there any limit to the amount of data you put in my blob?
No. We’ll keep pushing data in until you delete your export. We’ll stop if we hit the outer limits for blob storage, but that’s pretty huge. It’s up to you to control how much storage you use.
Great stuff! But why don't you guys store this into Azure DocumentDB? That store is all about JSON documentents and would allow for much easier querying data using SQL-like syntax?
I agree about the DocumentDB question. Will you support that?
Excellent question, we’re considering to support additional storage services moving forward to simplify the consumption of exported data, DocumentDB is one of them.
There’re many advantages to exporting data to blob storage as well, to mention a few:
• Blob storage is mega-scalable and can support very large data volumes.
• It’s quite simple to stream data from blob storage to other destinations.
• It’s also quite simple to use HDInsights to process the data stored in blob storage.
• It will cost our customers less to store exported data in blob storage.
Hope this answers your question.
This is great guys! Is web test results in the works? ETA?
Please consider a way to have Power BI consume the output of Application Insights. Power BI can connect to Azure Blob, but it will not understand the JSON files.
Agree 100% with Marius on this. I'm going to have to say that having AppInsights as a Data Source in Power BI is a MUST HAVE killer-feature.
Gerardo, we're working on making web-test results available for export. It should be available in a few months.
Marius, Omer – integration with Power BI is high priority on our backlog moving forward.
Marius, if you don’t have large data volumes you can use Power Query for Excel to upload JSON documents from blob to Power BI.
support.office.microsoft.com/…/Introduction-to-Microsoft-Power-Query-for-Excel-6e92e2f4-2079-4e1f-bad5-89f6269cd605
Links for Code sample for importing continuous export blobs to SQL DB. Power BI reports can be built on top of that.
azure.microsoft.com/…/app-insights-code-sample-export-telemetry-sql-database
https://sesitai.codeplex.com/
The continous export is not working when language set to Chinese.
In the requests logging the metrics duration is captured. What is the scale for this metrics. Milli seconds, nano seconds or just seconds?
>>Gerardo, we're working on making web-test results available for export. It should be available in a few months.
Any updates?
Can you setup continuous integration using a REST call to ARM or do you have to use the portal?
Hi,
I was wondering if there is any other way to export the Application Insights telemetry.
Continuous Export seems to be a very easy and effective way but is very expensive.
Is there any other way to do it?
Thanks,
Ishani