Experiencing Data Latency and Data Gaps for Many Data Types – 01/13 – Resolved


Final Update: Sunday, 17 January 2016 00:33 UTC

We've confirmed that all systems are back to normal with no customer impact as of 1/16, 22:30 UTC. Our logs show the incident started on 1/13, 12:30 UTC and that during the incident time period , a subset of customers may have experienced latency and data gaps for some part of data.
  • Root Cause: Complete RCA is pending.
  • Lessons Learned: We will continue to work on improving our telemetry.
  • Incident Timeline: 82 Hours - 1/13, 12:30 UTC  through 1/16, 22:30 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Saturday, 16 January 2016 15:42 UTC

The processing component is processing the current data with out latency and chewing the back log data slower than expected due to unexpected issue in processing component however we are catching up. Performing the steps to fix the issue to improve the processing speed of the backlog data. Some customers may still continue to experience data latency.

  • Work Around: None
  • Next Update: Before 01/16 22:00 UTC

-Application Insights Service Delivery Team


Update: Saturday, 16 January 2016 10:14 UTC

The processing component is now working as expected, It is processing the current data with out latency and going through the backlog data faster. Some customers may still continue to experience data latency.

  • Work Around: None
  • Next Update: Before 01/16 16:30 UTC

-Application Insights Service Delivery Team


Update: Saturday, 16 January 2016 04:14 UTC

Backlog Latency has come down and we are closing the gap soon.It should hopefully catch up shortly. Some set of customers may still experience latency when accessing data.

  • Work Around:None
  • Next Update: Before 01/16 10:30 UTC

-Application Insights Service Delivery Team


Update: Friday, 15 January 2016 21:59 UTC

Current data is processing without any latency. The backlog data has improved in terms of processing speed. We are almost catching up and continue to keep monitoring the service.Customers may still experience some sort of latency for the backlog data.

  • Work Around: None
  • Next Update: Before 01/16 04:00 UTC

-Application Insights Service Delivery Team


Update: Friday, 15 January 2016 15:03 UTC

We have found that mitigation was taking longer than expected and have started work for performance improvements in processing component in order to speed up the process of the backlog data. Some customers would have experienced data gaps during the impact time period 1/15 11:40 AM UTC to 1/15 12:20 PM UTC but it will be back-filled and customers may continue to experience data latency however current data is processing without latency.
  • Work Around: None
  • Next Update: Before 01/15 21:30 UTC

-Application Insights Service Delivery Team


Update: Friday, 15 January 2016 08:26 UTC

Performed some steps to improve the processing speed and processing component is now working as expected. Some customers may continue to experience data latency and we estimate 6 hours before all latency is addressed.

  • Work Around:
  • Next Update: Before 01/15 14:30 UTC

-Application Insights Service Delivery Team


Update: Friday, 15 January 2016 01:41 UTC

We are recovering a bit more quicker now. The latency is starting to come down.A Subset of customers may experience data loss for data sent during the time period 1/13 13:50 UTC - 1/13 22:20 UTC and  1/14 8:50 UTC  through 1/14 15:20 UTC. We continue to actively work on this issue and drive it towards resolution.

  • Work Around: None
  • Next Update: Before 01/15 08:00 UTC

-Application Insights Service Delivery Team


Update: Thursday, 14 January 2016 19:37 UTC

We are still on the way to recovery and making some improvements.We scaled up our service to process the latent data.This should help speed up the processing pipeline and help us recover faster.

  • Work Around: None
  • Next Update: Before 01/15 02:00 UTC


-Application Insights Service Delivery Team

Update: Thursday, 14 January 2016 13:19 UTC

Recovery is taking longer than expected as we continue to process data.We are working on adding additional components to back fill the data. Some set of customers would still experience latency when accessing data.

  • Work Around: None
  • Next Update: Before 01/14 19:30 UTC

-Application Insights Service Delivery Team


Update: Thursday, 14 January 2016 07:22 UTC

Recovery is taking longer than expected as we continue to process data.We are working on adding additional components to back fill the data. Some set of customers would still experience latency when accessing data.

  • Work Around: None
  • Next Update: Before 01/14 13:30 UTC

-Application Insights Service Delivery Team


Update: Wednesday, 13 January 2016 23:28 UTC

Recovery is taking longer than expected as we continue to process data.We are working on adding additional components to backfill the data. Some set of customers would still experience latency when accessing data.

  • Work Around: None.
  • Next Update: Before 01/14 06:30 UTC

-Application Insights Service Delivery Team


Update: Wednesday, 13 January 2016 18:56 UTC

Root cause has been isolated to an issue with one of the Azure deployment. We are working with Azure to address this. Some customers may still experience latency and we estimate 3 hours before all latency issues are addressed.
  • Work Around: None
  • Next Update: Before 01/13 23:00 UTC

-Application Insights Service Delivery Team


Initial Update: Wednesday, 13 January 2016 14:21 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency. The following data types are affected: Availability,Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace.
  • Work Around: None
  • Next Update: Before 01/13 18:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Application Insights Service Delivery Team




Skip to main content