Experiencing Data Latency for Many Data Types – 11/19 – Resolved


Final Update: Friday, 20 November 2015 06:57 UTC

We've confirmed that all systems are back to normal with no customer impact as of 11/20, 06:55 UTC. Our logs show the incident started on 11/19, 16:05 UTC and that during the 14 hours 50 Minutes that it took to resolve the issue  small percentage of  customers experienced Data latency for multiple data types.
  • Root Cause: The failure was due to a dependent service which once scaled out helped us mitigate the impact .
  • Lessons Learned: We are continually working on improvements to mitigation and resolution strategies and technology
  • Incident Timeline: 14 Hours & 50 Minutes  - 11/19, 16:05 UTC through 11/20, 06:55 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Friday, 20 November 2015 02:36 UTC

Root cause has been isolated to an update on one of our dependent services which was impacting latency of multiple data types. Multiple data types are now working as expected. Some customers may experience multiple periods of time where latency is outside of 2 hour latency. We have found that mitigation was taking longer than expected and have started work for scaling out a dependent service in order to speed up the process of the backlog data.
  • Work Around: None
  • Next Update: Before 11/20 09:00 UTC

-Application Insights Service Delivery Team


Update: Thursday, 19 November 2015 21:17 UTC

Root cause has been isolated to an update on one of our dependent services which was impacting latency of multiple data types. Multiple data types are now working as expected. Some customers may experience multiple periods of time where latency is outside of 2 hour latency however current data is processing without latency. This data is in a backlog and we have scaled that out to help process the influx of data as quickly as possible.
  • Work Around: None
  • Next Update: Before 11/20 09:30 UTC

-Application Insights Service Delivery Team


Update: Thursday, 19 November 2015 19:15 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience over 2 hours of latency across multiple data types. We have done multiple mitigation steps but have found additional issues that we are currently investigating. We are working to establish the start time for the issue, initial findings indicate that the problem began at 11/17 ~13:00 UTC. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 11/20 01:30 UTC

-Application Insights Service Delivery Team


Initial Update: Thursday, 19 November 2015 16:05 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency. The following data types are affected: multiple data types.

  • Work Around: None
  • Next Update: Before 11/19 20:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Application Insights Service Delivery Team


Skip to main content