Experiencing Data Latency for Many Data Types – 05/27 – Resolved


Final Update: Friday, 27 May 2016 22:16 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 5/27, 22:16 UTC. Our logs show the incident started on 5/27, 13:00 UTC and that during the 9 hours that it took to resolve the issue customers experienced data latency for many data types outside of the 2 hour SLA.
  • Root Cause: The failure was due to multiple issues within a back end service causing processing to be reduced.
  • Lessons Learned: We have collected logs that we will use to further diagnose the issue and work on improvements that will reduce incidents like this in the future.
  • Incident Timeline: 9 Hours & 16 minutes – 5/27, 13:00 UTC through 5/27, 22:16 UTC.

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Randy Rees


Update: Friday, 27 May 2016 16:37 UTC

Root cause has been isolated to an issue with an back end service which was impacting multiple data types. To address this issue we have restarted the back end service and collected telemetry to help diagnose the issue further. Data currently being sent is being processed within our 2 hour SLA. Data that was sent during the affected time period is latent by over our 2 hour SLA. The backlog of this data is actively being processed and we expect the backlog to be fully processed within 6 hours.
  • Next Update: Before 05/27 23:00 UTC

-Randy Rees


Initial Update: Friday, 27 May 2016 13:34 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency. The following data types are affected: Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace.
  • Work Around: None
  • Next Update: Before 05/27 18:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Rama


Skip to main content