Experiencing Data Latency for Many Data Types – 04/20 – Resolved


Final Update: Thursday, 21 April 2016 04:27 UTC

We've confirmed that all systems are back to normal with no customer impact as of 4/21, 02:10 UTC. Our logs show the incident started on 4/20, 15:00 UTC and that during the 10 hours that it took to resolve the issue customers saw latency for windows of time that was outside of SLA.
  • Root Cause: The failure was due to a dependency service experiencing issues in their backend which has now been resolved.
  • Lessons Learned: We have worked with the dependency service to mitigate the issue as well as we are working on an upgrade that will improve our resiliency in the future.
  • Incident Timeline: 10 Hours & 15 minutes - 4/20, 15:00 UTC through 4/21, 02:10 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Randy


Update: Wednesday, 20 April 2016 22:19 UTC

Root cause has been isolated to slow processing in one of our back end services which was impacting multiple data types. All data is currently within our 2 hour SLA, however, we are continuing with high priority to do mitigation steps to improve our resiliency in this area.
  • Work Around: none
  • Next Update: Before 04/21 10:30 UTC

-Randy


Update: Wednesday, 20 April 2016 16:04 UTC

Root cause has been isolated to slow processing in one of our back end services which is impacting multiple data types for a subset of customers. To address this issue we rebooted the affected services. A subset of customers will experience a window of data that is latent above the 2 hours SLA. Current data is not latent and within our 2 hour SLA
  • Work Around: None
  • Next Update: Before 04/20 22:30 UTC

-Randy


Skip to main content