Experiencing Data Latency for Many Data Types – 10/26 – Resolved


Final Update: Wednesday, 28 October 2015 02:08 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 10/28, 00:00 UTC. Our logs show the incident started on 10/26, 15:50 UTC and that during the 32 hours that it took to resolve the issue, a subset of our customers experienced impact in two phases.

  1. First Phase : From 10/26 18:30 UTC to 10/26 20:30 UTC : Data latency outside of SLA for all the data types.
  2. Second Phase : From 10/26 20:30 UTC to 10/28 00:00 UTC : Data gap for the data ingested between 10/26 16:30 UTC  and 10/26 18:20 UTC 
  • Root Cause: The root cause was isolated to the lags in processing our main processing service.
  • Incident Timeline: 32 Hours & 10 minutes – 10/26, 15:50 UTC through 10/28, 00:00 UTC.

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Tuesday, 27 October 2015 16:36 UTC

We observe that 95% of the data has been back filled. Customers should now be able to see almost all their data without any gaps. The team will now focus on mitigating the remaining minor data gap.
 
  • Next Update: Before 10/28 05:00 UTC

-Application Insights Service Delivery Team


Update: Tuesday, 27 October 2015 08:34 UTC

The back filling of the data gap continues. Most of the data has been back filled. Back fill process is taking a bit longer than expected.All current service parameters are showing healthy.

  • Next Update: Before 10/27 15:00 UTC

-Application Insights Service Delivery Team


Update: Tuesday, 27 October 2015 04:26 UTC

The back filling of the data gap continues. Most of the data has been back filled. We expect this to finish in the next 4 hours. 

  • Next Update: Before 10/27 08:30 UTC

-Application Insights Service Delivery Team


Update: Tuesday, 27 October 2015 00:28 UTC

The mitigation measures put in place by the App Insights team continue to back fill the data gap. This process is taking a bit longer than we initially expected, however we expect this to complete in the next 3 hours.

  • Next Update: Before 10/27 03:30 UTC

-Application Insights Service Delivery Team


Update: Monday, 26 October 2015 21:02 UTC

We isolated the problem to a specific component in our processing service. The team has now put in place mitigation measures to contain the impact of the issue caused by this components. Some customers may experience a data gap in their data ingested between 10/26 16:30 UTC  and 10/26 18:20 UTC and we estimate 4 hours before this data gap is back filled.

  • Next Update: Before 10/27 1:30 UTC

-Application Insights Service Delivery Team


Initial Update: Monday, 26 October 2015 18:47 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency. The following data types are affected: Availability,Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace.
  • Next Update: Before 10/26 21:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Application Insights Service Delivery Team


Skip to main content