Experiencing Data Latency for Many Data Types - 6/1 - Resolved


Final Update: Friday, 6/5/2015 20:35 UTC

We have not yet recovered all of the backlogged data, but will combine recovery status and root cause analysis on this blog post:

http://blogs.msdn.com/b/applicationinsights-status/archive/2015/06/03/experiencing-data-latency-for-many-data-types-6-3-second-occurrence-investigating.aspx

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Thursday, 6/4/2015 18:51 UTC

We are continuing to process through the backlog.  Our telemetry indicates approximately 2.5 hours of data left to index.  Processing of backlog data is prioritized behind current data.  We do not have a current ETA on full processing, as processing speed is influenced by a number of external factors, including current throughput.

Thank you for your patience.

Work Around: none
Next Update: 6/5/2015 19:00 UTC

-Application Insights Service Delivery Team


Update: Wednesday, 6/3/2015 20:53 UTC

Processing of the backlog of data between 6/2 14:00 UTC and 6/2 21:25 UTC is proceeding as expected.  We will update this blog once the backlog is complete.

Work Around: none
Next Update: At processing completion time. 

-Application Insights Service Delivery Team


Update: Wednesday, 6/3/2015 00:21 UTC

Root cause has been isolated to intermittent storage interruptions which was impacting write operations for our back-end storage. To address this issue we have adjusted the processing queue to feed from the most-recent data, starting at 21:25 UTC.  Some customers may experience latency for data between 14:00 UTC and 21:25 UTC and this data will become available as soon as it is processed.

Work Around: none
Next Update: Before 6/3/2015 17:00 UTC

-Application Insights Service Delivery Team


Update: Tuesday, 6/2/2015 19:33 UTC

We continue to make good progress through the backlog of data.  Some customers may experience continued data latency with multiple data types.  We currently have no estimate for resolution.

Work Around: none
Next Update: Before 6/3/2015 00:00 UTC

-Application Insights Service Delivery Team


Update: Tuesday, 6/2/2015 16:58 UTC

Our DevOps team has completed the mitigation plan and we are making good progress through the accumulated backlog of data.  Some customers continue to experience data latency with multiple data types.  We currently have no estimate for resolution.

Work Around: none
Next Update: Before 19:00 UTC

-Application Insights Service Delivery Team


Update: Monday, 6/1/2015 23:35 UTC

Our DevOps team continues to work through our mitigation plan, which is proceeding slowly but safely.  Some customers continue to experience data latency with multiple data types.  We currently have no estimate for resolution.

Work Around: none
Next Update: Before 6/2/2015 17:00 UTC

-Application Insights Service Delivery Team


Update: Monday, 6/1/2015 22:09 UTC

Our DevOps team continues to work through our mitigation plan.  Some customers continue to experience data latency with multiple data types.  We currently have no estimate for resolution.

Work Around: none
Next Update: Before 6/2/2015 00:00 UTC

-Application Insights Service Delivery Team


Update: Monday, 6/1/2015 20:04 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time, but we understand that some indices have not been archived properly, which can lead to latency.  While we are working to archive the indices properly, some customers will continue to experience data latency.  We are working to establish the start time for the issue, initial findings indicate that the problem began at 06/01 17:30 UTC.  We currently have no estimate for resolution.

Work Around: none
Next Update: Before 22:00 UTC

-Application Insights Service Delivery Team


Initial Update: Monday, 6/1/2015 17:59 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience data Latency & data access issue. The following data types are affected: Availability, Customer Event, Dependency, Exception, Metric, Page Load, Page View, Performance Counter, Request, Trace.

Work Around: none
Next Update: Before 20:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Application Insights Service Delivery Team

 
 
 
 
 
 
 
 
 
 

Skip to main content