Experiencing Data Latency for Many Data Types - 6/20 - Resolved


Final Update: Wednesday, 6/24/2015 23:53 UTC

All missing exception data has been reprocessed and customers should not experience data gap for exception data between 06/20, 20:30 UTC and 06/21, 18:52 UTC.

Root Cause: The failure was due to issues with data processing service.
Lessons Learned: We have updated our service to make sure this issue doesn't happen in the future.
Incident Timeline:  20 Hours & 22 minutes - 06/20, 22:30 UTC through 06/21, 18:52 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Wednesday, 6/24/2015 20:06 UTC

Some customers may still experience data gap for exception data between 06/20, 20:30 UTC and 06/21, 18:52 UTC. We have started reprocessing of the missing exception data. We estimate that all data will be reprocessed within next 24 hours or less.

Work Around: none
Next Update: Before 6/25 20:00 UTC

-Application Insights Service Delivery Team


Update: Tuesday, 6/23/2015 22:24 UTC

Some customers may still experience data gap for exception data between 06/20, 20:30 UTC and 06/21, 18:52 UTC. We are still working on a solution to reprocess that missing data. There is no estimate for when this missing data will be reprocessed at the moment.

Work Around: none
Next Update: Before 6/24 20:00 UTC

-Application Insights Service Delivery Team


Update: Monday, 6/22/2015 19:23 UTC

Some customers may still experience data gap for exception data between 06/20, 20:30 UTC and 06/21, 18:52 UTC. We are working on a solution to reprocess that missing data. We estimate that all data will be reprocessed within next 24 hours or less.

Work Around: none
Next Update: Before 6/23 20:00 UTC

-Application Insights Service Delivery Team


Update: , 6/21/2015 19:22 UTC

We’ve confirmed that all systems are back to normal with no impact to current data as of 06/21, 18:52 UTC. Our logs show the incident started on 06/20, 20:30 UTC and that during the 23 hours that it took to resolve the issue majority of customers experienced data latency over 2 hours for most of data types. Some customers may experience data gap for exception data between 06/20, 20:30 UTC and 06/21, 18:52 UTC. We are working on a solution to reprocess that missing data.

Work Around: none
Next Update: Before 6/22 20:00 UTC

-Application Insights Service Delivery Team


Update: , 6/21/2015 12:14 UTC

Root cause has been isolated to issues with data processing service. Data processing service is now working as expected and back log data is catching up. Some customers may still experience data latency over 2 hours for the following data types: event, exception, request, page view, performance counter. We estimate 8 hours before all data latency is processed.

Work Around: None
Next Update: Before 6/21 20:00 UTC

-Application Insights Service Delivery Team


Update: , 6/21/2015 08:05 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time.  Customers continue to experience data latency over 2 hours (current data latency is between 8 and 10 hours) staring at 06/20 ~22:30 UTC. We have started the mitigation and all of the data types (minus the exception data) are catching up. We currently have no estimate for resolution.

Work Around: none
Next Update: Before 6/21 12:00 UTC

-Application Insights Service Delivery Team


Update: , 6/21/2015 03:58 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Customers continue to experience data latency over 2 hours (current data latency is between 6 and 8 hours) staring at 06/20 ~22:30 UTC. At this point most of data types are affected (Exception, Metric, Request, Events, Page View, Page Load, Performance Counter, Trace). We currently have no estimate for resolution.

Work Around: none
Next Update: Before 6/21 08:00 UTC

-Application Insights Service Delivery Team


Update: , 6/21/2015 00:39 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience data latency over 2 hours. We are working to establish the start time for the issue, initial findings indicate that the problem began at 06/20 ~22:30 UTC. At this point most of data types are affected (Exception, Metric, Request, Events, Page View, Page Load, Performance Counter, Trace). We currently have no estimate for resolution.

Work Around: none
Next Update: Before 6/21 04:00 UTC

-Application Insights Service Delivery Team


Initial Update: Saturday, 6/20/2015 22:39 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency. The following data types are affected: Exception, Metric, Request.

Work Around: None
Next Update: Before 6/21 02:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Application Insights Service Delivery Team

 
 
 
 
 
 
 
 
 

Skip to main content