Experiencing Data Gaps for Many Data Types – 6/20 – Resolved


Final Update: , 6/21/2015 03:40 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 6/21/2015, 01:30 UTC. Our logs show the incident started on 6/19/2015, 23:49 UTC and that during the 25 Hours & 41 minutes that it took to resolve the issue some customers may have experienced data gaps for their recently registered apps. New app registrations will continue to take up to 15 minutes before telemetry will be collected.

Root Cause: The failure has been isolated to an interruption in communication between backend systems.
Lessons Learned: We have gathered all of the applicable logs and telemetry and will be working with our partner teams to understand the root cause of the communication interruption.
Incident Timeline: 25 Hours & 41 minutes -6/19/2015, 23:49 UTC through 6/21/2015, 01:30 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: , 6/21/2015 01:38 UTC

Root cause has been isolated to backend connectivity issues. To address this issue, we have deployed hotfixes to the affected nodes. Telemetry for apps registered starting at 06/20 00:00 UTC is now viewable if the client buffer was not exceeded. For some customers, there may be a delay of up to 15 minutes before telemetry starts to be collected for their new app registrations.

Work Around: none
Next Update: Before 6/21/2015 08:00 UTC

-Application Insights Service Delivery Team


Update: Saturday, 6/20/2015 19:27 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is believed to be related to backend connectivity issues. A hotfix is being developed and rolled out to the affected systems. Only new customers/apps, which were created after 06/20 ~00:00 UTC are affected. We currently have no estimate for resolution.

Work Around: none
Next Update: Before 6/21/2015 01:30 UTC

-Application Insights Service Delivery Team


Update: Saturday, 6/20/2015 14:16 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience data gaps for many data types. We are working to establish the start time for the issue, initial findings indicate that the problem began at 06/20 ~00:00 UTC. At this point, only new customers/apps, which were created after 06/20 ~00:00 UTC are affected. We currently have no estimate for resolution.

We also had a brief window of query failures, which would affect 10-30% of customers between 12:45 and 13:30 UTC on 6/20.

Work Around: none
Next Update: Before 21:00 UTC

-Application Insights Service Delivery Team


Update: Saturday, 6/20/2015 10:50 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience data gaps for many data types. We are working to establish the start time for the issue, initial findings indicate that the problem began at 06/20 ~00:00 UTC. At this point, only new customers/apps, which were created after 06/20 ~00:00 UTC are affected. We currently have no estimate for resolution.

Work Around: none
Next Update: Before 15:00 UTC

-Application Insights Service Delivery Team


Update: Saturday, 6/20/2015 06:45 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience data gaps for many data types. We are working to establish the start time for the issue, initial findings indicate that the problem began at 06/20 ~00:00 UTC. At this point, only new customers/apps, which were created after 06/20 ~00:00 UTC are affected. We currently have no estimate for resolution.

Work Around: None
Next Update: Before 11:00 UTC

-Application Insights Service Delivery Team

 


Initial Update: Saturday, 6/20/2015 04:09 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Gaps. The following data types are affected: Customer Event, Dependency, Exception, Metric, Page Load, Page View, Performance Counter, Request, Trace. Only a small percentage of data is affected (~0.7%).

Work Around: None
Next Update: Before 6/20 07:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Application Insights Service Delivery Team

 
 
 
 
 
 

Skip to main content