Experiencing Data Latency for Application Insight Event and Request Data Types - 5/4 - Resolved


Final Update: Monday, 5/4/2015 23:58 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 5/4, 23:30 UTC. Our logs show the incident started on 5/4, 16:45 UTC and that during the seven hours that it took to resolve the issue, most of our customers experienced latency with viewing data related to Customer Events.

Root Cause: The failure was due to a sudden degradation of performance of the messaging infrastructure that we use to relay data across the different Application Insight components.
Chance of Reoccurrence: Minimal
Incident Timeline:  6 Hours & 45 minutes - 5/4, 16:45 UTC through 5/4, 23:30 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Monday, 5/4/2015 22:12 UTC

This is to update that except for Events Data type all other data types have recovered and are operating within the SLA. We are working on recovering Events Data type. Our current estimate is that Events will recover in next 2-3 hours.

Work Around:  None
Next Update: Before 00:00 5/5/2015 UTC

-Application Insights Service Delivery Team


Update: Monday, 5/4/2015 19:36 UTC

Our DevOps team continues to investigate issues within Application Insights. We identified a bad node in one of our upstream services which as been taken out of rotation since. However, the recovery is slower than expected and we are actively looking into it.

As a result customers may experience a >2 hours of latency for data types Events, Requests and RuntimeDependency.

Work Around: None
Next Update: Before 22:00 UTC

-Application Insights Service Delivery Team


Initial Update: Monday, 5/4/2015 17:30 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency outside of SLA while trying to access data related to the following data types : Request, Event and RDD.

Work Around: none
Next Update: Before 19:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Application Insights Service Delivery Team

 
 
 

Skip to main content