Experiencing Data Latency for Multiple Functional Areas – 03/08 – Mitigated


Final Update: Tuesday, 08 March 2016 08:00 UTC
We’ve confirmed that all systems are back to normal with no customer impact as of 03/08, 08:00 UTC. Our logs show the incident started on 03/07, 14:50 UTC and that during the 17 hours & 10 minutes that it took to resolve the issue some customers would have experienced data latency beyond 2 hours.

Root Cause: Root cause has been isolated to an upgrade of an upstream service which was impacting processing pipeline.
Lessons Learned: We have collected telemetry logs and identified steps to be taken to avoid these kind of scenarios in future.
Incident Timeline: 17 Hours & 10 minutes – 3/7, 14:50 UTC through 3/8, 08:00 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Praveen Vavilla

Update: Tuesday, 08 March 2016 05:00 UTC

Telemetry data is being processed as expected and there is no latency in the current data. However some customers may experience data latency for older data and we estimate nearly 6 hours before all the latency issue is addressed.

•Work Around: none
•Next Update: Before 03/08 11:00 UTC
-Praveen Vavilla

Update: Monday, 07 March 2016 22:23 UTC

Root cause has been isolated to an upgrade of an upstream service which was impacting our ability to process telemetry data efficiently. To address this issue more capacity was added to downstream service in order to compensate for lost capacity due to upgrade. Telemetry is being processed as expected however some customers may experience data latency for older data and we estimate nearly 6 hours before all the latency issue is addressed.

•Work Around: none
•Next Update: Before 03/08 04:30 UTC
-Pankaj Suri

Initial Update: Monday, 07 March 2016 04:00 PM UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience latency outside of SLA for data ingested into App Insights. The following data types are affected: Availability,Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace.
Next Update: Before 03/07 10:58 PM UTC
We are working hard to resolve this issue and apologize for any inconvenience.

-Praveen Vavilla


Skip to main content