Final Update: Tuesday, 08 March 2016 08:00 UTC
We’ve confirmed that all systems are back to normal with no customer impact as of 03/08, 08:00 UTC. Our logs show the incident started on 03/07, 14:50 UTC and that during the 17 hours & 10 minutes that it took to resolve the issue some customers would have experienced data latency beyond 2 hours.
Root Cause: Root cause has been isolated to an upgrade of an upstream service which was impacting processing pipeline.
Lessons Learned: We have collected telemetry logs and identified steps to be taken to avoid these kind of scenarios in future.
Incident Timeline: 17 Hours & 10 minutes – 3/7, 14:50 UTC through 3/8, 08:00 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.
Update: Tuesday, 08 March 2016 05:00 UTC
Telemetry data is being processed as expected and there is no latency in the current data. However some customers may experience data latency for older data and we estimate nearly 6 hours before all the latency issue is addressed.
•Work Around: none
•Next Update: Before 03/08 11:00 UTC
Update: Monday, 07 March 2016 22:23 UTC
Root cause has been isolated to an upgrade of an upstream service which was impacting our ability to process telemetry data efficiently. To address this issue more capacity was added to downstream service in order to compensate for lost capacity due to upgrade. Telemetry is being processed as expected however some customers may experience data latency for older data and we estimate nearly 6 hours before all the latency issue is addressed.
•Work Around: none
•Next Update: Before 03/08 04:30 UTC
Initial Update: Monday, 07 March 2016 04:00 PM UTC
We are aware of issues within Application Insights and are actively investigating. Some customers may experience latency outside of SLA for data ingested into App Insights. The following data types are affected: Availability,Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace.
Next Update: Before 03/07 10:58 PM UTC
We are working hard to resolve this issue and apologize for any inconvenience.