Experiencing Latency and Data Loss for Many Data Types – 04/09 – Resolved


Final Update: Sunday, 10 April 2016 19:22 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 04/10, 19:10 UTC. Our logs show the incident started on 04/09, 16:30 UTC and that during the 27 hours that it took to resolve the issue some of customers experienced data latency for many data types.
  • Root Cause: The failure was due to Azure outage in East US datacenter.
  • Lessons Learned: Azure partner teams will address underlying platform issues to make sure such outages don’t happen in the future.
  • Incident Timeline:  26 Hours & 40 minutes – 04/09, 16:30 UTC through 04/10, 19:10 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Vitaliy


Update: Sunday, 10 April 2016 14:49 UTC

All Application Insights Services are now working as expected. Customer will see the current data without any latency and some may still continue to experience latency for backlog data. The following data types are affected: Metric,Page Load,Page View,Performance Counter,Request,Trace.
  • Work Around: None
  • Next Update: Before 04/10 21:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Girish K


Update: Sunday, 10 April 2016 01:09 UTC

Root cause has been isolated to outage in East US Azure datacenter which was impacting multiple Application Insights services. Azure teams fully mitigated East US outage as of 23:45 UTC on 4/09/2016. All Application Insights services are now working as expected. Some customers may experience data gaps for data sent between 16:30 UTC and 23:59 UTC on 4/09/2016 and we estimate 16 hours before all backlog data is processed. Data sent after 00:00 UTC on 4/10/2016 is not affected.
  • Work Around: Customers can use current data (data sent after 00:00 UTC on 4/10/2016)
  • Next Update: Before 04/10 13:30 UTC

-Vitaliy


Update: Saturday, 09 April 2016 19:49 UTC

Root cause has been isolated to outage in East US Azure datacenter (https://azure.microsoft.com/en-us/status/) which is impacting multiple Application Insights services. Azure teams started to apply mitigation procedures, but it might take up to 6 hours for mitigation to fully take affect. Some customers may experience data latency and Availability test data loss (when trying to download .webtest files) and we estimate 6+ hours before all those issues are addressed.
  • Work Around: None
  • Next Update: Before 04/10 02:00 UTC

-Vitaliy


Initial Update: Saturday, 09 April 2016 17:41 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Latency and Data Loss. The following data types are affected: Availability,Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace.
  • Work Around: None
  • Next Update: Before 04/09 20:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Vitaliy


Skip to main content