Experiencing Export Data Latency for All Data Types – 12/01 – Resolved


Final Update: Saturday, 03 December 2016 17:14 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 12/03, 05:00 UTC. Our logs show the incident started on 12/30, 21:00 UTC and that during the 56 hours that it took to resolve the issue small percentage of customers experienced continuous export data latency over 2 hours.
  • Root Cause: The failure was due to unexpected increase in data volumes which was impacting continuous export service
  • Lessons Learned: Continuous export scale setting will be adjusted to make sure this issue doesn’t happen in the future
  • Incident Timeline: 56 hours – 12/30, 21:00 UTC through 12/03, 05:00 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Vitaliy


Update: Saturday, 03 December 2016 00:48 UTC

Small number of customers may still experience export data latency over 2 hours until all backlog is processed. The estimate was revised to 24 hours for the rest of the backlog data to be processed.
  • Work Around: None
  • Next Update: Before 12/03 19:00 UTC

-Vitaliy


Update: Friday, 02 December 2016 21:00 UTC

Continuous Export service has processed through majority of backlog data. Small number of customers may still experience export data latency over 2 hours until all backlog is processed. We estimate that all data should be fully processed within next 12 hours.
  • Work Around: none
  • Next Update: Before 12/03 01:00 UTC

-Vitaliy


Update: Friday, 02 December 2016 03:19 UTC

Continuous Export service has processed through majority of backlog data. Small number of customers may still experience export data latency over 2 hours until all backlog is processed. 
  • Work Around: None
  • Next Update: Before 12/02 21:30 UTC

-Vitaliy


Update: Thursday, 01 December 2016 23:03 UTC

Root cause has been isolated to increase in data volumes which was impacting continuous export service. To address this issue we have increased the scale of continuous export service. Continuous Export service is now working as expected and processing through backlog of data. Some customers may experience export data latency over 2 hours until all backlog is processed. 
  • Work Around: none
  • Next Update: Before 12/02 03:30 UTC

-Vitaliy


Initial Update: Thursday, 01 December 2016 20:45 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency in Continuous Export. The following data types are affected: All data types
  • Work Around: none
  • Next Update: Before 12/01 23:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Vitaliy


Skip to main content