Experiencing Data Latency for Many Data Types – 01/29 – Resolved


Final Update: Saturday, 30 January 2016 05:30 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 01/30, 05:30 UTC. Our logs show the incident started on 01/29, 18:30 UTC and that during the 10 hours that it took to resolve the issue 10% of customers experienced latency on multiple datatypes.
  • Root Cause: The failure was due to bug in processing logic on Application Insights pipeline and an hot fix was deployed to mitigate the issue.
  • Lessons Learned: We have made a hot fix to handle this issue and also we have collected required telemetry data to investigate this issue further and avoid re occurrence of this kind of failures.
  • Incident Timeline: 10 Hours – 01/29, 18:30 UTC through 01/30, 05:30 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Saturday, 30 January 2016 00:43 UTC

Root cause has been isolated to a bug in message processing logic which caused slowdown in data processing. Issue has been hot-fixed, since then data processing is progressing at healthy rate. Latest data flow is now working as expected however  there may be gaps for old data until we process all the backlogs.
  • Work Around: none
  • Next Update: Before 01/30 07:00 UTC

-Application Insights Service Delivery Team


Update: Friday, 29 January 2016 21:43 UTC

Our DevOps team continues to investigate issues within Application Insights. Our development team is working on addressing this issue. Some customers may continue to experience continue to experience gaps in historic data . We currently have no estimate for resolution.
  • Work Around: none
  • Next Update: Before 01/30 04:00 UTC

-Application Insights Service Delivery Team


Update: Friday, 29 January 2016 17:25 UTC

Root cause of the issue identified as a bug in message processing logic. Our development team is working on hotfix this issue. Currently, we have rebooted the underlying services to keep up with processing of the new data. Some customers may continue to experience gaps in historic data . 
  • Work Around: none
  • Next Update: Before 01/29 21:30 UTC

-Application Insights Service Delivery Team


Initial Update: Friday, 29 January 2016 14:20 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency. The following data types are affected: Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace.
  • Work Around: none 
  • Next Update: Before 01/29 18:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Application Insights Service Delivery Team


Skip to main content