Experiencing Latency and Data Loss for Multiple Functional Areas – 03/11 – Resolved


Final Update: Friday, 11 March 2016 15:55 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 03/11, 16:00 UTC. Our logs show the incident started on 03/11, 04:40 UTC and that during the 11 hours 20 minutes hours that it took to resolve the issue some customers might have experienced data latency.
  • Root Cause: The failure was due to dependent service of Application Insights platform which once recovered mitigated the issue.
  • Lessons Learned: We have collected required telemetry data and will be investigating more using the same  to avoid such occurrences in future. 
  • Incident Timeline: 11 Hours & 20 minutes – 03/11, 04:40 UTC through 03/11, 16:00 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Praveen


Update: Friday, 11 March 2016 14:16 UTC

Application insight processing service is chewing up the backlog data at an healthier rate without any hicups and no latency in the current data. We estimate 4 more hours before all the backlog data is processed. Some customers will continue to experience data latency for the back log data. 

  • Work Around: None
  • Next Update: Before 03/11 18:30 UTC

-Girish K


Update: Friday, 11 March 2016 07:41 UTC

Root cause has been isolated to issue with a dependent service of Application Insights platform which caused data latency. Once issues with dependent service is mitigated, Application insights services came back healthy. However we have backlog data which needs to be processed .We estimate 6 hours before all the backlog data is processed. Current data will be processed without any issues.
  • Work Around: None
  • Next Update: Before 03/11 12:00 UTC

-Girish K


Initial Update: Friday, 11 March 2016 05:29 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Latency and Data Loss. The following data types are affected: Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace.
  • Work Around: None
  • Next Update: Before 03/11 11:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Girish K




Comments (0)

Skip to main content