Experiencing Latency for Multiple Functional Areas – 03/18 – Resolved


Final Update: Friday, 18 March 2016 10:24 UTC

We've confirmed that all systems are back to normal with no customer impact as of 3/18, 10:30 UTC. Our logs show the incident started on 3/18, 4:35 UTC and that during the 5 hours & 55 minutes that it took to resolve the issue some customers would have experienced Latency and small percentage of customers would have experienced Data Loss.
  • Root Cause: Root cause is isolated to a performance issue with a dependent platform of Application Insights service.
  • Lessons Learned: We have collected telemetry logs and identified steps to be taken to avoid these kind of scenarios in future.
  • Incident Timeline: 5 Hours & 55 minutes - 3/18, 4:35 UTC through 3/18, 10:30 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Rama


Update: Friday, 18 March 2016 07:20 UTC

Root cause has been isolated to performance degradation with underlying platform service which once recovered helped in mitigating the current issue. However we do have telemetry backlog that needs to be processed in order to fully recover the data gaps. Application insights processing service is chewing up the backlog data at a healthier rate without any hiccups and no latency in the current data. We estimate 6 more hours before all the backlog data is processed. Some customers will continue to experience data latency for the backlog data.

  • Work Around: None.
  • Next Update: Before 03/18 13:30 UTC

-Rama


Initial Update: Friday, 18 March 2016 04:35 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Latency and small percentage of customers may experience  Data Loss. The following data types are affected: Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace.
  • Work Around: None
  • Next Update: Before 03/18 09:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Durga


Comments (0)

Skip to main content