Experiencing Data Latency for Many Data Types – 04/19 – Resolved


Final Update: Wednesday, 20 April 2016 05:51 UTC

We’ve confirmed that all systems are back to normal and the backlog data has also processed as of 4/19, 20:50 UTC. Our logs show the incident started on 4/19, 08:54 UTC and that during the 12 hours that it took to resolve the issue some customers might have experienced data latency for all data types except availability..
  • Root Cause: It was found that there were few back-end services which where responding to requests slowly. These services were scaled out and rebooted to mitigate the issue. We have not seen the slow writes since the reboot..
  • Lessons Learned: We have collected telemetry logs and identified steps to be taken to avoid these kind of scenarios in future.
  • Incident Timeline: 12 Hours - 4/19, 08:54 UTC through 4/19, 20:50 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Girish K


Update: Tuesday, 19 April 2016 20:24 UTC

We ran into further slowdowns which has extended the time in which the processing of the backlog will take. We are looking into ways to resolve the slowdowns that we are experiencing. The backlog is continuing to make progress in processing through the latent data. Customers will continue to see data gaps that are outside of SLA. Current data is within the 2 hour SLA. 
  • Work Around: none
  • Next Update: Before 04/20 08:30 UTC

-Randy


Update: Tuesday, 19 April 2016 14:32 UTC

We estimated 4 hours before all the backlog is processed.The processing of the backlog data is taking longer than expected. We scaled up our back-end services as well to fasten up the processing time.
  • Work Around: none
  • Next Update: Before 04/19 21:00 UTC

-Girish K


Update: Tuesday, 19 April 2016 10:43 UTC


Root cause has been isolated to slow processing in a back-end service which was impacting multiple data types. To address this issue we rebooted the affected service. Current Data processing is working as expected. Some customers may experience data gaps outside of 2 hour SLA for the data ingested between 9:00 AM - 10:00 AM UTC and we estimate 4 hours before all the backlog is processed.


  • Work Around: None
  • Next Update: Before 04/19 15:00 UTC

-Girish K


Skip to main content