Experiencing Data Latency for Customer Event Data Type - 9/30 - Resolved


Final Update: Wednesday, 9/30/2015 16:03 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 9/30, 14:00 UTC. Our logs show the incident started on 9/29, 23:30 UTC.

  • During the first phase of the incident from 9/29 23:30 UTC to 9/30 00:30 UTC, some of our customers would have seen data latency outside of SLA for the Event data type for a period of 5 to 10 minutes.
  • During the second phase of the incident from 9/30 00:30 UTC to 9/30 14:00 UTC, some of our customers would have seen gaps in their data ingested during  9/29 23:30 UTC to 9/30 00:30 UTC (1.5 hours) for the following data types - Customer Event, Exception, Performance Counter, Request and Trace.

Root Cause: The failure was due to an incorrect deployment configuration used in our processing components.
Incident Timeline: 16 Hours & 30 minutes - 9/29, 23:30 UTC through 9/30, 14:00 UTC.

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Wednesday, 9/30/2015 06:53 UTC

The team has set in place components to back fill the data gap that some customers may see for the time between 9/29 23:00 UTC to 9/30 00:30 UTC. This process is expected to finish in the next 12 hours. An update will be provided on this thread once this completes.

Next Update: Before 17:00 UTC

-Application Insights Service Delivery Team


Update: Wednesday, 9/30/2015 02:49 UTC

Root cause has been isolated to a bad deployment which impacted the App Insights processing pipeline. To address this issue we rolled back the bad deployment and reverted to the latest stable release. The processing pipeline is now working as expected and the data latency issue with the Events data type is resolved. However, some customers may see a gap in data for these data types - Customer Event, Exception, Performance Counter, Request and Trace for a time period of 1.5 hours from 9/29 23:00 UTC to 9/30 00:30 UTC. We are working on back filling this data and estimate around 12 hours to address this issue.

Next Update: Before 07:00 UTC

-Application Insights Service Delivery Team


Initial Update: Wednesday, 9/30/2015 01:08 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency. The following data types are affected: Customer Event.

Next Update: Before 02:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Application Insights Service Delivery Team

 

 

 


Skip to main content