Experiencing Data Latency for Trace Data Type – 9/1 – Resolved


Final Update: Wednesday, 9/2/2015 03:23 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 9/1, 1:00 UTC. Our logs show the incident started on 8/31, 19:00 UTC and that during the 6 hours that it took to resolve the issue some customers experience data latency of the the Trace data type. The residual affect of missing data for the affected timeline was resolved and data was successfully reprocessed at 9/2 3:00 UTC.

Root Cause: The failure was due to an missed configuration change during a deployment.
Lessons Learned: We will drive for more diligence of verifying configuration changes within our deployments.
Incident Timeline: 6 Hours – 8/31, 19:00 UTC through 9/1, 1:00 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Wednesday, 9/2/2015 00:15 UTC

Root cause has been isolated to a missed configuration change during a deployment which was impacting trace data. To address this issue we deployed a change with the correct configuration change. Trace data is now working as expected. Some customers may experience a data gap of the data that came through during the affected time period and we are working on a strategy to recover this data. Unforeseen complications have occurred while trying to re-process the data because of this the new estimated time for the affected data to be reprocessed is by 9/3/2015 00:00 UTC.

Next Update: Before 9/3/2015 00:00 UTC

-Application Insights Service Delivery Team


Update: Tuesday, 9/1/2015 16:55 UTC

Root cause has been isolated to a missed configuration change during a deployment which was impacting trace data. To address this issue we deployed a change with the correct configuration change. Trace data is now working as expected. Some customers may experience a data gap of the data that came through during the affected time period and we are working on a strategy to recover this data. The new time for the data to be re-processed is by 9/2/2015 00:00 UTC

Next Update: Before 9/2/2015 00:00 UTC

-Application Insights Service Delivery Team


Update: Tuesday, 9/1/2015 01:30 UTC

Root cause has been isolated to a missed configuration change during a deployment which was impacting trace data. To address this issue we deployed a change with the correct configuration change. Trace data is now working as expected. Some customers may experience a data gap of the data that came through during the affected time period and we are working on a strategy to recover this data and expect the data to be available again at about 17:00 9/1/2015 UTC.

Next Update: Before 17:00 9/1/2015 UTC

-Application Insights Service Delivery Team


Initial Update: Tuesday, 9/1/2015 00:16 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency. The following data types are affected: Trace.

Next Update: Before 01:16 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Application Insights Service Delivery Team

 

 
 
 
 
 
 
 


Skip to main content