Experiencing Data Access Issue in Azure Portal – 8/11 – Resolved


Final Update: Tuesday, 8/11/2015 23:28 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 8/11, 20:56 UTC. Our logs show the incident started on 8/11, 12:00 UTC and that during the 9 hours that it took to resolve the issue a very small number of customers experienced inability to access some of their data uploaded before 8/4.

Root Cause: The failure was due to a configuration setting that negatively impacted performance in our storage back-end.
Lessons Learned:  we have deployed a hotfix to update the configuration setting as a permanent resolution.
Incident Timeline: 8 Hours & 56 minutes – 8/11, 12:00 UTC through 8/11, 20:56 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Tuesday, 8/11/2015 20:52 UTC

Root cause has been isolated to a configuration setting that negatively impacted performance in our storage back-end. To address current impact, we flushed a cache.  This reduced errors, but is not a long-term fix.  As a next step, we are investigating deploying a hotfix which will update the configuration to permanently address the issue.  The error rate has dropped significantly, but some customers may experience continued errors and we estimate several hours before all access issues are addressed.

Work Around: none
Next Update: Before 23:00 UTC

-Application Insights Service Delivery Team


Update: Tuesday, 8/11/2015 19:03 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience errors when accessing data older than 8/4/2015. We are working to establish the start time for the issue, initial findings indicate that the problem began at 8/11/2015 12:00 UTC. We currently have no estimate for resolution.

Customers may also see issues retrieving data due to maintenance work as referenced here:

http://blogs.msdn.com/b/applicationinsights-status/archive/2015/08/08/application-insights-planned-maintenance-8-8-initial-notice.aspx

Work Around: none
Next Update: Before 21:00 UTC

-Application Insights Service Delivery Team


Initial Update: Tuesday, 8/11/2015 17:23 UTC

We are aware of issues within Application Insights and are actively investigating. A very small number of customers may experience and error accessing data older than 2 days.  The following data types are affected: Availability, Customer Event, Dependency, Exception, Metric, Page Load, Page View.

Work Around: none
Next Update: Before 19:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Application Insights Service Delivery Team

 
 
 

Skip to main content