Experiencing Alerting failure for Availability Data Type – 9/2 – Resolved


Final Update: Friday, 9/4/2015 18:40 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 9/4, 18:30 UTC. Our logs show the incident started on 8/28, 21:00 UTC and that during the 7 days that it took to resolve the issue 19 customers experienced the inability to enable/disable an availability web test.

Root Cause: The failure was due to an optimization change that occurred during a deployment. We turned of the optimization in order to resolve the issue.
Incident Timeline: 7 days & 3 hours – 9/4, 18:30 UTC through 8/28, 21:00 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Friday, 9/4/2015 00:18 UTC

Root cause has been isolated to an issue with a optimization deployment which is impacting the ability to disable and enable web tests. To address this issue we are working on a fix that will disable this optimization. Customers will continue to experience the inability to enable and disable web tests until the fix is released. The fix is taking longer than expected to prepare, thus we are now expecting this fix to be deployed by 9/4/2015 19:00 UTC.

Work Around:

  • Customers can still disable the alerting functionality of the webtest by clicking on the webtest and then edit. On that next screen click on alerts than disabled and then click save.
  • If historical data is not required a customer can delete the webtest and then re-create it at a later date if historical data is not important.
Next Update: Before 9/4/2015 19:00 UTC

-Application Insights Service Delivery Team


Update: Thursday, 9/3/2015 21:14 UTC

Root cause has been isolated to an issue with a optimization deployment which is impacting the ability to disable and enable web tests. To address this issue we are working on a fix that will disable this optimization. Customers will continue to experience the inability to enable and disable web tests until the fix is released. We expect this fix to be deployed by 9/4/2015 01:00 UTC.

Work Around:

  • Customers can still disable the alerting functionality of the webtest by clicking on the webtest and then edit. On that next screen click on alerts than disabled and then click save.
  • If historical data is not required a customer can delete the webtest and then re-create it at a later date if historical data is not important.
Next Update: Before 9/4/2015 01:00 UTC

-Application Insights Service Delivery Team


Initial Update: Wednesday, 9/2/2015 23:30 UTC

Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers are unable to disable availability webtests. We are working to establish the start time for the issue We currently have no estimate for resolution.

Work Around:

  • Customers can still disable the alerting functionality of the webtest by clicking on the webtest and then edit. On that next screen click on alerts than disabled and then click save.
  • If historical data is not required a customer can delete the webtest and then re-create it at a later date if historical data is not important.

Next Update: Before 9/3/2015 21:00 UTC

-Application Insights Service Delivery Team

 
 
 
 
 
 
 
 
 
 
 

Skip to main content