We’ve confirmed that all systems are back to normal with no customer impact as of 08/23, 10:30 PM UTC. Our logs show the incident started on 08/23, 9:00 PM UTC and that during the 1 hour 30 minutes that it took to resolve the issue 1% of customers experienced data gaps for billing history data.
- Root Cause: The failure was due to configuration changes to the billing service.
- Lessons Learned: We have collected logs that we will use to further diagnose the issue and work on improvements that will reduce incidents like this in the future.
- Incident Timeline: 1 Hour & 30 minutes – 08/23, 9:00 PM UTC through 08/23, 10:30 PM UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.
Update: Tuesday, 23 August 2016 21:40 UTC
DevOps is aware and is working on an issue with the Billing service which is impacting billing history data starting at 08/22 9:00 PM UTC.
Root cause for this has been isolated to the configuration change in billing service impacting billing history data for the limited number of customers. Affected customers will see data gap for the impacted duration.We completely understand the issue and are in the process of a hotfix deployment that will resolve the issue. However it will take few hours to complete.
- Work Around: None
- Next Update: Before 08/24 00:00 UTC