Postmortem: VSTS 4 September 2018

Postmortem – VSTS Outage – 4 September 2018 On Tuesday, 4 September 2018, VSTS (now called Azure DevOps) suffered an extended outage affecting customers with organizations hosted in the South Central US region (one of the 10 regions globally hosting VSTS customers). The outage also impacted customers globally due to cross-service dependencies. It required more…


TFS Database Import Service failures in Central US – 09/10 – Mitigated

Final Update: Monday, September 10th 2018 18:29 UTC Fix has been deployed and the imports have been verified to succeed in the United States. Sincerely,Samuli Update: Monday, September 10th 2018 17:20 UTC We have indentified an issue causing a high percentage of import jobs in the United States to fail. Team is preparing a hotfix…


False Alarm – Failure to load login page of VSTS using AAD in all regions – 09/09 – Closed

Final Update: Sunday, September 9th 2018 10:59 UTC We have reviewed our telemetry and discovered that there was no external user impact for this incident. We are sorry for alerting you unnecessarily. Sincerely,Dexter Initial notification: Sunday, September 9th 2018 10:40 UTC We’re investigating failures of loading the login page when using AAD in all regions….


Possible customer impacting event in North Central US – 09/06 – Mitigated

Final Update: Thursday, September 6th 2018 04:41 UTC The incident got auto mitigated. We’ve confirmed that all systems are back to normal as of Sep 6th 2018 2:30 AM UTC. Initial analysis suggests that it was related to Release Management issue observed today and we are working on finding a detailed root cause.Sorry for any…


Release Management Unavailable in South Central US – 09/06 – Mitigated

Final Update: Thursday, September 6th 2018 16:24 UTC We have confirmed that Release Management is still recovered. No regression over last 12 hours. We are mitigating this issue and will closely monitor Sincerely,Tom Release Management is currently recovered in South Central US We are continuing to actively monitor Release Management in South Central US while…


VSTS Impact due to Datacenter Outage in South Central US – 09/05 – Resolved

Update: Thursday, September 6th 00:50 UTC Storage accounts impacting VSTS services have been recovered We have validated that storage errors and performance of key workflows is back to normal  We will continue to monitor Thanks, Tom Update: Wednesday, September 5th 20:30 UTC We are seeing overall improvement in performance of VSTS features for users in South…


Intermittent Git failures in multiple regions – 08/31 – Mitigated

Final Update: Friday, August 31st 2018 13:10 UTC We’ve confirmed that all systems are back to normal as of 8/31 12:59 UTC. Our logs show the incident started on 8/30 21:44 UTC and that during the 15 hours and 15 minutes that it took to resolve the issue. During that time users connecting to Git…


Hosted Mac Build Pool Outage in South Central US – 08/29 – Mitigated

Final Update: Wednesday, August 29th 2018 23:21 UTC The rollback of the agent update has completed. We have confirmed that the issue has been resolved. Sincerely,Tom Update: Wednesday, August 29th 2018 22:48 UTC We are mitigating the issue currently by rolling back an Agent update. We will confirm back once the mitigation is fully deployed….


Build and Release failures when using BitBucket and OAuth in all regions. 08/29 – Mitigated

Final Update: Thursday, August 30th 2018 14:45 UTC We believe that all issues have now been resolved. Users should no longer see Build failures when using Bitbucket and OAuth.Sorry for any inconvenience this may have caused. Sincerely,Niall Update: Wednesday, August 29th 2018 17:20 UTC The configuration change to mitigate the issue on the VSTS side…