Azure Boards Slowness in Central US – 09/26 – Mitigated

Final Update: Wednesday, September 26th 2018 14:59 UTC We’ve confirmed that all systems are back to normal as of 26 September 14:08 UTC. Our logs show the incident started on 26 September 13:45 UTC and that during the 23 minutes that it took to resolve the issue. 0.3% of customers experienced Azure Boards slowness, primarily…


Customer impacting event in South Brazil – 09/25 – Mitigated

Final Update: Tuesday, September 25th 2018 22:16 UTC We’ve confirmed that all systems are back to normal as of 21:50 UTC on September 25 2018. Initial analysis shows one front end role instance was having issues after a recent deployment and that has since been mitigated. We have gathered additional diagnostics and will work towards…


Postmortem – VS Marketplace outage – 4 September 2018

On Tuesday, 4 September 2018, Visual Studio Marketplace suffered an extended outage affecting most of its customers. Marketplace hosts and serves extensions for the Visual Studio IDE, Visual Studio Code, and Azure DevOps. This was the first instance of the Marketplace service going down completely, and we sincerely apologize for the outage. What happened and…


Postmortem: VSTS 4 September 2018

Postmortem – VSTS Outage – 4 September 2018 On Tuesday, 4 September 2018, VSTS (now called Azure DevOps) suffered an extended outage affecting customers with organizations hosted in the South Central US region (one of the 10 regions globally hosting VSTS customers). The outage also impacted customers globally due to cross-service dependencies. It required more…


TFS Database Import Service failures in Central US – 09/10 – Mitigated

Final Update: Monday, September 10th 2018 18:29 UTC Fix has been deployed and the imports have been verified to succeed in the United States. Sincerely,Samuli Update: Monday, September 10th 2018 17:20 UTC We have indentified an issue causing a high percentage of import jobs in the United States to fail. Team is preparing a hotfix…


False Alarm – Failure to load login page of VSTS using AAD in all regions – 09/09 – Closed

Final Update: Sunday, September 9th 2018 10:59 UTC We have reviewed our telemetry and discovered that there was no external user impact for this incident. We are sorry for alerting you unnecessarily. Sincerely,Dexter Initial notification: Sunday, September 9th 2018 10:40 UTC We’re investigating failures of loading the login page when using AAD in all regions….


Possible customer impacting event in North Central US – 09/06 – Mitigated

Final Update: Thursday, September 6th 2018 04:41 UTC The incident got auto mitigated. We’ve confirmed that all systems are back to normal as of Sep 6th 2018 2:30 AM UTC. Initial analysis suggests that it was related to Release Management issue observed today and we are working on finding a detailed root cause.Sorry for any…


Release Management Unavailable in South Central US – 09/06 – Mitigated

Final Update: Thursday, September 6th 2018 16:24 UTC We have confirmed that Release Management is still recovered. No regression over last 12 hours. We are mitigating this issue and will closely monitor Sincerely,Tom Release Management is currently recovered in South Central US We are continuing to actively monitor Release Management in South Central US while…


VSTS Impact due to Datacenter Outage in South Central US – 09/05 – Resolved

Update: Thursday, September 6th 00:50 UTC Storage accounts impacting VSTS services have been recovered We have validated that storage errors and performance of key workflows is back to normal  We will continue to monitor Thanks, Tom Update: Wednesday, September 5th 20:30 UTC We are seeing overall improvement in performance of VSTS features for users in South…