503 Error Pages When Accessing Accounts in Central US – 06/29 – Mitigated

Final Update: Friday, June 29th 2018 18:30 UTC We’ve confirmed that all systems are back to normal as of 18:03 UTC. Underlying issue with storage in Central US has been mitigated Sincerely,Tom Initial Update: Friday, June 29th 2018 17:41 UTC We’re investigating Performance Degradation in Central US. Users may see 503 error pages when accessing…


Issues accessing Build Summary page in VSTS in multiple regions – 06/26 – Mitigated

Final Update: Tuesday, June 26th 2018 08:49 UTC We’ve confirmed that all systems are back to normal as of 2018-06-26 9:10 UTC. During this impacted time period users may have experienced 404 failures while accessing the build summary page. The cause of this issue is due to a recent change in the deployment and a…


Performance Degradation for Visual Studio Team Services Packaging Services in South Central US – 06/25 – Mitigated

Final Update: Monday, June 25th 2018 20:49 UTC We’ve confirmed that all systems are back to normal as of 06/25/2018 20:17 UTC. The issue occurred due to one of Azure storage stamp being unavailable and mitigation steps have been applied to restore the services. We will continue to work with our partners in Azure to…


Performance Degradation for Visual Studio Team Services Analytics Services in South Central US – 06/25 – Mitigated

Final Update: Monday, June 25th 2018 20:02 UTC We have confirmed that functionality has been restored The underlying issue is due to a database which came up in read-only mode after an upgrade We are following up on our end to understand root cause Sincerely,Tom Initial Update: Monday, June 25th 2018 18:57 UTC We’re investigating…


Performance Degradation in South Central US – 06/23 – Mitigated

Final Update: Saturday, June 23rd 2018 14:22 UTC We’ve confirmed that all systems are back to normal as of Saturday, June 23rd 2018 14:00 UTC . Our logs show the incident started on Saturday, June 23rd 2018 03:28 UTC and that during the 10 hours and 32 minutes that it took to resolve the issue…


Performance Degradation in South Brazil – 06/19 – Mitigated

Final Update: Tuesday, June 19th 2018 16:19 UTC We have taken steps in order to mitigate this issue and systems are returning to normal as of 16:10 UTC Sincerely,Pedro Initial Update: Tuesday, June 19th 2018 15:24 UTC We’re currently investigating notification delays and potential build issues in South Brazil. Next Update: Before Tuesday, June 19th…


Errors for Blobstore Requests in South Central US – 06/14 – Mitigated

Final Update: Thursday, June 14th 2018 20:45 UTC Storage team has applied mitigation that fully resolved the issue. We’ve confirmed that all systems are back to normal as of 6/14 20:09 UTC . Based on telemetry the impact started at 6/14 01:55 UTC. Storage team will complete a root cause analysis and we will update…


HTTP 500/503 Errors in South Central US – 06/13 – Mitigated

Final Update: Wednesday, June 13th 2018 19:15 UTC We’ve confirmed that all systems are back to normal as of 6/13 18:30 UTC after Azure has applied mitigation to the underlying Storage availability issue in South Central US region. More details about this incident can be found on Azure status portal. We will update this post…


Hosted Agent Assignment Delays in West Europe – 06/12 – Mitigated

Final Update: Tuesday, June 12th 2018 17:43 UTC We’ve confirmed that all systems are back to normal as of June 12th 17:15 UTC. Our logs show the incident started on June 12th 13:05 UTC and that during the 4 hours and 15 minutes that it took to resolve the issue, customers in West Europe who…


Investigating issues with hosted build agents in Central US – 06/12 – Mitigated

Final Update: Tuesday, June 12th 2018 08:37 UTC We’ve confirmed that all systems are back to normal as of June 12th 2018 08:06 UTC . Our logs show the incident started on June 12th 2018 03:15 UTC. We have identified the root cause to be related to a recent change that got deployed today. The…