Cloud Load Test issues with Visual Studio Team Services – 10/31 – Resolved

Update: Friday, 3 November 2017 10:00 UTC A fix has been deployed for this issue. Users can now perform new load test runs from the Azure portal without any wait time limitations. Sincerely,Sri Harsha Update: Thursday, 2 November 2017 15:07 UTC A fix for this issue is ready and being tested. It will be included…


Retroactive – Duplicate Download Artifacts Step with Release Management – 10/31 – Mitigated

Initial Update: Tuesday, October 31st 2017 11:27 UTC Users who used Release Management between 2017-10-31 07:06 UTC and 2017-10-31 09:53 UTC may have encountered failures in the “Download artifacts” steps of their release There was an feature flag enabled for the service which caused the “Download artifact” step to be performed in two parts, with…


Issues with Hosted Build and Load Test in South Central US region – 10/30 – Mitigated

Final Update: Monday, October 30th 2017 21:15 UTC Impact for Load Test users has been mitigated after our partners in Azure have fixed the underlying issues. Users can now choose ‘South Central US’ location while triggering their load runs. Sincerely,Sri Harsha Update: Monday, October 30th 2017 18:01 UTC Impact to users of Hosted Build has…


Issues with Visual Studio Team Services – Multiple features that use directory services (AAD graph) are affected in West Europe North Central US and Central US- 10/24 – Mitigated

Final Update: Tuesday, October 24th 2017 14:13 UTC We’ve confirmed that all systems are back to normal as of October 24th 2017 13:35 UTC. 3.4% of customers experienced slow calls to our directory services in the West Europe, North Central US, and Central US during the incident. We apologize for any inconvenience this may have…


Performance Issues with Visual Studio Team Services in Central US region – 10/23 – Mitigated

Final Update: Monday, October 23rd 2017 14:57 UTC We’ve confirmed that all systems are back to normal. Some customers experienced web experience failures, slow page loads, and hosted build failures in the Central US Region. User may have also seen poorly formatted web pages and 504 errors in their builds. Sorry for any inconvenience this…


Issues with Visual Studio Team Services – 10/16 – Mitigated

Final Update: Monday, October 16th 2017 14:03 UTC We’ve confirmed that all systems are back to normal as of 12:45 UTC Monday, October 16th 2017.The issue self-mitigated. As a additional precaution we have failed over the affected database. Root Cause: The issue was due to TempDB Contention on a single Database in the Western Europe…


Postmortem – Availability issues with Visual Studio Team Services on 10 October 2017

On 10 October 2017 we had a global incident with Visual Studio Team Services (VSTS) that had a serious impact on the availability of our service (incident blog here). We know how important VSTS is to our users and deeply regret the interruption in service and are working hard to avoid similar outages in the…


Issues with Visual Studio Team Services – 10/13 – Mitigated

Final Update: Friday, October 13th 2017 14:48 UTC We’ve confirmed that all systems are back to normal as of October 13th 14:20 UTC. Customers would have experience slow and failed commands in the North and South Central US and Western Europe Regions.Sorry for any inconvenience this may have caused. Root Cause: The failure was due…


Issues with Visual Studio Team Services – 10/13 – Mitigated

Final Update: Friday, October 13th 2017 11:47 UTC We’ve confirmed that all systems are back to normal as of October 13th at 11:25 UTC. Our logs show the incident started on October 13th at 10:40 UTC and that during the 45 minute window that it took to resolve the issue. During the affected time period…


Performance Issues with Visual Studio Team Services – 10/11 – Mitigated

Final Update: Wednesday, October 11th 2017 22:34 UTC We’ve confirmed that all systems are back to normal as of 22:00 UTC. Our logs show the incident started on 21:35UTC and that during the 25 minutes that it took to resolve the issue customers experienced slower than expected response times from the service in the South…