Performance Issues with Visual Studio Team Services in Central US region – 10/23 – Mitigated

Final Update: Monday, October 23rd 2017 14:57 UTC We’ve confirmed that all systems are back to normal. Some customers experienced web experience failures, slow page loads, and hosted build failures in the Central US Region. User may have also seen poorly formatted web pages and 504 errors in their builds. Sorry for any inconvenience this…


Issues with Visual Studio Team Services – 10/16 – Mitigated

Final Update: Monday, October 16th 2017 14:03 UTC We’ve confirmed that all systems are back to normal as of 12:45 UTC Monday, October 16th 2017.The issue self-mitigated. As a additional precaution we have failed over the affected database. Root Cause: The issue was due to TempDB Contention on a single Database in the Western Europe…


Postmortem – Availability issues with Visual Studio Team Services on 10 October 2017

On 10 October 2017 we had a global incident with Visual Studio Team Services (VSTS) that had a serious impact on the availability of our service (incident blog here). We know how important VSTS is to our users and deeply regret the interruption in service and are working hard to avoid similar outages in the…


Issues with Visual Studio Team Services – 10/13 – Mitigated

Final Update: Friday, October 13th 2017 14:48 UTC We’ve confirmed that all systems are back to normal as of October 13th 14:20 UTC. Customers would have experience slow and failed commands in the North and South Central US and Western Europe Regions.Sorry for any inconvenience this may have caused. Root Cause: The failure was due…


Issues with Visual Studio Team Services – 10/13 – Mitigated

Final Update: Friday, October 13th 2017 11:47 UTC We’ve confirmed that all systems are back to normal as of October 13th at 11:25 UTC. Our logs show the incident started on October 13th at 10:40 UTC and that during the 45 minute window that it took to resolve the issue. During the affected time period…


Performance Issues with Visual Studio Team Services – 10/11 – Mitigated

Final Update: Wednesday, October 11th 2017 22:34 UTC We’ve confirmed that all systems are back to normal as of 22:00 UTC. Our logs show the incident started on 21:35UTC and that during the 25 minutes that it took to resolve the issue customers experienced slower than expected response times from the service in the South…


Issues with Hosted Build on Visual Studio Team Services – 10/11 – Mitigated

Final Update: Wednesday, October 11th, 2017 14:00 UTC We’ve confirmed that all systems are back to normal as of 11 October 2017 12:44 UTC. During the affected time period a subset of customers may have experienced 504 failures pulling a subset of nuget packages from nuget.org while performing builds. We are sorry for any inconvenience…


Issues with Hosted Build on Visual Studio Team Services – 10/10 – Mitigated

Final Update: Tuesday, October 10th 2017 20:45 UTC We’ve confirmed that all systems are back to normal as of 20:45 UTC. Our logs show the incident started on 10th October 2017 14:25 UTC. Sorry for any inconvenience this may have caused. Root Cause: We are still investigating the root cause of this issue however we…


Issues with Visual Studio Team Services – 10/10 – Mitigated

Final Update: Tuesday, October 10th 2017 15:18 UTC Our telemetry indicates that the issue is mitigated globally as of 14:18 UTC and all VSTS services are fully operational. We are currently working on fixing the underlying cause and we will be posting a root cause analysis of this soon. Apologies for the inconvenience that this…


Issues with Visual Studio Team Services – 10/06 – Mitigated

Final Update: Friday, October 6th 2017 22:45 UTC We have worked with our partners in Azure who have mitigated the underlying infrastructure issue in Australia East region. VSTS Users in that region should not have any more issues accessing the service.We are closely working with Azure to understand the root cause and taking additional steps…