Issues with Visual Studio Team Services – Build – 09/20 – Mitigated

Final Update: Wednesday, 20 September 2017 19:03 UTC We’ve confirmed that all systems are back to normal as of September 20th 2017 17:20 UTC. The fix for the build queue failures has been deployed globally. During the incident, customers may have seen the following build queue error: “Value cannot be null. Parameter name: value”.  Sincerely,Daniel Update: Wednesday,…


Issues with Visual Studio Team Services – 09/20 – Mitigated

Final Update: Wednesday, September 20th 2017 14:05 UTC The issue has now been fully mitigated.Some users may have observed delays in their build requests during the time. This was due to a specific capacity issue in West Europe Azure region. As a mitigation step additional capacity has been added in North Europe Azure region. Root…


Issues with My.VisualStudio.com in Visual Studio Team Services – 09/18 – Mitigated

Final Update: Monday, 18 September 2017 23:38 UTC We’ve confirmed that all systems are back to normal, users are now able to load the my.visualstudio.com page. During this issue, customers were not able to access the my.visualstudio.com page. Root cause for this issue is still under investigation. Sincerely,Daniel Update: Monday, 18 September 2017 22:50 UTC…


Hosted Build issues with Visual Studio Team Services – 09/18 – Mitigated

Final Update: Monday, September 18th 2017 10:04 UTC We’ve confirmed that all systems are back to normal as of 18 September 2017, 05:45 UTC. Our logs show that the incident started on 18 September 2017, 1:45 UTC. A subset of users in the East Australia region might have experienced significant delays in build processing during…


Issues with build definition page in Visual Studio Team Services – 09/16 – Mitigated

Final Update: Saturday, September 16th 2017 05:20 UTC The hotfix deployment has finished and we’ve confirmed that the functionality to load the build definition is working as expected as of 16 Sept 2017 04:02 UTC.We apologize to the customers who were affected by this issue. Sincerely,Rakesh Reddy Update: Saturday, September 16th 2017 01:52 UTC We…


Issues with Visual Studio Team Services – 09/14 – Mitigated

Final Update: Thursday, September 14th 2017 19:46 UTC VSTS customers in South Central experienced three distinct periods of impact where performance in the service was degraded (see the top graph). At time of peak impact ~1,500 interactive users would have seen intermittent slow commands. The underlying issue was related to one of the five web…


Performance Issues with Visual Studio Team Services – 09/14 – Mitigated

Final Update: Thursday, September 14th 2017 13:10 UTC VSTS in the South Central US region is now back in a healthy state. The incident was mitigated after the bad web tier node was taken out of the load balancer. We’ve confirmed that users are no longer seeing slowness or errors. We are working to analyze…


Issues with Visual Studio Team Services – 09/12 – Mitigated

Final Update: Tuesday, September 12th 2017 12:44 UTC The hotfix deployment has finished and we’ve confirmed that the functionality to increase the Cloud Load Test paid minutes to “No Limit” is working as expected as of 12 Sept 2017 12:30 UTC. We apologize to the customers who were affected by this issue.. Sincerely,Aman Update: Tuesday,…


Performance Issues with Visual Studio Team Services – 09/08 – Mitigated

Final Update: Friday, September 8th 2017 16:53 UTC There were 2 short spikes in customer impact lasting ~10 minutes each. We have been able to track down the cause of impact to high CPU on web tiers on Extension Management service that happened when new extensions were published. While the complete root cause is under…


Issues with deleting work items in Visual Studio Team Services – 09/07 – Mitigated

Final Update: Friday, September 8th 2017 21:45 UTC The fix has been deployed to all affected scale units. Users should not be facing issues with work item deletion in VSTS. Sincerely,Manohar Update: Friday, September 8th 2017 13:45 UTC Our DevOps team continues to integrate the hot-fix across our global platformsThe roll out is completed on…