Postmortem: Hosted Build delays and Cloud Load Test errors on Visual Studio Team Services on 3/15

Customer Impact: For 132 minutes on 15 March 2017, users of Visual Studio Team Services experienced delays and some timeouts for their builds and load tests.  During the incident, builds for approximately 1,500 Hosted Build accounts did not start and eventually failed due to timeout errors. Also, tests for 40 Cloud Load Test accounts failed…


Hosted Build issues with Visual Studio Team Services – 03/29 – Resolved

Final Update: Thursday, 30 March 2017 01:57 UTC After investigating the issues with Hosted Build in the West Europe region, we decided to investigate additional regions which may also benefit from having an upgraded configuration database . During this investigation, we discovered that South Central US should be upgraded so that it would be able to serve…


Global issues with Visual Studio Team Services – 03/28 – Resolved

Update: Tuesday, 28 March 2017 23:36 UTC This is a retroactive notification to let you know about an incident that took place between 22:25 UTC and 22:58 UTC. A planned maintenance activity within our central SPS service started at approximately 22:00 UTC today.  This activity resulted in significant performance issues for SPS that impacted VSTS globally. The maintenance…


Connectivity issues with Visual Studio Team Services – 03/27 – Resolved

Update: Monday, 27 March 2017 10:45 UTC The service is healthy now. Following Visual Studio Team Services(VSTS) offerings via Azure Ibiza Portal (portal.azure.com ) were inaccessible to customers for a period of ~4hrs (6:08 UTC – 10:15 UTC): Team Services Accounts Team Projects Load Testing Continuous Delivery (Release Management) The cause of the issue has…


Login issues with Visual Studio Team Services – 03/24 – Resolved

Update: Saturday, 25 March 2017 00:30 UTC Azure Active directory team applied a mitigation to restore the Microsoft login service. Visual Studio Team Services login and account related features are now working as expected. We are working with our internal partner team to understand root cause and implement steps to avoid this issue in the…


Login issues with Visual Studio Team Services – 03/21 – Resolved

Update: Tuesday, 21 March 2017 20:17 UTC A fix has been deployed to restore the Microsoft login service. Visual Studio Team Services login and account related features are now working as expected. We are working with our internal partner team to understand root cause and implement steps to avoid this issue in the future. Next Update:…


Signin flow changes on Visual Studio Team Services – ADVISORY

We have recently implemented a change to the way users of Visual Studio Team Services sign in and sign out of our service.  The vast majority of users should see no impact from this change but we have found that some customers may experience issues signing into the product due to group policy based security…


Issues with Package Management in Visual Studio Team Services – 03/17 – Resolved

Update: Saturday, 18 March 2017 03:05 UTC We completed our investigation and we didn’t see any customer impact during this issue. We posted our initial communication as a caution after we saw an increase in operation cancelled exceptions. We did the further investigation and didn’t see any impact to customers due to these exceptions.  Sincerely,Bapayya Update:…


Issues with Pull Requests on Visual Studio Team Services – 03/17 – Mitigated

Update: Saturday, 18 March 2017 21:34 UTC We are providing additional details on workaround steps so that users can apply the mitigation steps based on the issues they are seeing for pull requests. We see that during yesterday’s incident some of our jobs that perform merges failed and left some pull requests in this state. We resolved…