Performance Issues with Visual Studio Team Services – 04/27 – Mitigated

Final Update: Thursday, April 27th 2017 21:01 UTC We’ve confirmed that all systems are back to normal as of 4/27 19:57. Users may have experienced degraded performance during this time. Sorry for any inconvenience this may have caused. Incident Timeline: 0 hours & 14 minutes – 4/27 19:43 UTC through 4/27 19:57 Sincerely,Daniel Initial Update: Thursday,…


Issues with Visual Studio Team Services – 04/25 – Mitigated

Final Update: Wednesday, April 26th 2017 01:04 UTC We’ve confirmed that all systems are back to normal as of 04-24-2017 23:30 UTC. The hotfix is released to all the impacted regions and users should now have no failures while performing gated check-ins. Root Cause: The issue surfaced due to bug in the sprint deployment which…


Long Build queue times on Visual Studio Team Services Hosted Build – 04/25 – Mitigated

Final Update: Tuesday, April 25th 2017 14:53 UTC Our investigation indicated that there were issues during build machine allocation at our partners in Azure. The issue got self healed and from our telemetry we see that the builds are now processing normally. We will be working with our partners from Azure to understand the underlying…


Issues with Marketplace – 04/25 – Mitigated

Final Update: Tuesday, April 25th 2017 13:11 UTC We’ve identified that the SQL service running on the reporting server that hosts the VSGallery DB replica was not restarting. We have also noticed that the reporting server was inaccessible post restart of the server. We have worked with MSDN China team to get the server back…


Performance issues with Visual Studio Team Services – 04/24 – Resolved

Final Update: Monday, 24 April 2017 19:41 UTC We’ve confirmed that all systems are back to normal as of 04-24-2017 18:53 UTC. Our logs show the incident started on 04-24-2017 18:40. Customers should now have no issues while accessing their Visual Studio Team Services accounts. Initial investigations suggests that the issue was due to high call rates on…


Connectivity issues with Marketplace – 04/22 – Resolved

Update: Saturday, 22 April 2017 02:44 UTC Users should now be able to access the market place without any issues.  During this impact window (Start Time: 2017/04/22 00:00 UTC  (2017/04/21 17:00 Pacific) and End Time: 2017/04/22 00:10 UTC  (2017/04/21 17:10 Pacific)) users might have faced connectivity issues on Market and might have got 500 errors. This issue was mitigated once…


Connectivity issues with Visual Studio Team Services – 04/22 – Resolved

Update: Saturday, 22 April 2017 03:41 UTC Root cause has been isolated to a missing registry entry for the builds using legacy OAuth validation. To address this issue we deployed a configuration change and the system is now working as expected. Users should now have no issues to connect to Visual Studio Team Services through CI/CD builds….


404 errors on Visual Studio Team Services – 04/21 – Resolved

Update: Friday, 21 April 2017 19:56 UTC The issue has been mitigated after the hotfix has been deployed to our Shared Platform Services (SPS). This was caused due to a forwards-compatibility issue with a feature where a service that hasn’t yet deployed the latest binaries interacts with SPS that is running on latest update, a…


Performance issues with Visual Studio Team Services – 04/21 – Resolved

Update: Friday, 21 April 2017 06:49 UTC Issue has been isolated to one of the web role instances that was unhealthy. All the requests going to this instance were failing. To address this issue we have collected necessary dumps and have pulled the role instance out of load balancer. Visual Studio Team Services is now…


Performance issues with Hosted Build Service – 04/18 – Resolved

Update: Tuesday, 18 April 2017 15:57 UTC Customers should no longer be seeing longer build queue times now. During the course of the incident, we identified that customers in West Europe and Central US regions observed longer than usual build queue times. The symptom for this issue to occur was high processor utilization on the…