Performance Degradation in South Central US – 12/12 – Mitigated

Final Update: Tuesday, December 12th 2017 21:15 UTC We’ve confirmed that all systems are back to normal as of December 12th 20:47 UTC. Our logs show the incident started on December 12th 20:32 UTC and that during the 15 minutes that it took to resolve the issue customers experienced slow performance and failures while using…


Issues while creating new accounts or accessing User profile and User hub in Visual Studio Team Services – 12/12 – Mitigated

Final Update: Tuesday, December 12th 2017 11:28 UTC The deployment has finished successfully and we’ve confirmed that all systems are back to normal as of December 12th 2017 11:20 UTC. Our logs show the incident started on December 12th 2017 04:40 UTC. During the course of this incident, customers would have observed the following issues….


Licensing Issues in multiple VSTS accounts – 12/07 – Mitigated

Final Update: Friday, December 8th 2017 16:38 UTC We’ve confirmed that all systems are back to normal as of December 8th 2017 12:12 UTC. Our logs show the incident started on November 29th 2017 12:19 UTC. and that during that time users may have noticed the access levels of users in their accounts are being…


Performance degradation impacting multiple VSTS locations – 12/06 – Mitigated

Final Update: Wednesday, December 6th 2017 15:19 UTC We’ve confirmed that all systems are back to normal as of 14:50 UTC. We apologize for any inconvenience this may have caused. During the incident, customers in West Europe, North Central US, Central US and South Central US may have seen an error page and/or slow commands….


Performance Degradation in West Europe – 12/05 – Mitigated

Final Update: Tuesday, December 5th 2017 15:22 UTC We’ve confirmed that all systems are back to normal as of 14:55 UTC. Our logs show the incident started on 14:30 UTC and that during the 25 minutes that it took to resolve the issue customers might have experienced slow and failed commands while using Visual Studio…


Performance degradation impacting multiple VSTS locations – 12/04 – Mitigated

Final Update: Monday, December 4th 2017 23:09 UTC The pressure on the central SPS service caused by ongoing upgrades is the reason for the current incident. We have temporarily paused the upgrades to mitigate the customer impact. Our engineers are actively working to understand the responsible call pattern is responsible for this increased load before…


Performance Degradation in South Central US – 12/01 – Mitigated

Final Update: Friday, December 1st 2017 20:16 UTC We’ve confirmed that all systems are back to normal as of Friday, December 1st 2017 19:20 UTC. Our logs show the incident started on Friday, December 1st 2017 18:50 UTC. and that during the 30 minutes that it took to resolve the issue. 1.6% of customers experienced…


Hosted Build delays in US region – 11/29 – Mitigated

Final Update: Wednesday, November 29th 2017 21:22 UTC We’ve confirmed that all systems are back to normal as of 20:55 UTC. Our logs show that customer impact occurred from 20:40 UTC – 20:55 UTC. At this time the backlog has fully cleared. We apologize for any inconvenience this may have caused. Sincerely,Colin Initial Update: Wednesday,…


Slow web UI performance in West Europe – 11/28 – Mitigated

Final Update: Tuesday, November 28th 2017 16:01 UTC We’ve confirmed that all systems are back to normal after the workaround was implemented in the West Europe region. We apologize for any inconvenience this may have caused. Sincerely,Daniel Update: Tuesday, November 28th 2017 15:11 UTC We continue to apply a workaround for the slow web UI…


User Hub may not be displaying all users in multiple regions – 11/28 – Mitigated

Final Update: Tuesday, November 28th 2017 20:50 UTC We’ve confirmed that all systems are back to normal. We deployed the fix successfully across all affected scale units. We apologize for any inconvenience this may have caused. Sincerely,Bapayya Update: Tuesday, November 28th 2017 17:10 UTC Our DevOps team continues to investigate issues with User Hub not…