Issues with sign-in link on visualStudio.com – 01/22 – Mitigated

Final Update: Monday, January 22nd 2018 21:54 UTC We’ve confirmed that all systems are back to normal as of 21:49 UTC after the rollback was completed and caches were cleared. Sorry for any inconvenience this may have caused. Sincerely,Sri Harsha Update: Monday, January 22nd 2018 21:15 UTC Failed deployment rollback has introduced a condition where…


Performance Degradation in West Europe – 01/19 – Mitigated

Final Update: Friday, January 19th 2018 12:28 UTC We’ve confirmed that all systems are back to normal as of 2017-01-19 11:30 UTC. Our logs show the incident started on 2017-01-19 10:30 UTC and that during the 1 hour that it took to resolve the issue customers might have experienced slow and failed commands while using…


VSTS extensions in all regions may be failing with an Access Token error – 01/19 – Mitigated

Final Update: Friday, January 19th 2018 13:12 UTC The issue has been mitigated.• We performed a service update earlier today which we suspected was related to the Access Token error.• We have since rolled back the update, which has resolved the issue. We are continuing to investigate root cause. Sincerely,Niall Initial Update: Friday, January 19th…


Performance degradation in multiple regions – 01/16 – Mitigated

Final Update: Tuesday, January 16th 2018 23:56 UTC The issue has been mitigated.• Telemetry indicated that calls to SPS Extensions were failing due to client authentication errors.• We performed a certificate update earlier today which we suspected was related to the authentication error• While we were in the process of rolling back the certificate update,…


Performance Degradation in North Central US – 01/15 – Mitigated

Final Update: Monday, January 15th 2018 10:41 UTC We’ve confirmed that all systems are back to normal as of 15 January 2018 09:25 UTC. Our logs show that the incident started on 15 January 2018 08:55 UTC and that during the 30 minutes duration of the incident, customers hosted on our North Central US Region would have experienced slow response times while accessing the…


Investigating performance issues with VSTS in all regions – 01/11 – Mitigated

Final Update: Thursday, January 11th 2018 19:58 UTC We’ve confirmed that all systems are back to normal as of 2018/01/11 19:25 UTC. The change to how we route authorization traffic has been reverted and performance has returned to acceptable levels. The VSTS Engineering team continues to investigate the root cause behind this issue. Sincerely,Daniel Update:…


Performance issues with Visual Studio Team Services – 01/05 – Resolved

Update: Friday, 5 January 2018 08:30 UTC This is a retroactive communication that users would have observed slowness between 5th January 2018 6:50 UTC to 7:10 UTC while accessing their VSTS accounts in West Europe region. We are doing a root cause analysis to avoid it from re-occurring. Sincerely,Vamsi


Performance Degradation in West Europe – 01/04 – Mitigated

Final Update: Thursday, January 4th 2018 16:51 UTC We’ve confirmed that all systems are back to normal as of 16:10 UTC. Our logs show the incident started on 15:00 UTC and that during the 1 hour and 10 minutes that it took to resolve the issue customers might have experienced slow and failed commands while…


Postmortem – Intermittent Failures for Visual Studio Team Services on 14 Dec 2017

On 14 December 2017 we began to have a series of incidents with Visual Studio Team Services (VSTS) for several days that had a serious impact on the availability of our service for many customers (incident blogs #1 #2 #3). We apologize for the disruption these incidents had on you and your team. Below we…


Performance Degradation in West Europe – 12/22 – Mitigated

Final Update: Friday, December 22nd 2017 05:10 UTC We’ve confirmed that all systems are back to normal as of December 22nd 03:50 UTC. Our logs shows that the incident started on December 22nd 03:30 UTC and that during the 20 minutes that it took to resolve the issue customers experienced slow performance and failures while…