Performance Degradation in West Europe – 01/29 – Mitigated

Final Update: Monday, January 29th 2018 13:27 UTC We’ve confirmed that all systems are back to normal as of January 29th 12:55 UTC. Our logs show the incident started on January 29th 12:05 UTC and that during the 50 minutes that it took to resolve the issue some customers may have experienced slow and failed…


Users may be experiencing slow and failed commands in West Europe – 01/26 – Mitigated

Final Update: Friday, January 26th 2018 15:21 UTC We’ve confirmed that all systems are back to normal. We’re continuing to investigate why the change was made to ensure this doesn’t occur on other DBs. Sincerely,Daniel Update: Friday, January 26th 2018 14:48 UTC The Engineering team has narrowed down the cause of the slow and failed…


Performance Degradation in West Europe – 01/26 – Mitigated

Final Update: Friday, January 26th 2018 12:39 UTC We’ve confirmed that all systems are back to normal as of 12:05 UTC. Our logs show the incident started on 11:20 UTC and that during the 45 minutes that it took to resolve the issue customers might have experienced slow and failed commands while using Visual Studio…


Performance Degradation in South Central US – 01/24 – Mitigated

Final Update: Wednesday, January 24th 2018 18:25 UTC The performance issues have been resolved after the impacted database has returned to healthy state. We have an understanding of what caused this issue and are taking steps to prevent recurrence. The impact lasted for ~15 minutes and impacted 62 users at peak. During the incident window,…


Issues with sign-in link on visualStudio.com – 01/22 – Mitigated

Final Update: Monday, January 22nd 2018 21:54 UTC We’ve confirmed that all systems are back to normal as of 21:49 UTC after the rollback was completed and caches were cleared. Sorry for any inconvenience this may have caused. Sincerely,Sri Harsha Update: Monday, January 22nd 2018 21:15 UTC Failed deployment rollback has introduced a condition where…


Performance Degradation in West Europe – 01/19 – Mitigated

Final Update: Friday, January 19th 2018 12:28 UTC We’ve confirmed that all systems are back to normal as of 2017-01-19 11:30 UTC. Our logs show the incident started on 2017-01-19 10:30 UTC and that during the 1 hour that it took to resolve the issue customers might have experienced slow and failed commands while using…


VSTS extensions in all regions may be failing with an Access Token error – 01/19 – Mitigated

Final Update: Friday, January 19th 2018 13:12 UTC The issue has been mitigated.• We performed a service update earlier today which we suspected was related to the Access Token error.• We have since rolled back the update, which has resolved the issue. We are continuing to investigate root cause. Sincerely,Niall Initial Update: Friday, January 19th…


Performance degradation in multiple regions – 01/16 – Mitigated

Final Update: Tuesday, January 16th 2018 23:56 UTC The issue has been mitigated.• Telemetry indicated that calls to SPS Extensions were failing due to client authentication errors.• We performed a certificate update earlier today which we suspected was related to the authentication error• While we were in the process of rolling back the certificate update,…


Performance Degradation in North Central US – 01/15 – Mitigated

Final Update: Monday, January 15th 2018 10:41 UTC We’ve confirmed that all systems are back to normal as of 15 January 2018 09:25 UTC. Our logs show that the incident started on 15 January 2018 08:55 UTC and that during the 30 minutes duration of the incident, customers hosted on our North Central US Region would have experienced slow response times while accessing the…


Investigating performance issues with VSTS in all regions – 01/11 – Mitigated

Final Update: Thursday, January 11th 2018 19:58 UTC We’ve confirmed that all systems are back to normal as of 2018/01/11 19:25 UTC. The change to how we route authorization traffic has been reverted and performance has returned to acceptable levels. The VSTS Engineering team continues to investigate the root cause behind this issue. Sincerely,Daniel Update:…