Performance issues with Visual Studio Team Services in West Europe – 02/15 – Resolved

Final Update: Thursday, 15 February 2018 19:50 UTC We have been able to isolate the issue to a network hardware power event in Azure West Europe data center. We will work with our partners in Azure to understand the detailed root cause analysis.  More details on the power event can be found at: https://azure.microsoft.com/en-us/status/history Sincerely,Sri Harsha…


Postmortem: Performance Issues and failures in VSTS West Europe – 7 February 2018

Customer Impact On 7 February 2018 we had an incident which impacted users in our Western European scale unit. During this time, users experienced slow performance and 503 errors (service unavailable) when interacting with VSTS services. Close to 5,000 users were impacted at the peak of the incident. The incident lasted for two and a…


Investigating issues where teams can be incorrectly added by branch policies to pull requests – all regions – 02/13 – Mitigated

Final Update: Tuesday, February 13th 2018 05:55 UTC Upon further investigation we’ve identified that the impact is not wide spread and it is limited to be with West US 2 region where our internal accounts are hosted. We have a hot fix being deployed to address this issue. We apologize for any inconvenience this issue…


Issues with Packaging in South Brazil – 02/08 – Mitigated

Final Update: Thursday, February 8th 2018 06:07 UTC We’ve confirmed that all systems are back to normal as of 2018-02-08 05:30 UTC. Our logs indicates that failures started on 2018-02-07 19:30 UTC .We have an understanding of what caused this issue and have taken steps to prevent recurrence. Sorry for any inconvenience this may have…


Performance Degradation in West Europe – 02/07 – Mitigated

Final Update: Wednesday, February 7th 2018 13:26 UTC We’ve confirmed that all systems are back to normal as of 12:40 UTC. Our logs show the incident started on 10:10 UTC and that during that time customers may have experienced connection issues to the VSTS service in the West Europe. Sorry for any inconvenience this may…


File Downloading Issues in West Europe – 02/05 – Mitigated

Final Update: Monday, February 5th 2018 18:19 UTC We’ve confirmed that all systems are back to normal as of February 5th 2018 17:57 UTC. Our logs show the incident started on February 2nd 2018 18:03 UTC and that during those 71 hours and 55 minutes that it took to resolve the issue 0.28% of customers…


Investigating issues with VSTS – 02/03 – Mitigated

Final Update: Saturday, February 3rd 2018 04:57 UTC We’ve confirmed that all systems are back to normal as of February 3rd 2018 04:45 UTC.We rolled back and invalid certificate update impacting a few regions. During the this incident, users would have experienced connection failure errors (404 errors) when trying to reach the site. Sorry for…


Performance Degradation in West Europe – 01/29 – Mitigated

Final Update: Monday, January 29th 2018 13:27 UTC We’ve confirmed that all systems are back to normal as of January 29th 12:55 UTC. Our logs show the incident started on January 29th 12:05 UTC and that during the 50 minutes that it took to resolve the issue some customers may have experienced slow and failed…


Users may be experiencing slow and failed commands in West Europe – 01/26 – Mitigated

Final Update: Friday, January 26th 2018 15:21 UTC We’ve confirmed that all systems are back to normal. We’re continuing to investigate why the change was made to ensure this doesn’t occur on other DBs. Sincerely,Daniel Update: Friday, January 26th 2018 14:48 UTC The Engineering team has narrowed down the cause of the slow and failed…


Performance Degradation in West Europe – 01/26 – Mitigated

Final Update: Friday, January 26th 2018 12:39 UTC We’ve confirmed that all systems are back to normal as of 12:05 UTC. Our logs show the incident started on 11:20 UTC and that during the 45 minutes that it took to resolve the issue customers might have experienced slow and failed commands while using Visual Studio…