Performance Degradation in South Central US – 02/22 – Mitigated

Final Update: Thursday, February 22nd 2018 19:28 UTC We’ve confirmed that all systems are back to normal as of 18:50 UTC. We have identified the root cause to be a regression introduced by a recent change and a fix is in the process of rolling out. Incident Timeline: 45 minutes – 18:05 UTC through 18:50…


Work Item Search issues in multiple regions – 02/22 – Mitigated

Final Update: Thursday, February 22nd 2018 19:40 UTC We have mitigated the issue by disabling a feature flag. Users should be able to see results for their work item search queries. There are some known performance issues associated with disabling this feature flag, so users may notice minor degradation in doing their queries. We are…


ADVISORY – East US region unavailable for Load Test users – RESOLVED

Final Update: Thursday, February 22 2018 15:40 UTC We have worked with our partners in Azure to resolve this issue in this region. East US region is now available as an option for users to execute Load tests. Sincerely, Sri Harsha Initial Update: Wednesday, February 21, 2018 14:49 UTC This is to make you aware…


Performance Degredation in South Central US – 02/21 – Mitigated

Final Update: Wednesday, February 21st 2018 13:20 UTC We’ve confirmed that all systems are back to normal as of February 21st, 13:13 UTC. Our logs show the incident started on February 21st, 2:30 UTC and that during the 11 hours and 13 minutes that it took to resolve the issue. A subset of Customers may…


Performance Degradation in South Central US – 02/20 – Mitigated

Final Update: Tuesday, February 20th 2018 22:54 UTC We’ve confirmed that all systems are back to normal as of 21:20 UTC. Initial investigation shows an issue with a database in that scale unit. We are engaging our partners in SQL Azure to understand the root cause. Users should not face any more issues or errors…


Performance issues with Visual Studio Team Services in West Europe – 02/15 – Resolved

Final Update: Thursday, 15 February 2018 19:50 UTC We have been able to isolate the issue to a network hardware power event in Azure West Europe data center. We will work with our partners in Azure to understand the detailed root cause analysis.  More details on the power event can be found at: https://azure.microsoft.com/en-us/status/history Sincerely,Sri Harsha…


Preliminary Postmortem: Performance Issues and failures in VSTS West Europe – 7 February 2018

Edit February 26, 2018: We have just posted an updated and complete postmortem here: https://blogs.msdn.microsoft.com/vsoservice/?p=16295 Customer Impact On 7 February 2018 we had an incident which impacted users in our Western European scale unit. During this time, users experienced slow performance and 503 errors (service unavailable) when interacting with VSTS services. Close to 5,000 users were…


Investigating issues where teams can be incorrectly added by branch policies to pull requests – all regions – 02/13 – Mitigated

Final Update: Tuesday, February 13th 2018 05:55 UTC Upon further investigation we’ve identified that the impact is not wide spread and it is limited to be with West US 2 region where our internal accounts are hosted. We have a hot fix being deployed to address this issue. We apologize for any inconvenience this issue…


Issues with Packaging in South Brazil – 02/08 – Mitigated

Final Update: Thursday, February 8th 2018 06:07 UTC We’ve confirmed that all systems are back to normal as of 2018-02-08 05:30 UTC. Our logs indicates that failures started on 2018-02-07 19:30 UTC .We have an understanding of what caused this issue and have taken steps to prevent recurrence. Sorry for any inconvenience this may have…