Availability issues and Performance Degradation in multiple regions – 05/22 – Mitigated

Final Update: Tuesday, May 22nd 2018 17:08 UTC We’ve confirmed that all systems are back to normal as of 5/22 16:55 UTC. Our logs show the incident started on May 22 2018 14:55 UTC that during the 2 hours that it took to resolve the issue, users have experienced availability impact and severe performance degradation…


Investigating issues with MSA login failure in multiple regions – 05/22 – Mitigated

Final Update: Tuesday, May 22nd 2018 04:59 UTC We have confirmed that all systems are back to normal as of May 22nd 2018 03:40 UTC. Our logs show the incident started on May 22nd 2018 02:28 UTC. Our Partner team has identified preliminary cause to be related to a recent deployment task that impacted instances…


Performance Degradation in South Central US – 05/10 – Mitigated

Final Update: Thursday, May 10th 2018 06:56 UTC We’ve confirmed that all systems are back to normal as of 2018-05-10 05:05 UTC. Our logs show the incident started on 01:45 UTC Thursday May 10th. Sorry for any inconvenience this may have caused. Sincerely,Anmol Update: Thursday, May 10th 2018 04:54 UTC Our DevOps team continues to…


Invalid signature error while accessing Load Test Manager Test automation or Nuget packages in VSTS 05/03 – Investigating – Mitigated

Final Update: Friday, May 4th 2018 09:35 UTC Hotfix was completed across all regions . We’ve confirmed that all systems are back to normal as of 2018-05-04 03:30 UTC. Our logs shows that the incident started on 2018-04-11 19:50 UTC. Sorry for any inconvenience this may have caused. Sincerely,Zainudeen Update: Friday, May 4th 2018 03:40…


Performance Degradation in West Europe – 05/02 – Mitigated

Final Update: Wednesday, May 2nd 2018 09:53 UTC We’ve confirmed that all systems are back to normal as of May 2nd 2018 8:15 UTC . Root cause for this issue is still under investigation. Sorry for any inconvenience this may have caused. Sincerely,Zainudeen Update: Wednesday, May 2nd 2018 09:17 UTC The incident was auto mitigated…


Performance degradation in North Central US – 04/26 – Mitigated

Final Update: Thursday, April 26th 2018 15:30 UTC We’ve confirmed that all systems are back to normal as of 14:55 UTC. VSTS identity and authentication features that leverage Azure Active Directory (AAD) were slower than expected. We’ll be working with our AAD partner to understand root cause. We apologize for any inconvenience this may have…


Performance Degradation in South Central US – 04/24 – Mitigated

Final Update: Tuesday, April 24th 2018 23:56 UTC We’ve confirmed that all systems are back to normal. A recent change has been rolled back to mitigate the 404 errors users were experiencing in the South Central US region. We’ve also noticed slow performance for some users in this region, root cause for this issue is…


Postmortem: Global VSTS CI/CD outage due to service bus failure – 13 April 2018

Customer Impact:  On 13 April 2018, we had an incident which impacted CI/CD workflows in all data centers.  This was caused by a global Service Bus instance, which we use to orchestrate CI/CD workflows, to be unavailable due to authentication errors.  Users reported that their CI/CD pipelines were stuck at various stages including releases which…


Release Management performance degradation in West Europe – 04/18 – Mitigated

Final Update: Wednesday, April 18th 2018 13:52 UTC We’ve confirmed that all systems are back to normal as of 12:30 UTC. While the issue has self-healed, during the incident we were able to collect key diagnostic information from our web front-ends that the team is actively reviewing in order to under the root cause of…