Customer impacting event in West Europe – 10/17 – Mitigated

Final Update: Wednesday, October 17th 2018 09:33 UTC We’ve confirmed that all systems are back to normal as of 08:25 UTC. Our logs show the incident started on 07:40 UTC and that during the 45 minutes that it took to resolve the issue, a subset of customers in West Europe region would have observed slowness/failures…


Postmortem: Azure DevOps Service Outages in October 2018

Earlier this month, Azure DevOps experienced several significant service outages, for which we are deeply sorry. As with every significant live site incident, we have completed a detailed root cause analysis for these. Due to the proximity of these incidents and common underlying causes, we wanted to share the details with you to ensure that…


Authentication issues in all regions – 10/11 – Resolved

Final Update: Saturday, October 13th 2018 00:19 UTC Our partners in Azure have posted more details about the issue on the Azure status page. You can find it titled ‘Azure Portal and DevOps Accessibility’ dated 10/11 on https://azure.microsoft.com/status/history. For your convenience, pasting it below: Summary of impact: Between 15:59 UTC and 18:35 UTC on 11…


Azure DevOps Impact due to authentication service outage – 10/11 – Mitigated

Final Update: Thursday, October 11th 2018 14:06 UTC We’ve confirmed that all systems are back to normal as of 13:40 UTC. Our logs show the incident started on 12:10 UTC and that during the 1 hour and 30 minutes that it took to resolve the issue. 7.5% of customers experienced performance degradation and outages across…


User performance degradation across multiple regions – 10/08 – Mitigated

Final Update: Tuesday 16 October 2018 11:19 UTC Summary of impact: Between 11:40 and 14:10 UTC on 08 Oct 2018, a subset of Azure DevOps customers may have experienced error messages, latency, and/or authentication issues. The service may have been down for a subset of customers between 12:35 and 13:05 UTC. In addition, a small subset…


Investigating partial or no search results while using Azure DevOps Search functionality across all regions – 10/08 – Mitigated

Final Update:Tuesday, October 9th 2018 15:10 UTC We’ve confirmed that all systems are back to normal as of 15:10 UTC. Sorry for any inconvenience this may have caused. Chance of Re-occurrence: Low Lessons Learned: We are working both minimizing resource-intensive activities in our post-deployment steps, and are also working targeting monitors specifically to detect post-deployment issues in the…


Errors accessing Work Items via Client OM event in all regions – 10/03 – Mitigated

Final Update: Thursday, October 4th 2018 05:53 UTC We’ve confirmed that all systems are back to normal as of 4th October 04:02 UTC. The hotfix deployment is completed on all the rings. Our logs show that the incident started on 3rd October 4:40 UTC. Sorry for any inconvenience this may have caused. Root Cause: We…