Releases might not be completing successfully in Central US region – 10/25 – Mitigated

Final Update: Thursday, October 25th 2018 20:31 UTC We believe the issue is mitigated. We’ve already discussed repair items (on the incident bridge) that should prevent a recurrence and will be filing these against the owning teams shortly. Sincerely,Kishore Update: Thursday, October 25th 2018 19:16 UTC We have identified the problem and are in the…


Customer Impacting Event in Cental US – Mitigated

Final Update: Thursday, October 25th 2018 09:24 UTC We’ve confirmed that all systems are back to normal as of 7:50 UTC. Our logs show the incident started on 6:15 UTC and that during the 1 hour and 35 minutes that it took to resolve the issue, customers might have observed slowness in Central US.Sorry for…


Hosted build delays in Central US – 10/22 – Mitigated

Final Update: Monday, October 22nd 2018 23:22 UTC We’ve confirmed that all systems are back to normal as of 22:50 UTC. Our logs show the incident started at 19:40 and that during the 3 hours and 10 minutes that it took to resolve the issue, an unknown number of customers experienced build delays upwards of…


Customer impacting event in West Europe – 10/17 – Mitigated

Final Update: Wednesday, October 17th 2018 09:33 UTC We’ve confirmed that all systems are back to normal as of 08:25 UTC. Our logs show the incident started on 07:40 UTC and that during the 45 minutes that it took to resolve the issue, a subset of customers in West Europe region would have observed slowness/failures…


Postmortem: Azure DevOps Service Outages in October 2018

Earlier this month, Azure DevOps experienced several significant service outages, for which we are deeply sorry. As with every significant live site incident, we have completed a detailed root cause analysis for these. Due to the proximity of these incidents and common underlying causes, we wanted to share the details with you to ensure that…


Authentication issues in all regions – 10/11 – Resolved

Final Update: Saturday, October 13th 2018 00:19 UTC Our partners in Azure have posted more details about the issue on the Azure status page. You can find it titled ‘Azure Portal and DevOps Accessibility’ dated 10/11 on https://azure.microsoft.com/status/history. For your convenience, pasting it below: Summary of impact: Between 15:59 UTC and 18:35 UTC on 11…


Azure DevOps Impact due to authentication service outage – 10/11 – Mitigated

Final Update: Thursday, October 11th 2018 14:06 UTC We’ve confirmed that all systems are back to normal as of 13:40 UTC. Our logs show the incident started on 12:10 UTC and that during the 1 hour and 30 minutes that it took to resolve the issue. 7.5% of customers experienced performance degradation and outages across…


User performance degradation across multiple regions – 10/08 – Mitigated

Final Update: Tuesday 16 October 2018 11:19 UTC Summary of impact: Between 11:40 and 14:10 UTC on 08 Oct 2018, a subset of Azure DevOps customers may have experienced error messages, latency, and/or authentication issues. The service may have been down for a subset of customers between 12:35 and 13:05 UTC. In addition, a small subset…


Investigating partial or no search results while using Azure DevOps Search functionality across all regions – 10/08 – Mitigated

Final Update:Tuesday, October 9th 2018 15:10 UTC We’ve confirmed that all systems are back to normal as of 15:10 UTC. Sorry for any inconvenience this may have caused. Chance of Re-occurrence: Low Lessons Learned: We are working both minimizing resource-intensive activities in our post-deployment steps, and are also working targeting monitors specifically to detect post-deployment issues in the…