Customer Impacting event in Azure DevOps in MultiRegions – 11/09 – Mitigated


Final Update: Friday, November 9th 2018 18:50 UTC

We’ve confirmed that all systems are back to normal as of 2018/11/08 17:30 UTC. Our logs show the incident started on 2018/11/09 09:08 UTC and that during the 8 hours and 22 minutes that it took to resolve the issue.

  • Root Cause: The failure was due to background activity which caused cache invalidations
  • Chance of Re-occurrence: Low
  • Lessons Learned: We are working both minimizing resource-intensive activities in our post-deployment steps, and are also working targeting monitors specifically to detect post-deployment issues in the future.

Thanks,
Ian


Update: Friday, November 9th 2018 15:55 UTC

Our DevOps team continues to investigate issues impacting several regions - West Europe, North and Central US. Root cause is not fully understood at this time, however we are progressing with the investigation on 1-2 hypothesis. The problem began at 8:45am UTC, was mitigated at 11:15, and recurred at 12:20pm UTC. We currently have no estimated time for resolution.

  • Next Update: Before Friday, November 9th 2018 20:00 UTC

Sincerely,

Alberto.


Update: Friday, November 9th 2018 12:57 UTC

Our Azure DevOps team continues to investigate issues on Multiple Regions. We are continuing to investigate the Root Cause.
We currently have no estimated time for resolution.

  • Next Update: Before Friday, November 9th 2018 16:00 UTC

Sincerely,
Ariel Jacob


Update: Friday, November 9th 2018 10:25 UTC

Our Azure DevOps team continues to investigate issues on Multiple scale units. Root cause is not fully understood at this time .The problem began at November 9th 2018 8:50 UTC. We currently have no estimated time for resolution.

  • Next Update: Before Friday, November 9th 2018 13:30 UTC

Sincerely,
Priyanka


We are actively investigating performance issues with Azure DevOps in MultiRegions. Some customers may experience slower performance than usual while accessing their accounts.

Next Update: Before Tuesday, November 9th 2018 10:30 UTC

Sincerely,
Priyanka.


Skip to main content