User performance degradation across multiple regions – 10/08 – Mitigated


Final Update: Tuesday 16 October 2018 11:19 UTC

Summary of impact: Between 11:40 and 14:10 UTC on 08 Oct 2018, a subset of Azure DevOps customers may have experienced error messages, latency, and/or authentication issues. The service may have been down for a subset of customers between 12:35 and 13:05 UTC.

In addition, a small subset of Azure Portal customers may have experienced issues logging into the Azure portal.

Preliminary root cause: Engineers determined that a network device experienced a hardware fault and that network traffic was not automatically rerouted.

Mitigation: Engineers took the faulty network device out of rotation and rerouted network traffic to mitigate the issue. Azure DevOps took a while to recover and reestablish all the connections, after that network traffic re-routing.

Sincerely,
Pedro


Update: Monday, October 8th 2018 15:00 UTC

We’ve confirmed that all systems are back to normal as of 14:10 UTC. Our logs show the incident started on 12:25 UTC and that during the 2 hours and 45 minutes that it took to resolve the issue. 2% of customers experienced outage and performance degradation in the West Europe region. Sorry for any inconvenience this may have caused.

  • Root Cause: We're still investigating the Root Cause.
  • Chance of Re-occurrence: Medium
  • Incident Timeline: 2 hours & 45 minutes – 12:25 UTC through 14:10

Sincerely,
Pedro


Update: Monday, October 8th 2018 13:51 UTC

Our DevOps team continues to investigate issues with slow and failed commands in several regions. North Central US and Central US are now mitigated, West Europe is still impacted. Root cause is not fully understood at this time. The problem began at 2018/10/08 12:54 UTC. We currently have no estimated time for resolution.

  • Next Update: Before Monday, October 8th 2018 15:00 UTC

Sincerely,
Alberto


Initial notification: Monday, October 8th 2018 13:03 UTC

We're investigating User performance degradation on our main authentication service across multiple regions

  • Next Update: Before Monday, October 8th 2018 13:35 UTC

Sincerely,
Pedro


Skip to main content