Customer impacting event in West Europe – 10/17 – Mitigated


Final Update: Wednesday, October 17th 2018 09:33 UTC

We’ve confirmed that all systems are back to normal as of 08:25 UTC. Our logs show the incident started on 07:40 UTC and that during the 45 minutes that it took to resolve the issue, a subset of customers in West Europe region would have observed slowness/failures while accessing AzureDevOps service. Sorry for any inconvenience this may have caused.

  • Root Cause: The initial symptoms indicate high database worker thread contention , we are working to find which process caused this.
  • Chance of Re-occurrence: Medium, as we understand the symptom and the process to mitigate it.
  • Lessons Learned: While we don't fully understand what caused the high database contention, but we do understand that the database was starved of resources due to which the requests from the web servers were slowed down. We failed over the database which mitigated the issue.
  • Incident Timeline: 45 minutes – 07:40 UTC through 08:25 UTC

Sincerely,
Aman


We are actively investigating performance issues with Azure Devops. Some customers may experience slower performance than usual while accessing their accounts.

  • Next Update: Before Wednesday, October 17th 2018 09:55 UTC

Sincerely,
RajeshKumar


Skip to main content