HTTP 500/503 Errors in South Central US – 06/13 – Mitigated


Final Update: Wednesday, June 13th 2018 19:15 UTC

We’ve confirmed that all systems are back to normal as of 6/13 18:30 UTC after Azure has applied mitigation to the underlying Storage availability issue in South Central US region. More details about this incident can be found on Azure status portal.

We will update this post with a pointer to a detailed RCA as soon as it is available.

Sincerely,
Sri Harsha


Update: Wednesday, June 13th 2018 18:41 UTC

Underlying issue has been identified by storage team and mitigations are underway. The problem began at 15:55UTC. Rough estimate for mitigation is within next hour. Impacted services include, but are not limited to, Version Control, Build, Artifacts, WIT, and MySubscriptions - Product Search

  • Work Around: None
  • Next Update: Before Wednesday, June 13th 2018 19:45 UTC

Sincerely,
Samuli


Update: Wednesday, June 13th 2018 17:37 UTC

Our DevOps team together with the storage team continues to investigate issues with multiple VSTS services. This includes, but is not limited to Version Control, Build, Artifacts, and MySubscriptions. The issues are due to problems in our underlying storage. The problem began at 15:55UTC.

  • Work Around: None
  • Next Update: Before Wednesday, June 13th 2018 18:45 UTC

Sincerely,
Samuli


Initial Update: Wednesday, June 13th 2018 16:37 UTCWe're investigating increased numbers of 500 and 503 errors in South Central US.

  • This seems to primarily impact Version Control
  • This seems to be due to an underlying storage issue
  • We are investigating further
  • Next Update: Before Wednesday, June 13th 2018 17:10 UTCSincerely,Dexter

Skip to main content