VSTS Impact due to Datacenter Outage in South Central US – 09/05 – Resolved


Update: Thursday, September 6th 00:50 UTC

  • Storage accounts impacting VSTS services have been recovered
  • We have validated that storage errors and performance of key workflows is back to normal 
  • We will continue to monitor

Thanks, Tom


Update: Wednesday, September 5th 20:30 UTC

  • We are seeing overall improvement in performance of VSTS features for users in South Central.  We are continuing to monitor performance closely.
  • Azure mitigation for storage accounts hosted in South Central is ongoing.  Some customers may continue to see impact to Artifacts and Packaging services.
  • For more details regarding South Central US resources recovery please refer Azure Status Dashboard : https://status.azure.com

Next Update:  Thursday, September 6th 00:30 UTC

Thanks, Tom


Update: Wednesday, September 5th 18:30 UTC

  • VSTS customers hosted in South Central US continue to be impacted by this outage. Users may experience general slowness or failures using the product, in particular using version control and work item tracking.
  • Customers using Artifacts Services may be impacted as storage issues in South Central US are being mitigated.
  • For more details regarding South Central US resources recovery please refer Azure Status Dashboard : https://status.azure.com

Next Update:  Wednesday, September 5th 20:30 UTC

Thanks,
Mauro


Update: Wednesday, September 5th 13:40 UTC

  • South Central US is recovered for the most part, with the exception of Artifacts service in that region facing storage issues, as the relevant storage clusters are not fully recovered.
  • The intermittent failures faced by Extension search powered by  VS MarketPlace have been mitigated.
  • For more details regarding South Central US resources recovery please refer Azure Status Dashboard : https://status.azure.com

Next Update:  Wednesday, September 5th 17:30 UTC

Thanks,
Aman


Update: Wednesday, September 5th 09:50 UTC

  • Recovery efforts continue.
  • South Central US region is partially recovered, a subset of customers will be unable to access their accounts or face performance issues. We are working to mitigate those failures.
  • Extension search powered by VS MarketPlace service will face intermittent failures while loading, we are in the process of upgrading the impacted SQL Azure database to mitigate the issue.
  • For more details regarding South Central US resources recovery please refer Azure Status Dashboard : https://status.azure.com

Next Update:  Wednesday, September 5th 13:30 UTC

Thanks,
Aman


Update: Wednesday, September 5th 04:55 UTC 

Mitigation efforts continue, as we work on getting the Azure resources in South Central US online.
For more details about the issue please refer Azure Status Dashboard : https://status.azure.com

Next Update: Wednesday, September 5th 09:00 UTC

Thanks,
Aman


Update: Wednesday, September 5th 01:00 UTC

Mitigation steps continue for Azure resources in South Central.  VSTS services in South Central continue to be unavailable at this time.   

Next Update: Wednesday, September 5th 04:00 UTC 

Thanks,
Tom


Update: Tuesday, September 4th 22:08 UTC

A subset of VSTS users in South Central US may notice that their organizations are accessible, but will still encounter issues with different parts of the service. The mitigation efforts are still underway. Power has been restored to the datacenter along with software load balancers for the Storage scale units. Azure engineers are currently working to restore complete connectivity to Azure storage and all dependent services including VSTS.

More updates on the recovery efforts can be found at https://azure.microsoft.com/en-us/status or on twitter at https://twitter.com/AzureSupport

Next Update: Wednesday, September 5th 00:00 UTC

Thanks,
Sri Harsha


Update: Tuesday, September 4th 20:15 UTC

Azure engineers still continue to work on mitigation efforts needed to bring all the resources online in the South Central US data center. From VSTS perspective, some users may start to see some signs of recovery, but mitigation efforts are still ongoing.

VSTS Organizations hosted in South Central US region may not be able to access the service. Organizations hosted outside of this impacted region may still see some impact in parts of the service such as releases, dashboards and build submission using Mac pools.

Next Update: Tuesday, September 4th 22:15 UTC

Thanks,
Sri Harsha


Update: Tuesday, September 4th 19:13 UTC

Azure engineers are working to restore normal operations in the affected data center. Impact for VSTS users remain the same. We don't have an ETA at the moment, unfortunately. More up to date status on the Azure incident can be found at https://azure.microsoft.com/en-us/status

Work Around: None

Next Update: Tuesday, September 4th 20:15 UTC

Thanks,
Tom


Update: Tuesday, September 4th 18:30 UTC

We still continue to work with our partners in Azure to restore availability to the VSTS services in South Central US region. We don't have an ETA at the moment. More up to date status on the Azure incident can be found at https://azure.microsoft.com/en-us/status

Work Around: None

Next Update: Tuesday, September 4th 19:15 UTC

Thanks,
Tom


Update: Tuesday, September 4th 17:45 UTC

Our partners in Azure are continuing to bring services online in South Central US region. Organizations hosted in that region will continue to be unavailable until the current issues are fixed. Additionally, customers with organizations outside of the South Central US region may also be experiencing impact with their CI/CD workflows, dashboards due to some internal infrastructure dependencies.

Work Around: None

Next Update: Tuesday, September 4th 18:30 UTC

Thanks,
Tom


Update: Tuesday, September 4th 17:00 UTC

We are currently impacted by a datacenter outage in South Central US which is causing significant impact across many VSTS services.
We are attempting to mitigate the issue with the dashboards from our side, and working closely with Azure to mitigate the issue
Azure is currently posting updates for this issue at https://azure.microsoft.com/en-us/status/ with the current title of "Multiple Services - South Central US - Mitigating"

Work Around: None

Next Update: Tuesday, September 4th 17:45 UTC

Thanks,
Dexter


Update: Tuesday, September 4th 16:25 UTC

We are currently impacted by a datacenter outage in South Central US which is causing significant impact across many VSTS services, some known issues are listed below

Known Issues:

  • Users located in South Central US will be unable to access their account until the underlying issue is resolved.
  • Most US-based customers will be unable to use Release Management
  • Dashboards with 3rd party widgets will fail to load
  • Mac Pools are impacted across all regions
  • Customers may also experience other slow or failed workflows

Azure is currently posting updates for this issue at https://azure.microsoft.com/en-us/status/ with the current title of "Multiple Services - South Central US - Investigating"

Work Around: None

Next Update: Tuesday, September 4th 17:00 UTC

Thanks,
Dexter


Update: Tuesday, September 4th 12:35 UTC

We continue to investigate issues in our services impacting all regions. The underlying issue seems to be a datacenter cooling outage in South Central US which is impacting all services in a specific part of the datacenter.

Users located in South Central US will be unable to access their account until the underlying issue is resolved. Other regions may experience high failure rates and load times due to dependencies in South Central US. We are working to mitigate the impact for users outside of South Central US to the best of our abilities.

Work Around: None

Next Update: Tuesday, September 4th 16:35 UTC

Thanks,
Dexter


We are actively investigating performance issues with Visual Studio Team Services. Customers may experience issues while accessing their Visual Studio Team Services accounts.

Sincerely,
Kalpana

 


Skip to main content