Performance degradation in all regions – 11/16 – Mitigated


Final Update: Thursday, November 16th 2017 18:27 UTC

Customers in the Central US region are no longer experiencing slow web calls. The VSTS team continues to investigate the root cause and ensure that a fix is in place.
We apologize for any inconvenience this may have caused.

Sincerely,
Daniel


Update: Thursday, November 16th 2017 17:21 UTC

The fix has been successfully implemented in the West Europe region and we are no longer seeing any customer impact.

In Central US the VSTS Engineering team continues to work to deploy the fix. We are still seeing intermittent impact in this region, customers may experience a slow web experience.

  • Next Update: Before Thursday, November 16th 2017 18:30 UTC

Sincerely,
Daniel


Update: Thursday, November 16th 2017 16:32 UTC

A fix has been identified for the VSTS Web UI performance issues being experienced in the Central US and West Europe region. We expect the fix to be rolled out over the next 2-3 hours.

  • Next Update: Before Thursday, November 16th 2017 18:45 UTC

Sincerely,
Daniel


Update: Thursday, November 16th 2017 15:38 UTC

The broad impact across multiple regions we saw before has been resolved, and most regional VSTS scale units have returned to a healthy state. What we know is that the earlier outage was related to performance issues in our central SPS service. The SPS issue was due to problems with SQL performance, we've hotfixed one known issue and are evaluating other optimizations.

Customers hosted in Central US and West Europe are still seeing intermittent impact and may experience a slow web experience. The VSTS Engineering team is currently investigating to determine root cause and identify a fix.

  • Next Update: Before Thursday, November 16th 2017 17:45 UTC

Sincerely,
Daniel


Update: Thursday, November 16th 2017 13:34 UTC

During the impact period, our telemetry shows that the configuration database of our Shared Platform Service has experienced slowdowns which resulted in the customer impact across multiple Visual Studio Team Services regions. While the customer impact has subsided, the engineering team continues to investigate in order to understand the root cause of the slowdown.

  • Next Update: Before Thursday, November 16th 2017 15:45 UTC

Sincerely,
Ladislau


Initial Update: Thursday, November 16th 2017 12:51 UTC

We're investigating performance degradation across multiple Visual Studio Team Services regions.
While the customer impact is now gone, the engineering team has been engaged, and are investigating the root cause of the issue.

  • Next Update: Before Thursday, November 16th 2017 13:25 UTC

Sincerely,
Ladislau


Skip to main content