Issues with Visual Studio Team Services – 10/10 – Mitigated


Final Update: Tuesday, October 10th 2017 15:18 UTC

Our telemetry indicates that the issue is mitigated globally as of 14:18 UTC and all VSTS services are fully operational. We are currently working on fixing the underlying cause and we will be posting a root cause analysis of this soon. Apologies for the inconvenience that this may have caused you.

Sincerely,
Ladislau


Update: Tuesday, October 10th 2017 14:12 UTC

We have scaled out the number of web roles in the Shared Platform Service to improve the customer experience and help mitigate the issue. We are also in the process of deploying a build that contains a fix to the impacted library to remove the code path that was exercising the problematic regular expression. We’re starting to see the impact reducing but we’ll update with more information soon.

  • Next Update: Before Tuesday, October 10th 2017 16:15 UTC

Sincerely,
Ladislau


Update: Tuesday, October 10th 2017 12:12 UTC

Based on the information from the traces and the memory dumps, our working theory is that the high CPU consumption on the Shared Platform Service web roles is due to a code change that introduced a regular expression in one of the underlying libraries that our product took a dependency on. We are currently scaling out the web roles in Shared Platform Service as well as working on a fix the underlying library. We currently have no estimated time for resolution.

  • Next Update: Before Tuesday, October 10th 2017 14:30 UTC

Sincerely,
Ladislau


Update: Tuesday, October 10th 2017 10:20 UTC

We continue to investigate the incident. We have collected the traces, and memory dumps from one of the web tiers of the Shared Platform Service. The engineering team is currently going through the traces and the memory dumps in order to pinpoint the cause of the high CPU consumption that is believed to be the root cause of the incident. We currently have no estimated time for resolution.

  • Next Update: Before Tuesday, October 10th 2017 12:30 UTC

Sincerely,
Ladislau


Update: Tuesday, October 10th 2017 08:15 UTC

We continue to investigate the incident. The issue has been isolated to slow calls to our central Shared Platform Service due to high processor time on all web tiers.

Our telemetry indicates that the issue started at 07:00 UTC on 10th October 2017. Users might experience issues accessing the service.

  • Next Update: Before Tuesday, October 10th 2017 10:30 UTC

Sincerely,
Vamsi


Initial Update: Tuesday, October 10th 2017 07:21 UTC

A potentially customer impacting alert is being investigated. Triage is in progress and we will provide an update with more information.

  • Next Update: Before Tuesday, October 10th 2017 08:30 UTC

Sincerely,
Vamsi


Skip to main content