Issues with Visual Studio Team Services – 12/2 – Resolved


Final update: Thursday, 3 December 2015 00:04 AM UTC

Our DevOps have applied the necessary mitigation steps to resolve the issue and bring the service back to normal.  As for initial root cause, some of the roles in the farm across our Scale Units hit a caching bug that was triggered by the earlier outage.  At this moment, we do not understand root cause of the caching bug, however we have taken the required dumps to do final root cause analysis and get to the bottom of the issue.  

We are truly sorry for the inconvenience this has caused our customers. 

Sincerely

VS Team Services Service Delivery Team

—–

Update: Wednesday, 2 December 2015 23:26 UTC

A change initiated by our partners in AAD that triggered this outage across VS Team Services Scale Units, has since been rolled back.  Unfortunately, this rollback did not mitigate issues with the service as we had hoped.  Our DevOps are actively working on understanding why the rollback did not resolve the issue and actively looking at various mitigations. 

Customer impact: A small percentage of our customers across Australia, Europe and US scale units might experience 500 errors when using the service.

We apologize for the inconvenience this is causing our customers.

Sincerely,

VS Team Services Service Delivery Team

———

Initial Update: Wednesday, 2 December 2015 22:02 UTC

We are actively investigating issues with Visual Studio Team Services. Some customers may experience issues signing in and accessing their VSTS accounts.  

We are working to resolve this issue and apologize for any inconvenience.

Sincerely,
VS Team Services Service Delivery Team