Issues with Visual Studio Team Services – 08/02 – Mitigated


Final Update: Thursday, August 3rd 2017 04:50 UTC

After working with our DNS service provider, we discovered a bad DNS server in Tokyo which was causing users in Asia to intermittently be unable to resolve any address off of "visualstudio.com" including their VSTS accounts. To mitigate the issue, the DNS server was removed from rotation and we have confirmed that after forcefully flushing DNS, users were no longer having the issue. Users may continue to hit the issue over the next hour as their DNS gets refreshed.

Sorry for any inconvenience this may have caused.

  • Root Cause: The failure was due to a bad DNS server in Tokyo
  • Chance of Re-occurrence: Low - Root cause is now understood.
  • Lessons Learned: We will work with our DNS service provider to improve their monitoring of DNS as well as look at ways we can proactively identify intermittent DNS issues.
  • Incident Timeline: We are still working with our DNS service provider to identify the exact impact timelines of when the server went bad. The server was removed from rotation at 04:32 UTC.

Sincerely,
Eric


Update: Thursday, August 3rd 2017 02:41 UTC

We have begun to receive more reports of the issue and are investigating.

  • Next Update: Before Thursday, August 3rd 2017 04:45 UTC

Sincerely,
Eric


Final Update: Wednesday, August 2nd 2017 18:40 UTC

We have confirmation from multiple users that the issue is no longer occurring so we are considering the incident mitigated as of now. We will continue investigating root cause in partnership with our DNS service provider.

Sincerely,
Eric


Initial Update: Wednesday, August 2nd 2017 17:11 UTC

  • We are currently investigating reports of some customers in the Asia region experiencing intermittent DNS issues accessing www.visualstudio.com and their accounts.
  • We are in the process of engaging our DNS service provider and an update will be provided with more information.
  • Next Update: Before Wednesday, August 2nd 2017 19:15 UTC

Sincerely,
Eric


Skip to main content