Issues with Visual Studio Team Services Hosted Build – 08/16 – Mitigated

Final Update: Wednesday, August 16th 2017 23:27 UTC We see build queue times have returned to normal levels after applying a fix.Root Cause: A bug was found with a cleanup procedure that caused builds to queue longer than expected. A fix has been put in place and we do not expect this issue to reoccur….


Issues with Visual Studio Team Services – 08/16 – Mitigating

Update: Wednesday, August 16th 2017 18:25 UTC During the process of investigation it has been found that the time required to deploy the fix is going to take longer than originally expected. Our revised estimated time for resolution is now August 17th 2017 12:30 UTC. Work Around: Affected users can use the Azure Powershell (v2)…


Issues with Visual Studio Team Services – 08/15 – Mitigated

Final Update: Tuesday, August 15th 2017 21:35 UTC We’ve confirmed that all systems are back to normal as of 21:15 UTC. ~1,213 customers experienced feeds and packaging issues in the South Central US, West Europe, and Australia Southeast regions. We apologize for any inconvenience this may have caused. Incident Timeline: 1 hours and 40 minutes –…


Advisory: Git vulnerability with submodules

The Git community has disclosed a serious security vulnerability in Git that can lead to arbitrary code execution. This has been assigned CVE 2017-1000117. The Visual Studio Team Services (VSTS) team takes security issues very seriously.  We encourage all users to update their Git clients as soon as possible to address this issue. If you’re…


Performance Issues with Visual Studio Team Services – 08/15 – Resolved

Update: Tuesday, 15 August 2017 04:04 UTC We’ve confirmed that the Database SKU upgrade helped in mitigating the impact at Tuesday, August 15th 2017 03:05 UTC. However, a fix is being rolled out to completely mitigate the DB contention. All necessary logs have been collected and we are closely working with our partners in SQL Azure to…


Issues with Visual Studio Team Services – 08/11 – Mitigated

Final Update: Friday, August 11th 2017 16:23 UTC We’ve confirmed the service is healthy. Our monitors alerted us to an elevated error rate across multiple scale units, but we have since been able to confirm that isn’t alarming and there is no user impact. We apologize for the false alarm. Sincerely,Sri Harsha Initial Update: Friday,…


Postmortem: Service outage for Visual Studio Team Services on 4 August 2017

On 4 August 2017, we experienced a major service incident that lasted just over 2 hours affecting customers accessing VSTS accounts in either a US or Brazil datacenter (incident blog here). This was a repeat of an issue that impacted customers in Asia that occurred on 2 August 2017 (incident blog here) albeit with greater…


Issues with Visual Studio Team Services – 08/09 – Mitigated

Final Update: Wednesday, August 9th 2017 18:15 UTC We’ve confirmed that all systems are back to normal as of 18:00 UTC. We have forced a fail over of the database which has mitigated the issue. All necessary logs have been collected and we are closely working with our partners in SQL Azure to root cause…


Issues with Visual Studio Team Services – 08/04 – Mitigated

Final Update: Friday, August 4th 2017 13:17 UTC We’ve confirmed that all systems are back to normal as of 4th August 2017 12:34 UTC. Our logs indicate the incident started on 4th August 2017 10:30 UTC and during the 2 hours and 5 minutes that it took to resolve the issue, users in South Central…


Issues with Visual Studio Team Services – 08/02 – Mitigated

Final Update: Thursday, August 3rd 2017 04:50 UTC After working with our DNS service provider, we discovered a bad DNS server in Tokyo which was causing users in Asia to intermittently be unable to resolve any address off of “visualstudio.com” including their VSTS accounts. To mitigate the issue, the DNS server was removed from rotation…