Processing issues with Visual Studio Team Services in South Central US- 10/5 – Mitigated

Final Update: Thursday, 6 October 2016 00:00 UTC We’ve confirmed that all systems are back to normal as of 05 October 2016 23:45 UTC. Our logs show the incident started on 05 October 2016 21:48 UTC and that during the 2 hours that it took to resolve the issue, customers may have noticed issues with Hosted Builds, Queuing…


Performance issues with Visual Studio Team Services – 11/10 – Resolved

Final Update: Thursday, 10 November 2016 17:03 UTC We’ve confirmed that all systems are back to normal as of 10 November 2016 ~16:46 UTC. Our logs show the incident started on 10 November ~08:20 UTC and that during the duration of this incident a subset of customers hosted in our West Europe region on the impacted partition database would have experienced slow…


File Not found errors using Deploy Azure RM Web App – 11/11 – Resolved

Update: Friday, 11 November 2016 17:11 UTC Between 12:30 and 16:00 UTC today, users of Visual Studio Team Services using Deploy AzureRM Web App task might have faced ‘File not found’ errors. This has been resolved by pushing out a configuration change and users should not see any more issues.  We’re working on a detailed…


Issues with Market place feature in Visual Studio Team Services – 11/14 – Resolved

Final Update: Tuesday, 15 November 2016 06:11 UTC We’ve confirmed that all systems are back to normal as of 15 November 2016 06:11 UTC. Fix has been rolled out to mitigate PAT tokens issue. Users should be able to create publisher, create/update extension either by using command line tools or portal. We suggest to regenerate…


Performance issues with Visual Studio Team Services – 11/14 – Resolved

Final Update: Monday, 14 November 2016 15:40 UTC We’ve confirmed that all systems are back to normal as of 14 November 2016 15:20 UTC. Our logs show the incident started on 14 November ~12:24 UTC and that during the duration of this incident a subset of customers hosted in our West Europe region on the impacted partition databases would…


Performance issues with Visual Studio Team Services – 11/15 – Mitigated

Update: Tuesday, 15 November 2016 01:11 UTC We had connectivity issues on a SQL Database causing performance issues in North Central US region. We are actively working on a Root cause analysis. This issue is already mitigated and customers shouldn’t see any issue currently. Impact Start Time: 2016-11-14 22:52 UTC. Impact End Time: 2016-11-14 23:16 UTC. We understand that customers…


Performance issues with Visual Studio Team Services – 11/15 – Mitigated

Update: Tuesday, 15 November 2016 17:11 UTC Root cause has been isolated to a SQL database performance issue which impacted Visual Studio Team Services users in the North central US region. To address this issue we have worked with SQL Azure team and as per their recommendation we have upgraded database SKU and also took…


Connectivity issues with Packaging service – 11/15 – Resolved

Update: Wednesday, 16 November 2016 02:11 UTC Root cause has been isolated to a config change where a switch from preview to a paid packaging service experienced issues due to underlying licensing data state for preview users. The affected accounts have been fixed and issue should be mitigated now. If you are still experiencing issues,…


Issues with Hosted Build in Visual Studio Team Services – 11/17 – Root Cause

Customer Impact:For approximately 6 hours on 17th November 2016, many Hosted Build customers experienced impact due to their builds not being queued for processing. The peak of the impact occurred between 18:30 and 19:30 UTC.  At 00:40 UTC the following day the mitigation was completely applied and customers would have seen their builds behaving as…


Issues with work item search feature in Visual Studio Team Services – 11/18 – Resolved

Update: Friday, 18 November 2016 18:18 UTC Root cause has been isolated to Full Text search on Azure SQL DB’s which was impacting search feature in Visual Studio Team Services, this issue impacted only a subset of users in West Europe. To address this issue we worked with SQL Azure team and mitigated the issue….