Intermittent access issues with VSSubscriptions – 1/13 – Resolved

Final Update: Saturday, 14 January 2017 08:02 UTC We’ve confirmed that all Subscription jobs are back to normal as of 14 January 2017 ~07:30 UTC. Jobs are running successfully and all message have been processed without any errors. Root Cause: Jobs failed with access exceptions and timeout errors Chance of Reoccurrence: Low Lessons Learned: Processing of large…


Issues with Azure RM deployments in Visual Studio Team Services – 01/17 – Investigating

Update: Tuesday, 17 January 2017 14:11 UTC Our DevOps team continues to investigate issues with Azure RM deployments where the customers may experience Build or RM deployment failures with error message “[error]Error: Could not find file ‘tempSetParameters.xml’” when Azure web app task is executed in Legacy 1.x Build/RM agent. We have deployed a fix for…


Issues with package management extension in Visual Studio Team Services – 01/13 – Resolved

Update: Friday, 13 January 2017 23:23 UTC Root cause has been isolated to resolving service identities and we fixed the issue by doing a configuration change. Customers should not see any issues during install/try/buy package management extension installation in VSTS. Sincerely,Bapayya Update: Friday, 13 January 2017 21:07 UTC Our DevOps team continues to investigate issues with package management extension…


Connectivity issues with loading accounts on Visual Studio Team Services – 01/12 – Resolved

Update: Thursday, 12 January 2017 18:00 UTC Root cause has been isolated to a bug where few user accounts failing to load when installing or buying an extension. To address this issue we deployed a hotfix and the issue is fixed now. Sincerely,Bapayya Update: Thursday, 12 January 2017 16:57 UTC Our DevOps team found an…


Performance issues with Service Hooks notifications in Visual Studio Team Services – 01/11 – Resolved

Update: Wednesday, 11 January 2017 18:31 UTC Root cause has been isolated to a single account where notifications are delayed and its not a widespread issue in South Central US region. We are working with the impacted account owner to understand why they are causing spike in service hooks notifications. Sincerely,Bapayya Initial Update: Wednesday, 11…


Issues with Visual Studio Team Services – South Central – 01/07 – Resolved

Update: Saturday, 7 January 2017 04:45 UTC From Saturday, 7 January 2017  2:00 UTC to 2:09 UTC customers in South Central may have experienced errors while using VSTS including work items, version control, build and release. Initial root cause points to an issue with SQL Azure that self mitigated.  We are following up with the SQL Azure…


Issues with Extensions containing Build/Release tasks on West European Scale units – 01/06 – Resolved

Update: Friday, 6 January 2017 02:59 UTC Root cause is still being determined.  To address this issue we flushed the cache in the extensions management service and agent tasks are now working as expected. Sincerely,Eric Update: Friday, 6 January 2017 01:20 UTC We are actively investigating issues with extensions containing Build/Release tasks in Western European…


Performance Issues issues with Visual Studio Team Services – 01/05 – Resolved

Update: Friday, 6 January 2017 00:31 UTC Root cause has been isolated to high SQL CPU on one account which was impacting all web requests to VSTS in South Central US as there was ASP.NET queuing.  To address this issue we upgraded the database to a higher SKU.  VSTS in South Central US is now working as expected. Sincerely,Eric…


Performance issues with Test Results of Visual Studio Team Services – 01/04 – Resolved

Update: Wednesday, 4 January 2017 22:49 UTC Root cause has been isolated to a code bug which was impacting the test results REST API when users were querying for all result details for a given set of result ids.  To address this issue a hotfix was deployed at 4 January 2017 18:17 UTC The test results REST API is now working as…


Hosted build issues with Visual Studio Team Services – 01/03 – Resolved

Update: Tuesday, 3 January 2017 20:34 UTC Hosted build is now working as expected. Full RCA is in progress Sincerely,Eric Update: Tuesday, 3 January 2017 19:54 UTC An issue with the job which is responsible for resizing the build pool has been identified.  We have manually increased the pool size and implemented a fix for the…