Hosted Build delays in US region – 11/29 – Mitigated

Final Update: Wednesday, November 29th 2017 21:22 UTC We’ve confirmed that all systems are back to normal as of 20:55 UTC. Our logs show that customer impact occurred from 20:40 UTC – 20:55 UTC. At this time the backlog has fully cleared. We apologize for any inconvenience this may have caused. Sincerely,Colin Initial Update: Wednesday,…


Slow web UI performance in West Europe – 11/28 – Mitigated

Final Update: Tuesday, November 28th 2017 16:01 UTC We’ve confirmed that all systems are back to normal after the workaround was implemented in the West Europe region. We apologize for any inconvenience this may have caused. Sincerely,Daniel Update: Tuesday, November 28th 2017 15:11 UTC We continue to apply a workaround for the slow web UI…


User Hub may not be displaying all users in multiple regions – 11/28 – Mitigated

Final Update: Tuesday, November 28th 2017 20:50 UTC We’ve confirmed that all systems are back to normal. We deployed the fix successfully across all affected scale units. We apologize for any inconvenience this may have caused. Sincerely,Bapayya Update: Tuesday, November 28th 2017 17:10 UTC Our DevOps team continues to investigate issues with User Hub not…


Slow web UI performance in West Europe and North Central US – 11/21 – Mitigated

Final Update: Tuesday, November 21st 2017 18:44 UTC We’ve confirmed that all systems are back to normal. The workaround has been applied successfully across all affected scale units. We apologize for any inconvenience this may have caused. Sincerely,Daniel Update: Tuesday, November 21st 2017 18:19 UTC The VSTS Engineering team has identified a workaround for the…


Hosted Builds are experiencing delays in South Brazil – 11/16 – Mitigated

Final Update: Friday, November 17th 2017 19:39 UTC We’ve confirmed that all systems are back to normal. Builds in South Brazil should no longer be delayed. We apologize for any inconvenience this may have caused. Sincerely,Daniel Update: Thursday, November 16th 2017 19:03 UTC Hosted Builds in South Brazil continue to be delayed past our SLA….


Performance degradation in all regions – 11/16 – Mitigated

Final Update: Thursday, November 16th 2017 18:27 UTC Customers in the Central US region are no longer experiencing slow web calls. The VSTS team continues to investigate the root cause and ensure that a fix is in place.We apologize for any inconvenience this may have caused. Sincerely,Daniel Update: Thursday, November 16th 2017 17:21 UTC The…


ADVISORY – South Brazil region unavailable for Load Test users

This is to make you aware that we have removed South Brazil as an option for the regions to execute Load tests. Users are advised to follow the workaround mentioned below. Workaround : We recommend Load Test users to choose a location other than ‘South Brazil’ when triggering a load run. See ‘Run and Analyze…


Performance Degradation in South Central US – 11/15 – Mitigated

Final Update: Wednesday, November 15th 2017 22:11 UTC We’ve confirmed that all systems are back to normal as of 11/15/17 21:20 UTC. Our logs show the incident started on 11/15/17 @ 21:00 UTC. Some customers in South Central US may have experienced slow performance. Sorry for any inconvenience this may have caused. Sincerely,Preethi Initial Update:…


Hosted Builds are experiencing delays in West Europe – 11/15 – Mitigated

Final Update: Wednesday, November 15th 2017 16:17 UTC We’ve confirmed that all systems are back to normal as of 15:45 PM UTC. The backlog of build requests in West Europe has cleared and customers should be able to queue builds quickly again. We apologize for any inconvenience this may have caused. Sincerely,Daniel Update: Wednesday, November…


Postmortem for VSTS Outages on 6 November 2017

On 6 November 2017 we experienced a series of incidents which occurred in close proximity to each other, causing significant impact to our customers during that time (service blog here). The combined impact of these incidents resulted in an outage nearly 3 hours long for our customers. The greatest impact was to users in the…