Hosted Builds are experiencing delays in South Brazil – 11/16 – Mitigated

Final Update: Friday, November 17th 2017 19:39 UTC We’ve confirmed that all systems are back to normal. Builds in South Brazil should no longer be delayed. We apologize for any inconvenience this may have caused. Sincerely,Daniel Update: Thursday, November 16th 2017 19:03 UTC Hosted Builds in South Brazil continue to be delayed past our SLA….


Performance degradation in all regions – 11/16 – Mitigated

Final Update: Thursday, November 16th 2017 18:27 UTC Customers in the Central US region are no longer experiencing slow web calls. The VSTS team continues to investigate the root cause and ensure that a fix is in place.We apologize for any inconvenience this may have caused. Sincerely,Daniel Update: Thursday, November 16th 2017 17:21 UTC The…


ADVISORY – South Brazil region unavailable for Load Test users

This is to make you aware that we have removed South Brazil as an option for the regions to execute Load tests. Users are advised to follow the workaround mentioned below. Workaround : We recommend Load Test users to choose a location other than ‘South Brazil’ when triggering a load run. See ‘Run and Analyze…


Performance Degradation in South Central US – 11/15 – Mitigated

Final Update: Wednesday, November 15th 2017 22:11 UTC We’ve confirmed that all systems are back to normal as of 11/15/17 21:20 UTC. Our logs show the incident started on 11/15/17 @ 21:00 UTC. Some customers in South Central US may have experienced slow performance. Sorry for any inconvenience this may have caused. Sincerely,Preethi Initial Update:…


Hosted Builds are experiencing delays in West Europe – 11/15 – Mitigated

Final Update: Wednesday, November 15th 2017 16:17 UTC We’ve confirmed that all systems are back to normal as of 15:45 PM UTC. The backlog of build requests in West Europe has cleared and customers should be able to queue builds quickly again. We apologize for any inconvenience this may have caused. Sincerely,Daniel Update: Wednesday, November…


Postmortem for VSTS Outages on 6 November 2017

On 6 November 2017 we experienced a series of incidents which occurred in close proximity to each other, causing significant impact to our customers during that time (service blog here). The combined impact of these incidents resulted in an outage nearly 3 hours long for our customers. The greatest impact was to users in the…


Web UI Performance Degradation in West Europe – 11/14 – Mitigated

Final Update: Tuesday, November 14th 2017 16:06 UTC We’ve confirmed that all systems are back to normal as of 11th of November 2017 15:05 UTC. Our logs show the incident started on 11th of November 2017 14:40 UTC and that during the 45 minutes that it took to resolve the issue. Customers may have experienced…


Performance Degradation on Web UI in South Central US – 11/13 – Mitigated

Final Update: Monday, November 13th 2017 21:45 UTC We’ve confirmed that all systems are back to normal as of 20:50 UTC. Our logs show the incident started on 19:45 UTC and during that duration users may have experienced intermittent slow performance across the web UI in South Central US. Sorry for any inconvenience this may…


WebUI access is slow or failing in West Europe – 11/13 – Mitigated

Final Update: Monday, November 13th 2017 14:23 UTC We’ve confirmed that all systems are back to normal as of 11/13/2017 12:35 UTC. Our logs show the incident started on 11/13/2017 12:10 and that during the 25 minutes that it took to resolve the issue. Customers experienced slow and failed commands within the UI in the…


Web UI Performance Degradation and Failures in West Europe – 11/09 – Mitigated

Final Update: Thursday, November 9th 2017 21:44 UTC We have identified the root cause of the performance issue. A function which used a regex evaluation was triggering high CPU on our application tier which caused the underlying performance issue. Our telemetry indicates that each occurrence of this issue self mitigated within 30 minutes or less….