Performance Degradation in South Brazil – 06/19 – Mitigated

Final Update: Tuesday, June 19th 2018 16:19 UTC We have taken steps in order to mitigate this issue and systems are returning to normal as of 16:10 UTC Sincerely,Pedro Initial Update: Tuesday, June 19th 2018 15:24 UTC We’re currently investigating notification delays and potential build issues in South Brazil. Next Update: Before Tuesday, June 19th…


Errors for Blobstore Requests in South Central US – 06/14 – Mitigated

Final Update: Thursday, June 14th 2018 20:45 UTC Storage team has applied mitigation that fully resolved the issue. We’ve confirmed that all systems are back to normal as of 6/14 20:09 UTC . Based on telemetry the impact started at 6/14 01:55 UTC. Storage team will complete a root cause analysis and we will update…


HTTP 500/503 Errors in South Central US – 06/13 – Mitigated

Final Update: Wednesday, June 13th 2018 19:15 UTC We’ve confirmed that all systems are back to normal as of 6/13 18:30 UTC after Azure has applied mitigation to the underlying Storage availability issue in South Central US region. More details about this incident can be found on Azure status portal. We will update this post…


Hosted Agent Assignment Delays in West Europe – 06/12 – Mitigated

Final Update: Tuesday, June 12th 2018 17:43 UTC We’ve confirmed that all systems are back to normal as of June 12th 17:15 UTC. Our logs show the incident started on June 12th 13:05 UTC and that during the 4 hours and 15 minutes that it took to resolve the issue, customers in West Europe who…


Investigating issues with hosted build agents in Central US – 06/12 – Mitigated

Final Update: Tuesday, June 12th 2018 08:37 UTC We’ve confirmed that all systems are back to normal as of June 12th 2018 08:06 UTC . Our logs show the incident started on June 12th 2018 03:15 UTC. We have identified the root cause to be related to a recent change that got deployed today. The…


Performance Degradation in West Europe – 06/09 – Mitigated

Final Update: Sunday, June 10th 2018 01:32 UTC We’ve confirmed that all systems are back to normal as of June 10th 2018 01:10 UTC. Our logs show the incident started on June 9th 21:20 UTC and that during the 3 hours and 50 minutes that it took to resolve the issue. Users accessing VSTS may…


Performance Degradation in West Europe – 06/07 – Mitigated

Final Update: Thursday, June 7th 2018 14:38 UTC The mitigation has been applied to all the application tiers and we have confirmed that all systems are back to normal as of 14:02 UTC. The current hypothesis is that the performance degradation was caused by memory pressure on the application tiers. The memory pressure resulted in…


Performance Degradation in West Europe – 06/04 – Mitigated

Final Update: Monday, June 4th 2018 14:26 UTC We’ve confirmed that all systems are back to normal as of 2018-06-04 12:55 UTC. Our logs show the incident started on 2018-06-04 09:27 UTC and that during the 3 hours and 28 minutes that it took to resolve the issue, 0.11% of customers experienced performance degradation in…