Failures Using VSTS UI and GIT in North Central US – 07/31 – Mitigated

Final Update: Tuesday, July 31st 2018 18:59 UTC We’ve confirmed that all systems are back to normal as of 18:47 UTC. Sorry for any inconvenience this may have caused. We will be following up with the Azure Storage team for a full RCA. Root Cause: The failures were due to an Azure Storage outage in…


Build Delays in West Europe – 07/30 – Mitigated

Final Update: Monday, July 30th 2018 10:49 UTC We’ve confirmed that all systems are back to normal as of 07/30/2018 10:25 UTC. Our logs show the incident started on 07/30/2018 08:19 UTC and that during the 2 hours and 6 minutes that it took to resolve the issue. 653 of customers experienced build delays in…


Possible customer impacting event in all regions – 07/26 – Mitigated

Final Update: Thursday, July 26th 2018 14:50 UTC We’ve confirmed that all systems are back to normal as of 13:55 UTC. Our logs show the incident started on 12:10 UTC and that during the 1 hour and 45 minutes that it took to resolve the issue 0.23% of customers experienced slow responses or errors accessing…


Investigating issues with Hosted Build in South Central US – 07/20 – Mitigated

Final Update: Friday, July 20th 2018 23:12 UTC We’ve confirmed that all systems are back to normal as of 2018-07-20 22:15 UTC. Our logs show the incident started on 2018-07-20 20:30 UTC and during the incident small set of customers have noticed delays in their queued Hosted Builds getting processed. We are actively pursuing root…


Imported accounts experiencing 404s and/or are missing features in all regions – 07/20 – Mitigated

Final Update: Friday, July 20th 2018 19:34 UTC We’ve confirmed that all systems are back to normal as of 2018-07-20 19:27 UTC. This was due to a recent change introduced in the Visual Studio Marketplace service which impacted the extensions of imported accounts. During this incident, customers that recently imported their accounts using Data Import…


Possible customer impacting event in West US 2 – 07/20 – Mitigated

Final Update: Saturday, July 21st 2018 11:59 UTC We’ve confirmed that all systems are back to normal as of 7‎/‎20‎/‎2018‎ ‎14‎:‎31 UTC. Our logs show the incident started on 7‎/‎20‎/‎2018‎ ‎13‎:‎43 UTC and that during the 48 minutes that it took to resolve the issue. 0% of customers experienced access issues in the West US…


Hosted agent assignment delays in West Europe – 07/20 – Mitigated

Final Update: Friday, July 20th 2018 16:30 UTC While the Azure team continues to investigate the root cause of the performance degradation while trying to re-image virtual machines, we were able to fully mitigate the issue by increasing the capacity of our machine pools in North Europe in order to clear the build queue that…


Performance Degradation in West Europe – 07/20 – Mitigated

Final Update: Friday, July 20th, 2018 10:18 UTC We’ve confirmed that all systems are back to normal as of July 20th, 2018 09:50 UTC. Our logs show the incident started on July 20th, 2018 08:50 UTC and that during the 1 hour it took to mitigate the issue customers must have observed intermittent slowness while…


Investigating performance degradation issue South Central US – 07/14 – Mitigated

Final Update: Saturday, July 14th 2018 13:15 UTCWe’ve confirmed that all systems are back to normal as of 12:10 UTC. Our logs show the incident started on 08:00 UTC and that during the 4 hours and 10 minutes that it took to resolve the issue. Only a very small subset of customers may have experienced…