Possible customer impacting event in West US 2 – 07/20 – Mitigated

Final Update: Saturday, July 21st 2018 11:59 UTC We’ve confirmed that all systems are back to normal as of 7‎/‎20‎/‎2018‎ ‎14‎:‎31 UTC. Our logs show the incident started on 7‎/‎20‎/‎2018‎ ‎13‎:‎43 UTC and that during the 48 minutes that it took to resolve the issue. 0% of customers experienced access issues in the West US…


Hosted agent assignment delays in West Europe – 07/20 – Mitigated

Final Update: Friday, July 20th 2018 16:30 UTC While the Azure team continues to investigate the root cause of the performance degradation while trying to re-image virtual machines, we were able to fully mitigate the issue by increasing the capacity of our machine pools in North Europe in order to clear the build queue that…


Performance Degradation in West Europe – 07/20 – Mitigated

Final Update: Friday, July 20th, 2018 10:18 UTC We’ve confirmed that all systems are back to normal as of July 20th, 2018 09:50 UTC. Our logs show the incident started on July 20th, 2018 08:50 UTC and that during the 1 hour it took to mitigate the issue customers must have observed intermittent slowness while…


Investigating performance degradation issue South Central US – 07/14 – Mitigated

Final Update: Saturday, July 14th 2018 13:15 UTCWe’ve confirmed that all systems are back to normal as of 12:10 UTC. Our logs show the incident started on 08:00 UTC and that during the 4 hours and 10 minutes that it took to resolve the issue. Only a very small subset of customers may have experienced…


Data imports and other asynchronous jobs are slow in all regions – 07/13 – Mitigated

Final Update: Friday, July 13th 2018 20:02 UTC The team has implemented a short term mitigation that has allowed Data Import jobs to continue. After stopping the deployment and adding additional capacity, the issue was mitigated. We apologize for any inconvenience this may have caused. Sincerely,Daniel Update: Friday, July 13th 2018 17:42 UTC Our DevOps…


Release Manager connection errors when calling Azure Resource Manager in multiple regions – 07/13 – Mitigated

Final Update: Friday, July 13th 2018 12:02 UTC After some investigation, we were able to identify that this issue is only happening on a small subset of customers with specific environments and should not affect most customers. Chance of Re-occurrence: Low Sincerely,Pedro Initial notification: Friday, July 13th 2018 10:00UTC We’re investigating a recurrence of failures…


Release Manager connection errors when calling Azure Resource Manager in all regions – 07/12 – Mitigated

Final Update: Thursday, July 12th 2018 21:17 UTC We’ve confirmed that all systems are back to normal as of 2018/07/12 19:40 UTC. Our logs show the incident started on 2018/07/09 20:40 UTC and that during the 71 hours that it took to resolve the issue, a small percentage of customers experienced intermittent Release Manager connection…


Performance degradation in North Central US – 07/11 – Mitigated

Final Update: Wednesday, July 11th 2018 15:55 UTC We’ve confirmed that all systems are back to normal as of 15:20 UTC. Our logs show the incident started on 5:45 UTC and that during the 9 hours and 35 minutes that it took to resolve the issue. 1% of customers experienced performance degradation in the North…


Release Manager Pipelines Issues in Multiple Regions – 07/10 – Mitigated

Final Update: Wednesday, July 11th 2018 00:13 UTC We’ve confirmed that all systems are back to normal as of 19:00 UTC 7/10/2018. Our logs show the incident started on 14:00 UTC 7/09/2018 and that during the 29 hours that it took to resolve the issue, a small number of customers experienced ECONNRESET errors while trying…


Performance Degradation in multiple regions- 07/04 – Mitigated

Final Update: Wednesday, July 4th 2018 09:55 UTC We’ve confirmed that all systems are back to normal as of July 4th, 2018 09:55 UTC. During the course of the investigation, we identified the load is caused by a single user. We ended up blocking this user which addressed high CPU and slow commands in all…