Builds using Nuget and .NET extensions may experience failures in multiple regions – 08/08 – Mitigated

Final Update: Wednesday, August 8th 2018 13:23 UTC We’ve confirmed that all systems are back to normal as of 08/08/2018 14:12 UTC. Our logs show the incident started on 08/07/2018 20:30 UTC and that during the 18 hours that it took to resolve the issue. Customers may have experienced build failures in multiple regions. Sorry…


Email notification in all regions – 08/02 – Mitigated

Final Update: Friday, August 3rd 2018 23:46 UTC We’ve confirmed that email delivery is working as expected as of 08/03/2018 21:20:45 UTC. We are sorry for any inconvenience this may have caused. Sincerely,Eric Update: Friday, August 3rd 2018 00:32 UTC The sync below is expected to run over the next 24 hours. In the meantime,…


Investigating issues with code search in multiple regions – 08/02 – Mitigated

Final Update: Thursday, August 2nd 2018 11:04 UTC Our investigations show that this only affects internal Microsoft employees, 3rd party customers are not affected. We are sorry for any inconvenience this may have caused. Sincerely, Alberto Initial notification: Thursday, August 2nd 2018 10:30 UTC We’re investigating issues with code search in Australia East, Canada Central,…


Analytics in South Central US – 08/01 – Mitigated

Final Update: Wednesday, August 1st 2018 14:35 UTC We’ve confirmed that all systems are back to normal as of 13:46 UTC . Our logs show the incident started on 13:10 UTC and that during the 36 minutes that it took to resolve the issue. Customers experienced Analytics delays in the South Central US. Sorry for…


Failures Using VSTS UI and GIT in North Central US – 07/31 – Mitigated

Final Update: Tuesday, July 31st 2018 18:59 UTC We’ve confirmed that all systems are back to normal as of 18:47 UTC. Sorry for any inconvenience this may have caused. We will be following up with the Azure Storage team for a full RCA. Root Cause: The failures were due to an Azure Storage outage in…


Build Delays in West Europe – 07/30 – Mitigated

Final Update: Monday, July 30th 2018 10:49 UTC We’ve confirmed that all systems are back to normal as of 07/30/2018 10:25 UTC. Our logs show the incident started on 07/30/2018 08:19 UTC and that during the 2 hours and 6 minutes that it took to resolve the issue. 653 of customers experienced build delays in…


Possible customer impacting event in all regions – 07/26 – Mitigated

Final Update: Thursday, July 26th 2018 14:50 UTC We’ve confirmed that all systems are back to normal as of 13:55 UTC. Our logs show the incident started on 12:10 UTC and that during the 1 hour and 45 minutes that it took to resolve the issue 0.23% of customers experienced slow responses or errors accessing…


Investigating issues with Hosted Build in South Central US – 07/20 – Mitigated

Final Update: Friday, July 20th 2018 23:12 UTC We’ve confirmed that all systems are back to normal as of 2018-07-20 22:15 UTC. Our logs show the incident started on 2018-07-20 20:30 UTC and during the incident small set of customers have noticed delays in their queued Hosted Builds getting processed. We are actively pursuing root…


Imported accounts experiencing 404s and/or are missing features in all regions – 07/20 – Mitigated

Final Update: Friday, July 20th 2018 19:34 UTC We’ve confirmed that all systems are back to normal as of 2018-07-20 19:27 UTC. This was due to a recent change introduced in the Visual Studio Marketplace service which impacted the extensions of imported accounts. During this incident, customers that recently imported their accounts using Data Import…