Final Update: Wednesday, August 8th 2018 13:23 UTC
We’ve confirmed that all systems are back to normal as of 08/08/2018 14:12 UTC. Our logs show the incident started on 08/07/2018 20:30 UTC and that during the 18 hours that it took to resolve the issue. Customers may have experienced build failures in multiple regions. Sorry for any inconvenience this may have caused.
- Root Cause: The failure was due to a change that recently occurred which involved the NuGet and .NET extensions.
- Chance of Re-occurrence: Low
- Lessons Learned: We are working both minimizing resource-intensive activities in our post-deployment steps, and are also working targeting monitors specifically to detect post-deployment issues in the future.
- Incident Timeline: 18 hours – 08/07/2018 20:30 UTC through 08/08/2018 14:12 UTC
Update: Wednesday, August 8th 2018 12:13 UTC
Our DevOps team continues to investigate issues with builds failing due to extensions failing due an inability to look up an extension area. This is affecting the nuget, .Net Core, as well as a few other extensions. Initial investigation is pointing to a change which changes the way that we look up these particular extensions. The problem began at 8/7/2018 16:30 UTC. We are currently working on a rollback of the change and expect for it to go out within the next hour.
- Work Around: Some customers have reported a work around where they have changed the version of the Extension having issues from Ver. 2 to 1.
- Next Update: Before Wednesday, August 8th 2018 14:30 UTC
Initial notification: Wednesday, August 8th 2018 11:18 UTC
We're investigating a possible customer impacting event in all regions. We are triaging the issue and will provide more information as it becomes available.
We have become aware of an issue involving the NuGet extension in VSTS where some customers may be experiencing the error "Unable to locate the 'nuget' [b3be7473-68ea-4a81-bfc7-9530baaa19ad] area. The service containing that area may not be available in your region.". We are actively investigating this issue.
Some customers have reported a work around where they have changed the version of the NuGet Extension from Ver. 2 to 1 and seeing success.
- Next Update: Before Wednesday, August 8th 2018 11:50 UTC