Final Update: Monday, February 5th 2018 18:19 UTC
We’ve confirmed that all systems are back to normal as of February 5th 2018 17:57 UTC. Our logs show the incident started on February 2nd 2018 18:03 UTC and that during those 71 hours and 55 minutes that it took to resolve the issue 0.28% of customers experienced issues downloading files in West Europe. Sorry for any inconvenience this may have caused.
- Root Cause: The failure was due to a permissions bug in a new piece of the code
- Chance of Re-occurrence: Low - Mitigation has been put into place and a hotfix is being rolled out to prevent future occurrences
- Lessons Learned: We will be evaluating our monitoring to be able to detect these kinds of regressions sooner in the future
- Incident Timeline: 71 hours & 55 minutes – February 2nd 2018 18:03 UTC through February 5th 2018 17:57 UTC
Update: Monday, February 5th 2018 15:53 UTC
Our DevOps team continues to investigate issues with file downloads in the West Europe region. Root cause is not fully understood at this time, though it is believed to be an issue with how we are handling some permissions in a new piece of code. The problem began at 2018-02-02 18:00Z, and we are preparing a hotfix to hopefully mitigate the issue for our users.
- Work Around: None
- Next Update: Before Monday, February 5th 2018 18:00 UTC
Initial Update: Monday, February 5th 2018 14:46 UTC
We're investigating an issue with downloading files in West Europe.
We believe the issue is possibly related to some permission issues, and are looking into how to mitigate the issue for the users who are seeing this error.
Users may encounter a "500 Internal Server Error" when attempting to download files from Visual Studio Team Services
- Next Update: Before Monday, February 5th 2018 15:20 UTC