Final Update: Thursday, October 4th 2018 05:53 UTC
We’ve confirmed that all systems are back to normal as of 4th October 04:02 UTC. The hotfix deployment is completed on all the rings. Our logs show that the incident started on 3rd October 4:40 UTC. Sorry for any inconvenience this may have caused.
- Root Cause: We have identified that the issue is related to recent change that was deployed.
- Chance of Re-occurrence: Low
Update: Thursday, October 4th 2018 01:00 UTC
Our hotfix deployment to fix the issue has been progressing steadily across various rings and has got deployed successfully on half of the rings. It is expected to be complete across all regions/rings in another 12 hours.
- Next Update: Before Thursday, October 4th 2018 13:00 UTC
Update: Wednesday, October 3rd 2018 19:52 UTC
We have narrowed down the issue to a change that recently deployed. We are working on a hotfix deployment to revert this change and we expect the roll out to be completed within the next 6 hours.
- Next Update: Before Thursday, October 4th 2018 02:00 UTC
Initial notification: Wednesday, October 3rd 2018 18:27 UTC
We're investigating a possible customer impacting event in all regions. We are triaging the issue and will provide more information as it becomes available.
Customers may experience error WorkItemTypeDeniedOrNotExistException when querying work items, via the Client OM. The impact appears contrained to only customers using custom Work Item Types.
- Next Update: Before Wednesday, October 3rd 2018 19:27 UTC