Issues with Application Insights Web Test Data from Moscow Datacenter – 2/10 – Mitigated

Final Update:  Wednesday 2/11/2015 00:10 UTC   The networking issue has been resolved as of 2/10/2015 23:45 UTC.  Customers will see a gap for the 22-minute duration of the impact window  for web test results from the Moscow endpoint.   We are working with the networking team to determine the root cause of the networking interruption.   Thank…


Investigating Issues with VS Online – 2/7 – Mitigated

Update: Sun, Feb 8 2015 8:10 AM PST DevOps investigated the issue, intermittent login failures for one of our SQL azure database caused slowness to small subset of customers in US region. SQL Azure team took necessary steps to mitigate the issue. We are working with SQL Azure team to find root cause and steps…


Issues with Application Insights performance metric service – 2/5 – Resolved

Final Update: Thursday 2/5/2015 23:35 UTC We have resolved this issue by deploying a hotfix. Customers using the VSO portal will be able to see performance data starting at 2/5/2015 23:05 UTC. We understand the root cause of this issue and have taken steps to avoid future reoccurrences. We apologize for any inconvenience this may have caused….


Visual Studio Online Planned Maintenance: Feb 5th 9:00 UTC (01:00 PST) – Feb 8th 20:00 UTC (12:00 PST) – Delayed

Update: Thu, Feb 5 2015 4:45 PM After reviewing the telemetry from this morning’s incident, the SQL Azure team has root caused the incident to a recent performance regression that slows down the copying of the data between clusters, and results in slow secondary throttling on the source database. As soon as the SQL Azure team…


Issues with Application Insights -2/3- Mitigated

Update: Tuesday 2/3/2015 23:55 UTC The Application Insights team mitigated an issue with data services. Starting 02/03/2015 05:25 UTC until 02/03/2015 06:16 UTC, there was a long running garbage collection. This reduced the data processing rate, causing a data gap in user reports. All services are now running healthy.  We apologize for any inconvenience this…


Issue with VSO Service Hooks – 2/2 – Mitigated

Update: Wed, Feb 4 2015 12:40 AM We deployed the fix and all of the impacted customer should be able to access service hooks from VSO now. We thank you for your patience and apologize for the inconvenience caused. Sincerely VS Online Service Delivery team —————————————————– Update: Tue, Feb 3 2015 2:30 AM Devops identified…


Issues with Application Insights Search Service – 2/1 – Mitigated

Final Update: Monday 2/2/2015 06:10 UTC We have restored complete system health and all the services are running as expected . User should not see any issue while accessing application data or while running queries. There was no data loss during incident window ; impact was limited to data access. At this moment we consider this issue as mitigated and there…


Investigating Issues with VS Online – 1/30 – MItigated

DevOps investigated the issue, intermittent login failures for one of our SQL azure database caused slowness to small subset of customers in US region. SQL Azure team took necessary steps to mitigate the issue. We are working with SQL Azure team to find root cause and steps to take to avoid such further occurrences. We…


Investigating issues with VSO – 1/26 – Mitigated.

Update: 26 Jan 2015  5:50 PM UTC Our DevOps teams were able to identify the source of the performance degradation and have put in a mitigation in place to address the issue. Users will no longer experience any performance issues while accessing Visual Studio Online. We thank you for your patience and apologize for the inconvenience…


Issues with Cloud Load Testing – 1/24 – Mitigated

Update : Saturday 1/24/2015 17:00 PM UTC Issue is mitigate after DevOps took all the necessary mitigations. During the incident duration load test run would have stuck in queued state. Service is now running normally with no issues. We apologize for the inconvenience. Sincerely  VS Online Service Delivery team  ———————————————————–   Initial Update: Saturday 1/24/2015 06:51…