How we use User Voice to make a better product

To increase our engagement with our customers, we created the Visual Studio User Voice site to capture your feedback for Team Foundation Server (TFS) and Visual Studio Online (VSO). Since then more than 20,000 people have been participating and we have received over 2,300 suggestions. In the last year we have completed 150 suggestions. We are excited to have so many people who have taken the time to give us feedback. The goal of this post is to describe how we use the User Voice data, and what we have changed to improve our response to the User Voice feedback. If you think we can improve how we use User Voice, please leave a comment or reach out to me via email: ewald.hofman at microsoft.com.

Gaining insight

When we started with User Voice, we received a handful of suggestions per week. Many of these items were already on our backlogs and to be as transparent as possible to our supporters we marked these suggestions as “Under Review”. However we haven’t made any progress on them yet and as a result we haven’t responded to these suggestions for more than 4 years. It may not be a surprise that we created the perception of not listening to customers. We felt really bad about it, so late 2013 we started an initiative to improve how we interact with User Voice and its community.

The first step in the process was to groom the list of User Voice suggestions. We merged duplicates, updated the status of already completed suggestions and assigned the most popular suggestions to the correct feature teams. When we shared the top 20 most popular suggestions, trends and whether we were addressing them for the next major release with all the feature teams, it generated a lot of interest about User Voice. And we didn’t stop with that one time exercise, we continued to groom the User Voice suggestions to keep the list healthy, and regularly we share a summary report to all feature teams working on TFS/VSO.

Giving insight

After we gained the insight in the User Voice data, it was time for the next step. Working with User Voice was a one-way communication: we listened to the community, but did not update the status of completed suggestions nor did we tell you if we were working on something. To improve in this area we made two changes.

  1. Track suggestions on our backlogs: We did not complete the User Voice suggestions, because we did not have a link between our backlog and User Voice. Therefore the feature team was just not aware that a shipped feature was backed by User Voice. To improve that we created bi-directional links between our backlogs and the User Voice for the most popular suggestions. When we ship a feature we can easily find the User Voice suggestions that we need to update. If you see an open suggestion which you think is already completed, please let me know and I will get it fixed.
  2. Frequent updates on in-progress items: The “Under Review” state was overloaded because we used it for so many suggestions in the past. To better indicate which one we are actively working on or which ones are on our 6-month plan, we introduced the states “Started” and “Planned”. The goal is to give an update on in progress suggestions at least every 3 months. The majority of the Under Review suggestions are still on our backlog, but they are not part of the 6-month plan. We are going to bite the bullet and reset the state of all these Under Review suggestions, to make it very simple to you. If a suggestion has a status, you can expect us to give regular updates until we have completed the item.

We are listening…

To prioritize our backlogs we use many channels for input, and User Voice is one of these channels. And we take it very seriously.

  • Below you see the list of 150 suggestions that we have completed in the last year. Based the User Voice feedback we completed items which were not in our original plans, including the “Creating Queries – Token for Current Iteration” and “ability to add multiple activities per team member in sprint”. But we heard you loud and clear and we prioritized these items higher on our backlogs.
  • There are currently 49 items in progress (state is “Planned” or “Started”). Some of them have gone stale, but in the next few weeks all in progress suggestion will have an recent update. You can hold us accountable for that, so if you see an in-progress suggestion without a recent update don’t hesitate to reach out to me.
  • Recently Brian Harry has published a blog post which shows some of the items that we are planning to work on in the next few months, and that list is influenced by User Voice.
  • In each planning cycle the feature teams are being questioned why they are not working on one of the top 10 items

These are the suggestions that we have closed in the last year sorted on the number of votes.

Suggestion Votes
rename project in TFS 6159
Add “Create Unit Tests” menu option back! 3038
Creating Queries – Token for Current Iteration 2188
Make Code Review available to Professional users 1221
Customization of TFS Web Access 1201
ability to add multiple activities per team member in sprint 1094
Allow TFS Task Board to have custom colors 880
In the team Explorer, Change Default State value of a selected Work Item to “Associate” instead of “Resolve” on Check In 845
Excel integration with MTM 792
Allow tests to be grouped by class, category, and namespace in Test Explorer 699
Support more browsers and browser versions in MTM 520
Integrate Team Wiki / project documentation pages 471
Team Test – Improve the usability of Microsoft Test Manager 457
Bring back the view all pending changes window (VS11) 436
Use a build controller over more than one Team Project Collections 397
Provide the ability to prioritise Backlog items in a sprint backlog by dragging and dropping the item 340
Provide TF build activities like “deploy web app”, “deploy database”, or “copy file” 305
Allow tests to be paused, allow other tests to be run, and then resume the paused test case 293
Suite created by “Workitem and direct links” query in MTM 282
Implement TFS Build for linux (Create Linux build agent) 272
Continue support for Database Unit Tests in VS 2012 267
Testers should be able to get parameters for a test case from a csv file 260
Show me all test suites/ test plans a test case belongs to 245
Provide versioning of test cases to support multiple parallel releases 243
Allow testers to edit test cases inside Manual Test Runner 241
Support Windows 8 apps in tfspreview hosted build 234
Provide check-in/check-out functionality through Team System Web Access 218
Make the “My Work” section available in VS 2012 Professional 215
Add git blame support 212
Kanban board: More customizations and usability features 198
Add ID numbers to the Task Board tasks and work items (PBIs and Bugs) 185
Drag and drop reordering for multiple consecutive items in the product backlog 184
Test Suite Security Permissions in MTM 184
Allow testers to copy test cases by creating new duplicates, not adding references 179
Collapse Product Backlog Items marked as Completed. 170
Allow Test Case Work Items to be created for Native C++ Unit tests 166
Visual Studio Tools for Git Extension (Microsoft Git Provider) integration with Source Control Explorer window 164
Include feature of Task Board Enhancer plug-in 159
Provide more visibility for a test case’s result status 152
Add “tags” support in Git Provider 147
make Test Scribe word reports customizable 143
Allow Bugs to show on the Board 135
See which test plans/runs are running on a lab environment 134
Git Provider should *not* add new files to index automatically 130
Improve Test Manager performance 127
support Spec Explorer in Visual Studio 2012 126
provide a way to track history of test plan and test suites 98
Coded UI Test must support Silverlight 5 95
Fix the ctrl+e shortcut in test runner 85
Allow Git Tools for Visual Studio to Rebase on pull 81
Allow TFS task board to show Tasks without parent 76
By design the board only shows tasks that are associated with Product Backlog Items (it does not show ‘free-floating’ tasks). 76
Allow VSO Stakeholder License to use Feedback Client 75
Visual Studio 2012 coded UI test *must* support Chrome, Safari and FF4! 72
Provide customization for test plan, test results 72
Burndown chart should use the capacity of the team for the ideal trend 71
Improve the usability of the Lab manager component of test manager 71
Drop “Test Explorer” view and revert to the VS 2010 Test View and Test Results paradigm 70
Enable Kanban Column field to be used in work item queries 67
TFS Build: download Git submodules automatically 64
Make it possible to read/write the team capacity with the TFS api 61
coded ui Chrome browser support 57
Parameter passing while queuing builds from TFS Web 54
TFS Scrum: Show state of PBIs and Bugs on board 54
Bring color back to VS 2012 RC Source Control Explorer 51
Provide feedback when unit tests are running 51
Export test plan tree between projects and collections in MTM 51
add real horizontal swim lanes to the Kanban board 50
Support Deployment of Multiple Websites to an Azure Web Role from VSO 45
Add custom fields to allow better Team Build management 42
Make the TFS work item number copy-able 42
Allow cut and paste in Unit Test output window. 38
Add spell check functionality to MTM so that test case writers have the ability to spell check test case steps etc. 38
For TFS 2012 Beta, In scrum template : limit the Assigned To list to my team members instead of showing all existing users 35
Provide TFS Build Source Indexing for Git 34
Support Coded UI Testing for Windows Store applications 34
Provide an audit log of changes to TFS build configurations 32
task reordering 31
Allow using ENTER or SHIFT ENTER when writing test steps 31
Search tests by category in the Unit Test Explorer 29
Provide fine-grained permissions on test plan 27
Better commit message on Visual Studio Online pull request merges 22
Edit test steps within Visual Studio or TFS Web Access 22
Hi, we need the Silverlight plugin related to Coded UI in Visual Studio 2013. Our existing Testscripts testing Silverlight cannot run. 22
Drop MSTest, let any test framework plug into the IDE 21
Create a PowerPivot/Tabular Model store that can be populated from the TFS Service 20
more filteroptions for TestManager Testingcenter 19
Allow Visual Studio Online users with Basic license to view Features list 18
Allow visibility of Test Plan results summary from the Test page in TFS 16
Query based suites in MTM 14
email test case from MTM 13
When sending a work item to Outlook to use Outlook mail format options 12
Ability to copy multiple test cases simultanuously 12
Test Manager 2010: Export Test Cases and preserve the fields/format/parameters 11
Allow the Tester field to be configurable 11
Allow acceptance testers to access TFS test hub 10
Provide a way to filter the list of items in the product backlog (under Work – Backlog) on the web view 10
Provide a licence for non devs that allows Team Room access 10
build security 10
Enable searching through work item query results in Team Foundation Server 10
Make Test and Lab Management available to MSDN Premium Subscribers 10
There must be some description field of Test Suite 10
Merge Test Manager functionalities in Team Explorer / Web Access 9
Support the ability to deploy a test agent to Azure 9
Visual Studio Online (TFS) should show task numbers on task board 7
Reintroduce the shortcut keys for running and debugging the current unit test 7
Allow more than 20 backlog items to show in the “New” column of the Kanban board 6
Allow the tester to modify the test steps in Test Run screen just like QC allows. 6
VSO REST API: provide a way to disable rules for Migration scenarios from other providers. 5
Allow show tags on taks on the sprint board. 5
With VS 2012, give more grouping options for unit tests window. 5
Editing additional fields from the board cards 4
Add the ability to edit on the fly while running a test 4
Ability to collect Custom Logs via MTM test settings. 4
Allow the tester to have options to mark the test case as N/A, Not completed 4
Encode TFS work item queries in a URL 3
Assign/E-mail Stakeholders from Backlog Item 3
In Visual Studio Online, copy the comments with failed tests into the bug report that is generated 3
Improve navigating to links for touch screen in VSO 3
Display backlog items #id in backlog board view 3
Test Manger 2010 customization 3
Add the unit testing wizards back 3
add support for [shift+CR] Linebreaks in MTM Shared Steps, Test Steps, etc. 3
Integration between QTP & Team Foundation Server to support the CI process 3
Add unit testing to express editions 3
TFS Online change order of tasks 2
Set Task Remaining Hours to 0 when Status is set to Closed. 2
Enable copying of test case steps to a new test case 2
Add feature in TFS to export and import test case to and from TFS with steps,actions and expected result details 2
add visual studio online as a category on uservoice 1
Copy and paste test steps from Excel into Test Suite Grid view with alerts if others have modified the existing test cases since Iast pull 1
Moving user stories to new iteration should not change iterations of closed sub tasks 1
Freeze toolbar and states on Kanban Board 1
Allow browsing for WITs when pressing “link to…” 1
View test results in team web access 1
… to enable drag and drop for pbi in feature view. would be useful while assigning pbi to current sprint. 1
change the default config of the TFS2013 Kanban default swimline “Done” to the 20 most recent 1
bring back inline assigned to and effort fields on the task board 1
Inline search in query column headers 1
CodeLens Support For Visual Studio 2013 1
Allow Creation of Work Items w/o a Paid VS Online Subscription 1
Enable the ability to hyperlink to Test Plans from SharePoint 1
Extract parameter list from MTM to Excel/CSV 1
Export MTM 2013 test cases to excel . currently Test scribe or test case extractor not working in 2013 1
Limit limit available users for test points in MTM 1
Would like to have the hyperlink active for authoring test cases 1
perhaps have a license type that allows business users to run (only) test cases. 1
Make it possible to add and query by tags in MTM 2013 1
Display Team Project name in Test Manager 1
I have an issue with MTM: When I run a test and found a bug at step 13, I cannot pause the test,start running a test, following it step by s 1
6