Community Health for Visual Studio Team System– Part 1 (details)

Importer

As you read this blog post, you are part of the community for Visual Studio Team System. Having a healthy community is important not only for the long-term happiness of the customers (including yourself) but also the long-term success of the product. To help foster a healthy community, we have a working group called the Visual Studio Team System Community Council. This group has three goals:

·         Create and support a self-sustaining community

·         Provide a feedback loop for customers into our product development process

·         Earn the trust and loyalty of our customers

If you have an analytical nature, as I do, you might ask yourself “How can you tell if you have a healthy community?” Well, unfortunately, there is no community health meter that we can look at periodically to see whether the community is healthy. So we have a number of community initiatives, and we try to identify key metrics that will give us a sense for whether our initiatives are helping to achieve the above goals.

Over the past year, we have monitored the following community initiatives:

·         MSDN Forums –  covered in this post

·         Channel 9 Broadcasts

·         MSDN Chats

·         Visual Studio Team System Advisory Council

·         CodePlex (added in April)

·         Webcasts

·         Connect

·         Conferences

·         MVP Program

·         Blogs

In this post (part 1 of several), I want to talk a little bit about the MSDN Forums for Visual Studio Team System. Future posts will cover details about the other initiatives. In each of these posts, I will look at the metrics that we tracked, our interpretation of those metrics, and our plans for the initiatives moving forward.

MSDN Forums

For the past year, we monitored two primary metrics for the MSDN Forums: the number of posts and the percentage of posts that were answered within seven days. The number of posts gives us an idea how much traffic came to the forums for Visual Studio Team System. We try to answer all posts within seven days because, after seven days, it is unlikely that the original poster is still expecting an answer.  We do not have a specific target for the number of posts per month; we just hope to see that number slowly increasing over time. Our specific goal was to answer 80% of posts about Team System within seven days.

Progress against Goals

How did we do? We saw an average of 1000 posts per month across all of the Visual Studio Team System forums on MSDN. We answered only 70% of those questions within seven days, with a peak last summer and a valley over the holiday period:

We have seen improvements in some of the more challenging forums, such as the debugger forum where many questions require more time to investigate and answer.

Other Takeaways

As I gathered this data, I confirmed my suspicion that we had an incomplete set of metrics. The data prompted questions such as:

·         Are we (Visual Studio Team System) improving the health of our community if we ask our team members to answer more questions? What does it mean for the forums to become self-sustaining?

·         Why do we take so long to answer some questions? Does the investigation take awhile, or is there a long delay before anyone reads the question and posts an initial response?

Thinking about these questions, I decided that, for me, a “healthy” forum would have two things: return visitors and a significant percentage of questions being answered by the community (people who do not employed in Visual Studio). 

Now, I know some folks might read this post and ask why Microsoft employees do not answer 100% of the questions. In a healthy community, people want to feel as though they can contribute their thoughts, experiences, and ideas. Those of us here in Visual Studio may know how we designed the product to be used, but we cannot tell you how each of you use it every day to solve problems. You probably have encountered or will encounter situations that we did not anticipate. Your experience can help other customers. Plus, for the forums that I visit regularly, I like that feeling I get when I answer a question and help someone out. I am sure that others of you enjoy that as well, and I would rather see the Team System members focus on the posts that cannot easily be answered by community members. The community helps each other, and those of us in Visual Studio help with the questions that are not easily answered by the community.

So, looking ahead, I think we need to monitor some additional metrics:

·         Percentage of questions answered by non-Microsoft folks

·         Number of repeat visitors

·         Average response time. Average response time is the time that elapses between the original post and the first response from a different person. That tells me whether folks are likely to return to a forum or feel like they are just hollering down a well (which might be amusing for a few seconds but rarely productive J).

That raises the question of what our goals should be. Well, there is no magic number for the percentage of posts answered by the community. We will monitor the numbers to see trends over time and hope to see numbers gradually trend upward until they plateau.  Similarly, for repeat visitors, I think it is just an interesting data point to watch, like the number of posts per month.

For average response time, over the past year, we have seen an average of between 2.5 and 4 days in most of our forums. We will want to make sure that this number stays below 4, to give us a chance of having an answer within 7 days.

As a note, we do recruit frequent contributors to the MSDN forums when we are looking for MVP candidates.

Forums as a Feedback Mechanism

We have gotten some great feedback in the forums. Our User Education teams regularly review forum posts to identify areas where we may need to improve our product documentation. Our team develops a better understanding of how you are using the product by reading about your experiences. You give us good feedback not only about where we need to add or change features but also where we have perhaps made something overly complex. In some cases, for feature requests and suggestions, we may have asked you to submit your idea through the http://connect.microsoft.com/VisualStudio site. On this site, other customers can vote on your suggestion, and  we can track it and prevent it from falling through the cracks.

I think the MSDN forums have worked very well as a feedback mechanism, and I would like to thank all of you who have posted feedback there.

Final Thoughts

To summarize the results for the MSDN forum community, the past year was okay. We have some room for improvement in both our response rates and how we track health for the forums. On behalf of myself and the other members of the Visual Studio Team System Community Council, I ask you to send us your thoughts. Let us know how the MSDN Forums are working for you, where we are doing well, and where we have “room for improvement.” 

Looking Ahead

In the coming weeks, I will address the other community initiatives. Next time, I will cover the Visual Studio Team System Advisory Council, MSDN Chats, and Channel 9 broadcasts. If time permits, I may also share a few words about CodePlex.

-Steven

Content Lead, Team Edition for Database Professionals

Importer

Follow