Community Health for Visual Studio Team System– Part 2 (details)

This post is the second in a series about who the Visual Studio Team System Community Council is, what goals we’re trying to accomplish, and what results we’ve seen so far. In the year between August 2006 and July 2007, the council monitored the following community initiatives:

·         MSDN Forums –  covered in Part 1

·         Channel 9 Broadcasts – covered in this post

·         MSDN Chats – covered in this post

·         Visual Studio Team System Advisory Council – covered in this post

·         CodePlex (added in April)

·         Webcasts

·         Connect

·         Conferences

·         MVP Program

·         Blogs

In this post, I want to talk about Channel 9 broadcasts, MSDN chats, and the Visual Studio Team System Advisory Council. The VSTS Community Council consists of VSTS team members that are passionate about our community. The VSTS advisory council consists of customers that are passionate about the VSTS community. As I did in part 1, I will look at the metrics that we tracked, our interpretation of those metrics, and our plans for the initiatives moving forward.

Channel 9 Broadcasts

Channel 9 (https://channel9.msdn.com/) is an MSDN Web site that hosts a large collection of videos about various topics, and several of those videos pertain to Visual Studio Team System. Channel 9 provides a “push” mechanism for the product teams to provide additional information about features in Visual Studio Team System. The videos can provide a focused look at how customers can use a set of features to solve a particular problem. In addition, customers can obtain insight into what the product team was thinking when they created or updated specific features. 

Progress against Goals

For Channel 9, we track  two metrics: how many videos feature Visual Studio Team System and how many views all those videos get. We want to publish at least one new video per month. Although we don’t have a specific goal for how many views the videos get, the total number to date gives us a sense for how many customers are getting value from the videos.

How did we do? We released 12 videos in the year between August 2006 and the end of July 2007, increasing the overall total from 31 to 43. Although we hit our goal for the year, the release of new videos was rather “clumpy.” We published five videos in October 2006, one in January 2007, four in February 2007, and two in April 2007. 

At the start of the year, the total number of views was about 620,000. Since the beginning of the year, customers viewed VSTS videos about 380,000 times, bringing the total number of views to more than one million.  Last October, we added the “VSTS” tag to help you identify the content on Channel 9 that is specific to Visual Studio Team System.

Supporting the Community Initiatives

Channel 9 provides a clear feedback mechanism in that you can leave replies to any posted video. From a quick review, customers have used this mechanism sparingly, compared to the number of views. Nevertheless, Channel 9 appears to be positively received overall by the community. 

To give an example of useful feedback, you can tell us where we “got it wrong” in a video or in the product itself. Perhaps we misunderstood a problem that you are trying to solve. After viewing the video, you might think “but that is not how I work,” and you can use the feedback mechanism to let us know. Or you might discover a more effective way to use a feature. When we hear from you, we can better ensure that complex features contain the functionality that you need, make them easier to use, or both. Documentation is nice, but customers often prefer to see an experienced person use a feature. A video might also confirm that you are doing something correctly but not getting the kinds of results that you expect and need. In that situation, you might have uncovered a product bug.

Final Thoughts

Overall, Channel 9 has helped us share information with the community and receive feedback. We should probably consider integrating Connect with Channel 9 so that customers can log bugs or suggestions from the videos.  As we work on the next release of Visual Studio Team System, I expect that we will publish a batch of new content on Channel 9.

MSDN Chats

MSDN chats provide an opportunity for the product teams to interact directly with the VSTS community in real time. You can ask questions, provide feedback, and voice complaints.  Internally, each product team tries to have representatives in each chat. Product team members can take ownership of a question or comment. Alternately, questions can be assigned to a specific person. We often take this approach when only one person is well suited to answer a particular question.

Progress against Goals

We tracked two metrics for MSDN chats. We wanted to average one chat during each calendar month, and we wanted at least 100 customers to participate in each chat. We chose the first metric because we want to ensure that product team and the community have regular opportunities to interact. We chose the second metric to help us assess whether enough customers were participating in the chats to warrant spending time on them (as opposed to other community activities).

We had 11 chats in 12 months, but four months (January, March, May, and June) had no chats. We averaged 229 attendees per chat, which well exceeded our target.

Other Takeaways

We were also “unofficially” tracking the number of questions asked per month to give us some indication of how interactive the chats were. A few were a bit slow, but most kept us all very busy answering questions.

Supporting the Community Initiatives

MSDN chats provide their primary value to the community as a feedback mechanism. We have averaged more than 120 questions and comments per chat. Anyone can pose a question directly to the people that create and support Visual Studio Team System.  Even if the answers are not always what folks want to hear (for example, that their feature request did not make it into the current release), they might still like to know that we did hear their request. Customers can also vent dissatisfaction with the product.

For members of the product team, it can be painful to take a verbal beating over a bug or a feature that does not meet customer requirements. However, it is also part of our job. The product group folks (me included) often find ourselves in a difficult position. We would like to address every request, but we must live within the bounds of the division schedule, so we make priority tradeoffs. We want your critical feedback because we can then better understand what your scenarios are and where we need to improve.

Final Thoughts

Overall, MSDN chats provide a great feedback mechanism for many members of of our community. Going forward, we are raising the goal to 150 participants per chat. We will do our best to ensure that we have at least one chat per month. We will also organize the chats a bit further in advance. That way, you all get more lead time to prepare questions, and we can better ensure that we have coverage from all of the VSTS teams.

Visual Studio Team System Advisory Council

We regularly meet with a group of customers known as the Visual Studio Team System Advisory Council. This group contains about 30 people from a variety of companies. The goal of the advisory council is to get early feedback from a representative subset of our customers about our plans and on early releases of our product. This initiative also helps us strengthen our relationship with our customers, which has some additional benefits. It provides us a pretty consistent set of reviewers who can look at a variety of different proposals in the context of other proposals that have been discussed. The advisory council can then help the product teams prioritize and make tradeoffs. Also, the reviews force the product teams to refine their plans enough that people from outside the team can understand them.

Progress against Goals

We have two goals for the advisory council: at least two sessions per month and at least 25 active participants in each session.  We just missed the first goal, averaging 22 active participants in each session.  We did average a little more than two sessions per month; a few months had three sessions. We have received good, direct feedback from the customers during in-person sessions. In addition, some good discussion takes place between sessions on the mailing list for the advisory council.

Other Takeaways

We do need to try to achieve a greater balance on the advisory council so that we have representatives from a variety of job roles and industries, as well as from different sized organizations. We will regularly rotate out the inactive participants to give more vocal folks a chance to provide their feedback. At last check, about 24 companies were waiting to participate, which demonstrates customer passion for the opportunity to help steer our future plans.

Supporting the Community Initiatives

The advisory council supports our community initiatives by providing a feedback mechanism (much like the MSDN chats). By providing direct, focused contact early in the product cycles, the advisory council has a significant impact on our priorities and plans. And, at the end of the day, product teams want to deliver a product that addresses the priorities of their customers.

Final Thoughts

The VSTS advisory council has helped inform many product team decisions, from feature prioritization to our approach to documentation. Members of the product teams very much welcome the early feedback (yes, even when it criticizes their plans). I hope the advisory council members feel as rewarded by their participation as the product teams do.

Looking Ahead

In Part 3, I’ll look at CodePlex, webcasts, and the Microsoft Connect site/mechanism. Then I’ll have just a few initiatives to cover in the fourth and final part. I hope that you are enjoying this look into how we view, measure, and adjust our interactions with the VSTS community.

 

-Steven

2