Tell Us What You Think

  • Article

We love to hear your feedback and given that our final beta for Visual Studio 2010 is now available this is your best opportunity to influence the product. In Microsoft Test and Lab Manager(MTLM) we are using the Microsoft Customer Experience Improvement Program (CEIP) to gather application usage data. CEIP collects information about how our customers use Microsoft programs and contains no personally identifiable information. We use this information to improve the products and features customers use most often and to help solve problems.

Participation in the program is voluntary, and for beta release we automatically enable sending this data. Click Home –> Options

MTLM options dialog for CEIP opt-in

It has been a few weeks since folks have started using beta 2 and we are already seeing some good information come thru. Following are some of the metrics that we are collecting:

1) MTLM Usage

a. Most used views (~1000 launches): Run Tests, Plan Contents, Workitem Editor and Test Runner

b. Less frequently used views (~200 launches): Manage VM Template, SaveAs VM Template, Environment Editor, Recommended Tests

c. Views not launched at all: Configuration Editor, About and Settings dialog

2) Actions performed

a. Most folks navigate Back and not Forward (336 vs. 22 invocations of these actions respectively)

b. Docking commands are not used frequently, of 787 invocations of Test Runner, docking behavior was changed only 8 times

c. Help has 394 invocations.

d. Home button had 300 invocations.

3) Artifact Counts (test cases, plans, runs)

a. Artifact counts are < 99 for most items (plans, runs, configurations)

b. 3% users already have between 100 - 500 test cases

c. 10% users ran queries that returned more than 100 workitems

Some of the changes we are considering based on the data are mentioned below.

We saw that Test Plan Manager was launched 766 times but users opened Test Plan Editor from there only 90 times. Could this mean Test Plan Manager may not make sense as the first view in Organize group? The other views on the Organize group in order of preference were: Manage Configurations, Manage Test Cases, Manage S3s so should we elevate one of these to the first view?

Users find  the Go To Work Item functionality in TFC very useful so we provided the CTRL+G shortcut in MTLM too. But this has been invoked only once! We need to consider how we can make this more discoverable.

As time progresses we see more usage patterns and derive useful information to make MTLM more user friendly, but in the meantime your direct feedback is the best way for use to know what you think!

Regards,

Abhishek Lal