The Windows Feedback Program

Introducing Christina Storm who is a program manager on the Windows Customer Engineering feature team working on telemetry.  

In a previous article Steven has introduced the Windows 7 Feature Teams. I am a program manager working on telemetry on the Windows Customer Engineering team. Our team delivers the Windows Feedback Program, one of several feedback programs in place today that allow us to work directly with customers and make them part of our engineering process.

The Windows Feedback Program (WFP) has been active for several years during the Windows XP and Windows Vista product cycles, and we are currently ramping up to get all aspects of this program ready for Windows 7. At the core of this program is a large research panel of customers who sign up via our website https://wfp.microsoft.com during open enrollment. Customers choose to be part of a survey program, an automated feedback program or both. They then complete a 20-minute profiling survey, which later allows us to look at their feedback based on their profile. We have customers spanning a wide spectrum of computer knowledge in our program, and we are constantly working on balancing the panel to staff up underrepresented groups. The majority of customers who are spontaneously willing to participate in a feedback program like ours are generally enthusiastic about technology. They are early adopters of consumer electronics, digital devices and new versions of software. In contrast, customers who see the PC as a tool to get a job done tend to be a bit more reluctant to join. And we also need more female participants!

Customers who participate in the automated feedback program install a data collection tool developed by the Windows Telemetry Team. The privacy agreement of this program describes the data collections our tool performs. Here are a few examples:

  • Windows usage behavior including installed and used applications.
  • File and folder structures on your computer, including number of types of files in folders, such as number of jpg files in the Pictures folder.
  • System-specific information, such as hardware, devices, drivers, and settings installed on your computer.
  • Windows Customer Experience Improvement Program (CEIP) data.

From the collected data we create reports that are consumed by a large number of Windows feature teams as well as planners and user researchers. This chart below shows the answer to the following question: What is the most common file type that customers who participate in our program store on their PCs and what are the most popular storage locations?

Graph showing common file types and locations. The most common file type are .jpg across all typical locations.

The results help us both with planning for handling the volumes of data customers store on their PCs as well as mimicking real-life scenarios in our test labs by setting up PCs with similar file numbers and file sizes and distribution of files on the PCs.

These data collections furthermore allow us to create reports based on profiled panelists. The above chart may look different if we created it based on data delivered only by developers and then compare it to data delivered only by gamers, just to name a couple of different profiles that participate in our program. The Windows knowledge level sometimes makes a difference, too. Therefore it is very important to us that customers participate in this program who consider themselves Windows experts as well as customers who don’t enjoy spending too much time with the PC, who just see the PC as a tool to get things done. Based on the data, we may decide to optimize certain functionality for a particular profile.

In general, we utilize this data to better understand what to improve in the next version of Windows.  Let’s take a look at how the representative sample has their monitors configured.  First what resolutions do customers run with on their PCs?  The following table lists typical resolutions and the distribution based on the Windows Customer Experience Improvement Program, which samples all opt-in PCs (Note the numbers do not add to 100% because not every single resolution is included):

Distribution of common screen resolutions. Approximately 46% of customers run with 1600x1200 and 1280x1024. Approximately 10% of customers run with HD resolution.

One thing you might notice is that about 10% of consumers are running with HD or greater resolution.  In some of the comments, people were asking if our data represented the “top” or “power users”.  Given this sample size and the number of folks with industry leading resolution I think it is reasonable to conclude that we adequately represent this and all segments.  This sample is a large sample (those that opt-in) of an extremely large dataset (all Windows customers) so is statistically relevant for segmented studies,

We have found that a large percentage of our program participants lower their display resolution from the highest usable for their display. Looking at the data coming from the Windows Customer Experience Improvement Program to compare to, and noticed a similar trend: over 50% of customers with 1600x1200 screen resolution displays are adjusting their resolution down to 1024x768, likely because they find it uncomfortable to read the tiny text on high resolution displays. The negative effect of this resolution change is the loss of fidelity to the point where reading text in editors and web browsers is difficult. High definition video content also won’t be able to render properly.

Here is the data just for customers with displays capable of 1600x1200:

Actual running resolution for customers with 1600x1200 capable displays shows that 68% of customers reduce their actual screen resolution.

In a future blog post, one of the program managers from the Windows Desktop Graphics team is going to describe what we have done with that information to improve display quality and reading comfort in Windows 7.

We also frequently use our data to select appropriate participants for a survey. A researcher may be interested in sending out an online survey only to active users of virtual machine applications. We would first determine that group of users by looking at our “application usage” data and then create the list of participants for the researcher. Sometimes we combine automatically collected data with survey responses to analyze the relationship between a customer’s overall satisfaction and their PC configuration.

At the current point in time, 50% of our panel participants are using Windows XP and 50% are using Windows Vista. We are currently not offering open enrollment. If you are interested in being invited to this program, please send an email to winpanel@microsoft.com indicating “Notify me for enrollment” in the subject line. If you’d like to add a bit of information about yourself, including your Windows knowledge level, that would be much appreciated! We will add you to our request queue and make our best effort to invite you when we have capacity.

When we release the Windows 7 beta we will also be collecting feedback from this panel and asking for participation from a set of Windows 7 beta users. Our current plans call for signing up for the beta to happen in the standard Microsoft manner on https://connect.microsoft.com. Stay tuned!

-- Christina Storm