Connecting with Customers via Windows Error Reporting (Watson) and CEIP

Earlier this week, I provided a couple of glimpses at how product groups listen and respond to customers and partners. In the seven plus years I’ve been at Microsoft, I’ve seen lots of different ways how product teams listen and respond to customers. Previously, I worked along side the Engineering Excellence group, working on CPE across the product groups; now, I’m back in the Windows group… but I still work across product groups as they concentrate on improving customer satisfaction.

And core to the effort in most teams is the product and machine telemetry we get from millions and millions of customers through Microsoft Windows Error Reporting, the Customer Experience Improvement Program and software quality metrics (aka SQM, or ‘squim’). For more on SQM, see this prior post.

Windows Error Reporting dialog Windows Error Reporting (WER) is based on the technology that was originally known as Dr. Watson, the Windows program error debugger tool included in previous versions of Windows, as far back as Windows 3.0, and was included in Windows 98, Me, and XP.(Check out Raymond’s blog to understand why Windows Error Reporting is nicknamed “Dr. Watson”.)

WER captures software crash and hang data from end-users who agree to report it. You can access the data that is related to your applications online at https://winqual.microsoft.com. Gregg offers an overview on his blog on developers can use WER to debug what went wrong. In Windows, WER feedback technology is part of Microsoft Windows XP, Windows Vista, and Microsoft Windows Server 2003 and 2008 operating systems.

We know that the effort makes an impact: as outlined in the above article, our analysis shows that across all the issues that exist on the affected Windows platforms and the number of incidents received:

Fixing 20 percent of the top-reported bugs can solve 80 percent of customer issues.
Addressing 1 percent of the bugs would address 50 percent of the customer issues.

Considering the hundreds of millions of customers around the world, that’s a considerable impact.

Steve Ballmer outlined the effort in his 2002 letter on “Connecting to customers,” when he discussed how Microsoft can do a better job of serving our customers, and ultimately improve the customer experience through better communications and connections…

“The process of finding and fixing software problems has been hindered by a lack of reliable data on the precise nature of the problems customers encounter in the real world. Freeze-ups and crashes can be incredibly irritating, but rarely do customers contact technical support about them; instead, they close the program. Even when customers do call support and we resolve a problem, we often do not glean enough detail to trace its cause or prevent it from recurring.

“To give us better feedback, a small team in our Office group built a system that helps us gather real-world data about the causes of customers’ problems–in particular, about crashes. This system is now built into Office, Windows, and most of our other major products, including our forthcoming Windows .NET Servers. It enables customers to send us an error report, if they choose, whenever anything goes wrong.”

With WER, product groups get information that helps them identify the most common issues that our customers encounter on their computers, along with details that can help the teams debug the problem. If you’ve every run into a problem or application crash on your computer, chances are that you’ve seen the prompt to provide more information to Microsoft on the failure.

We also provide WER for Independent Software Vendors (ISVs) and Hardware Vendors (IHVs) on the MSDN forums, helping them find and correct issues with their application, ultimately leading to improvements in their products. These companies access our error reports database and discussion forums to address the errors and incompatibilities that creep up in their equipment drivers, software utilities and applications. (As noted on our sites, WER data is available to ISVs, IHVs and OEM and ODMs.)

Bill Gates said at PDC 2003…

“We allow anyone who has an application that runs on Windows to sign up and get the reports that relate to their application, and we’ve got winqual.microsoft.com where people can do that.

“Today we’ve seen a lot of that activity from the driver manufacturers, but we want to see even more at the application level so it gets us working together on anything where a user is not having a great experience.”

Coupled with the telemetry from WER is the feedback we get through the Microsoft Customer Experience Improvement Program (CEIP). Through these programs, hundreds of millions of customers help Microsoft improve upon the design, quality and features of our products and services. Participation in the CEIP is anonymous: when you sign up for the CEIP, information about how you use certain products is automatically sent to Microsoft, collected in the aggregate with CEIP data from other computers. Microsoft doesn’t use your collected CEIP data for any product marketing or promotions. (More details are provided in the Customer Experience Improvement Program Privacy Policy.)

According to the CEIP site…

CEIP collects information about how you use Microsoft programs and about some of the problems you encounter. Microsoft uses this information to improve the products and features you use most often and to help solve problems. Participation in the program is strictly voluntary, and the end results are software improvements to better meet your needs.

The Windows Server team uses the CEIP for their products, include Windows Server 2008. The team uses CEIP to improve the product in the following ways:

Helps to discover and fix software bugs in the operating system more quickly.
Helps to prioritize future Windows Server products and interim releases.
Helps to understand our customer system configurations more clearly, allowing Windows Server product teams to more accurately reproduce customer environments in their labs.
Helps to determine industry trends in computer hardware.

As John Song notes on the Dynamics CRM team blog, the team gathers information on how our customers are using the customization tool and how much they are customizing the out-of-the-box CRM.

“Through SQM/CEIP data, now we know that the Account is the most updated form in the CRM 3.0 (Well, that wasn’t a surprise for us, but the number of changes made was). A quarter of the CRM customers are adding good amount of new fields in the Account and other “main” forms.

It’s all about providing feedback to the products teams.

[Update: you can see the WSYP video here on YouTube: https://youtu.be/D28FkfJiauk]

When I’m asked about how this data is used, I often point people to my post “TwC’s site on Product Reliability… and WSYP” that has has a link to the WSYP Project, looking at how we leverage customer feedback in near real-time to improve software quality. (You can find this and other clips on the TechNet UK Spotlight page.)

And yes, Mauro is a real Microsoft employee. ; )

You can take advantage of WER to investigate the most frequently reported software crashes, resolve the problems, and inform your customers of the fixes, establishing a strong feedback loop with your customers.

Bookmark this on Delicious