With my post from earlier in the week as a background, I wanted to share with you how the Windows SDK is collecting CEIP-conformant data on usage of our product, and what our future plans are around gathering data.
We’ve separated our implementation of CEIP into two phases . The first phase will be implemented in the upcoming full (RTM) release of the Windows SDK for Windows Vista. In this phase, we’re collecting two pieces of user data around setup: How well the installation itself goes, and what custom install options you choose.
To test the quality of setup, we take two data points. First we get a datapoint that setup starts and then we get a second datapoint indicating whether it completes. These datapoints are built on top of existing code for ease of implementation. For instance, our setup application already logs 13 different ways that a setup can complete. We simply take note of those conditions and upload them to the CEIP database. Among those conditions are:
- ERROR_SUCCESS = 1
- ERROR_PRODUCT_INSTALL_FAIL = 2
- ERROR_INVALID_ARGUMENTS = 3
- ERROR_MISSING_DEPENDENCY = 4
- ERROR_INSUFFICIENT_PRIVILEGES = 5
A small binary file containing that information is created on the user’s %temp% folder as setup runs. At the close of setup, all users are shown an opt-in CEIP check box . If users leave the box as unselected, the information is simply purged from the %temp% folder. If the user selects the box, the file is uploaded to CEIP’s secure server and then is discarded. In either case, the content is purged from the user’s machine.
In the database, each of those datapoints are logged as discrete bits of data. From that data, we are able to cross-reference the information and begin to be able to answer some questions that we’ve wondered about for a long time. For instance, from the 13 points mentioned above, we can answer questions like “how many of our users who start install complete the install?” or “how many of our users have a network failure on install? Right now we only have anecdotal evidence to give us that information. These datapoints will give us real world information that will us understand the sorts of problems that users are running into and allow us to build a better setup. (Though, of course, we hope that most of the results show that users are getting ERROR_SUCCESS, which I think is my favorite error message ever.)
The second half of our data collection project for our upcoming release of the SDK is to track the state of each of the custom install options our users choose. We give users about sixteen different options on the Install Options screen of setup, but we don’t know how many users are actually choosing those options. Do most users simply take the default settings, or do they like to pick and choose?
Each of these datapoints are being collected so that we can drive better implementation decisions in the future. If most of our users are selecting all custom setup options, or if we find that many users are deselecting FxCop or selecting to install the .NET Compact Framework, that will allow us to make appropriate decisions about what is selected by default in the future to meet the needs of the majority of our users. Sure, the numbers will still be affected by the number of users who opt in to sending their data, but at least we’ll be getting a much broader amount of real world user data than we have been.
For the post RTM phase, we’re still trying to decide what questions to track. The driving factor in asking these question is to allow the SDK team to determine where to focus our energies.
We also are wrestling with the question of how to get information on more issues than just setup. Setup is a good first place to gather information since the SDK team controls all of our setup code, but it would be great to be able to track information on how you use other parts of the Windows SDK. We still have a lot of work to do around choosing approaches in those areas before we decide what to do there.
We’re really interested in your feedback, and we believe we can build a better product if we have data about how the Windows SDK is being used. Please comment to this blog if you have questions or feedback about CEIP or how the Windows SDK implements it. I hope at a minimum that you’ll consider clicking the checkbox on the final install screen that says “Help improve our products by sending your installation report to Microsoft. No information collected is used to identify you or contact you.” You can influence the future direction of the SDK simply by clicking one little checkbox.