Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Dare blogged about an interesting whitepaper from the Microsoft Experimentation Team about the concept of A/B testing and applications for websites.
"Practical Guide to Controlled Experiments on the Web: Listen to your Customers not to the HiPPO"
Note that HiPPO = Highest Paid Person's Opinion.
A/B testing (briefly, the concept of varying one or more settings/behaviors on a site for controlled subsets of users and analyzing resulting behaviors) was used extensively at my last two companies before joining Microsoft, so this was a good read. Section 5, Lessons Learned, is alone very worthwhile.
Recommended reading even if you aren't running a web application since the concepts could be applied to a variety of systems (but the inherent flexibility of websites makes them primary candidates for such testing.) Using A/B testing, you may be surprised to see how seemingly minor changes to a site can lead to major improvements in rates of use, clickthroughs, purchases, etc.
-Chris