Note that HiPPO = Highest Paid Person’s Opinion.
A/B testing (briefly, the concept of varying one or more settings/behaviors on a site for controlled subsets of users and analyzing resulting behaviors) was used extensively at my last two companies before joining Microsoft, so this was a good read. Section 5, Lessons Learned, is alone very worthwhile.
Recommended reading even if you aren’t running a web application since the concepts could be applied to a variety of systems (but the inherent flexibility of websites makes them primary candidates for such testing.) Using A/B testing, you may be surprised to see how seemingly minor changes to a site can lead to major improvements in rates of use, clickthroughs, purchases, etc.