Testing in Production (TiP) – It Really Happens–Examples from Facebook, Amazon, Google, and Microsoft

  Testing in Production (TiP) really happens, and yes that it s a good thing! Here are some examples from Facebook, Amazon, Google, and Microsoft. The Methodologies This is not a complete list of TiP methodologies, but explains the ones used in the examples given A/B Testing (aka Online Controlled Experimentation): I explain this in…


Exposure Control: Software Services Peep Show

  It’s About Software Quality Not sure what you were expecting here at Your Software Has Bugs, but ignore that picture on the left, and learn how Exposure Control let’s you roll out dangerous software while minimizing risk to your users by giving them only a peep into the your new service. peep n.  a…


Hooray for Buggy Software???

No this is not a piece on job security for Software QA Professionals. I was recently at a Test Team sprint-end demo. For at least two of the projects shown, the metric of interest to Dev managements was, “how many bugs did you find?” and when the answer was either a high number or a…


Testing with Real Users – User Interaction and Beyond, with Online Experimentation

My talk at the Better Software Conference was a lot of fun.  It was well received.  If you want to check out the slides, take a look here. Evidence shows than more than half of the ideas that we think will improve the user experience actually fail to do so—and some actually make it worse….


Software Testing Cage Match: Amazon.com vs. Microsoft

While I previously made some comparisons between Amazon.com and Microsoft’s different approaches to software testing in Building Services: The Death of Big Up-Front Testing (BUFT)?, I think now would be a fun and interesting time to do a deeper dive on this. Before I joined Amazon.com in 2005 as an SDET, while I was interviewing…

5

I’m Speaking At the Better Software Conference – June 9, 2010 in Las Vegas

  Testing with Real Users – User Interaction and Beyond, with Online Experimentation Evidence shows than more than half of the ideas that we think will improve the user experience actually fail to do so—and some actually make it worse. Instead of guessing, why not measure what your real users like and don’t like? Controlled,…


Is Chuck Norris on your software engineering team?

If I need to defeat a group of men terrorizing a circus family, then Chuck Norris is the man to call.  The tough guy, martial artist, action hero has become an internet meme – Chuck Norris Facts attest to the true bad-a**-ness of the one called Chuck.  For example one of my favorites is: There…


Building Services: The Death of Big Up-Front Testing (BUFT)?

 Let’s start with some definitions: Death Demise, end, termination, the state of no longer existing BUFT “Big Up-Front Testing” an activity or “Big Up-Front Test” a group that carries out that activity (the latter sounds cool, like “Big Pharma”) It’s an acronym of my own invention. At least I believe this to be the case…

2

Service Deployments Should be Ramps, not Cliffs

[Going to try a mini-post…keeping it short and focused] In my previous post I introduced you to Online Experimentation.  The general use case for experimentation is to take two or more websites or services, each with different features or designs, and test them against each other using real users.  It is assumed that all the…


Feeling TiPsy…Testing in Production Success and Horror Stories

One topic I hope to return to many times in this blog is that of TiP, or Testing in Production.  For a good introduction to this check out Ken Johnston’s blog entry TIP-ING SERVICES TESTING BLOG #1: THE EXECUTIVE SUMMARY which is an updated version of the executive summary of the ThinkWeek paper that I…

3