The Beta vs. CTP Dilemma

Over the last couple of years, the CTP or Community Technology Preview has been one of the bigger changes in the way Microsoft release software. Microsoft introduced the concept of CTPs with the “Whidbey” .NET Framework and Visual Studio release but I was happy to see the concept spread to SqlServer, Expression, WinFX and even Windows!

CTPs have been very valuable to us as we have been able to get some great feedback in near real time from customers rather than waiting for a long Beta cycle or worse until after the product has been released to hear the feedback.

But, of course, CTPs have not been without their paint points... Frequent releases come with variable quality and are often incompatible with other pre-release software putting developers in a state of “CTP Madness” where it is often difficult to figure out what two builds work together. JasonSu and I did create a cure – or at least a treatment – that I have heard helps a lot. But the pain certainly does not go to zero.

So, as we begin planning the interim release schedule for Orcas I find myself torn between the traditional value (and costs) of betas and the new value (and risk) of CTPs. The downside of betas is that in order to drive up the quality we need pretty much all hands on deck working on the beta... if there are a few features that are not quite done, it is hard to keep those features moving forward during the beta lock down. This effectively means we extend the release schedule by the Beta lockdown time. We also attempt to “line-up” Betas of related products so that customers can use them together. The plus side of betas is that because we drive the quality up and make them work together, conventionally wisdom is that more customers are able to try it out and get us feedback.

But I wonder if we did a good job with quality throughout the cycle making the builds fairly usable as they come out, could we replace the two or even three beta cycle with less betas and more regular CTPs? This of course only works if we get customers to give us the good quality, real-world feedback on CTPs.... if we don’t get the good feedback until very late in the cycle then the plan didn’t really save us anything.

So what do you think? Do “betas” offer you\your customers something specific that CTPs don’t? Are you able to pick up the CTPs and try real deployments on them or is that something you can only do with Betas?