The Beta vs. CTP Dilemma

Over the last couple of years, the CTP or Community Technology Preview has been one of the bigger changes in the way Microsoft release software.  Microsoft introduced the concept of CTPs with the “Whidbey” .NET Framework and Visual Studio release but I was happy to see the concept spread to SqlServer, Expression, WinFX and even Windows!    


CTPs have been very valuable to us as we have been able to get some great feedback in near real time from customers rather than waiting for a long Beta cycle or worse until after the product has been released to hear the feedback.


But, of course, CTPs have not been without their paint points… Frequent releases come with variable quality and are often incompatible with other pre-release software putting developers in a state of “CTP Madness” where it is often difficult to figure out what two builds work together.   JasonSu and I did create a cure – or at least a treatment – that I have heard helps a lot.  But the pain certainly does not go to zero. 


So, as we begin planning the interim release schedule for Orcas I find myself torn between the traditional value (and costs) of betas and the new value (and risk) of CTPs.  The downside of betas is that in order to drive up the quality we need pretty much all hands on deck working on the beta… if there are a few features that are not quite done, it is hard to keep those features moving forward during the beta lock down.  This effectively means we extend the release schedule by the Beta lockdown time.  We also attempt to “line-up” Betas of related products so that customers can use them together.  The plus side of betas is that because we drive the quality up and make them work together, conventionally wisdom is that more customers are able to try it out and get us feedback.     


But I wonder if we did a good job with quality throughout the cycle making the builds fairly usable as they come out, could we replace the two or even three beta cycle with less betas and more regular CTPs?  This of course only works if we get customers to give us the good quality, real-world feedback on CTPs…. if we don’t get the good feedback until very late in the cycle then the plan didn’t really save us anything. 


So what do you think?  Do “betas” offer you\your customers something specific that CTPs don’t?  Are you able to pick up the CTPs and try real deployments on them or is that something you can only do with Betas?

Comments (33)

  1. The only thing that did bother me was the incompatibility between VS and SQL2005.

    Giving us the opportunity to test out those products before release does help us to minimise the learning curve and deliver our products using this technology earlier. As we are giving feedback’s back to you guys also makes your product better. So keep the CTP’s with better co-ordination if needed between products.

  2. Mike Beeler says:

    CTP’s are generally a good thing. While you may not get the feedback in time to address something for the final public release, it does provide the development team with feedback they might not otherwise get.

  3. Michael Latta says:

    The key is going to be frequency.  The monthly or even every other month release cycle is too much churn.  I would go for something between the recent CTP and a full Beta.  If there was a new CTP every quarter that had enough testing to avoid any major issues that would be a good thing.  That cycle would give you time to get all things lined up to work together.  For example I ignored the recent Feb Vista/WinFX until late March waiting for EID.  I lost time in the month of Nov because I upgraded to the Nov WinFX CTP and there was no ink analysis compatible with that release and had to work on other things.  It is more important that the pieces all work together than that they come out so often.  The integrated Windows SDK rather than a bunch of separate SDK releases is the right idea.  Even if some component does not get updated (as in Jan WinFX getting no new WPF bits), the pretested compatibility is key to usability and developer investment.

  4. Sebastien Lambla says:

    The way I see it, quality in betas has not been substantially higher than the ones in ctp. Plus I’d rather have fairly unstable CTP on which i can upgrade my code to make it ready for the big day than have a separate branch cleaned up for quality but getting more and more distant from the actual code (and hence getting people to work on a version where the change delta is higher and higher as the CTP go forward).

    I see beta as being important only for major milestones where a go-live license is available, hence making whatever we write available for the public to try. That also means that go-live should exist much earlier in the process.

    That said this is only true for things like winfx, not really for visual studio. For that, i see no point in betas whatsoever.

  5. Me, I prefer the CTP over the

    Beta. It allows me to test items and give good information regarding…

  6. I agree with  Michael Latta that more time is needed between the CTPs, monthly is too often.

    I think a several quarterly CTPs followed by a Beta and then a Release Candidate would work.  The CTPs need some minimal testing, but not the full lockdown for a beta.

    And the Beta should be reasonably close to the actual release date (say <6 months), as I recall Whidbey Beta1 was more than a year before Visual Studio 2005 shipped.

    As part of this, get rid of the multiple Betas and multiple Release Candidates.  This also makes the definitions of "beta" and "release candidate" ring truer.  "Beta" means "we think we’re done, but you may still find some bugs", "release candidate" means "we’re ready to ship this puppy, but help us with a final sanity-check before we do."

  7. both are good, but i actually prefer Betas to CTP. my preference would be for you to release CTPs for the people that want to stay up to the minute; but still have the periodic Beta release for somebody that wants something a little more solid.

  8. T Rivers says:

    I like having both options. If MSFT is going more to a CTP release-cycle then please, at the very least, ensure a stable installation.  I’d rather spend my time reviwing the features of the product than trying to get it to install. Especially onto a virtual image.

    Go the CTP route, just make sure it installs well

  9. Personally, I don’t mind a CTP every 4-6 weeks. The DevDiv did a good job with CTPs, it’s the Windows division that doesn’t have a clue what they’re doing. CTPs are just public build drops, Betas are milestone releases.

    Keep an alpha, two betas and a release candidate, and put out CTPs whenever it is appropriate. And keep up the great work!

  10. IDisposable says:

    Having had access to BOTH all along, I must STRONGLY weigh in favor of CTP.  This allows much more open discussion and solution of problems inside the community, since you don’t have to worry if the "other guy" really is safe for NDA disclosure.

  11. I really liked the CTPs.  I like the frequency of the CTPs.

    I think that for frequent releases I would like to see some more work on the install story.  Midway through the WinFX ctps we got a "zap everything prior" tool that helped a lot.  I don’t know why the ctp installer couldn’t run this before hand.

    I also found it difficult because the CTPs come in a lot of pieces, and so their is a lot of work to get them in the right order.

    It seems a few days work would create an app that would 1) Unstall the prior CTP.  2) Download the new one, and 3) Install it with default options.  Thant way I could just leave my machine on overnight and be upgraded in the morning.

  12. I love CTP’s. If you have enough time and enthusiasm, you can learn a whole big new technology step by step. Someone watching Atlas monthlies, for example, will end up with a decent understanding of the big picture, while having enough time to play with the little details as they grow up.

    On the other hand, if you don’t follow close enough you may end up with a totally different thing than the one you have last check. This is true both for CTP’s as well as Beta’s though. There are still some people thinking that they have ObjectSpaces in 2005 and criticizing Atlas believing that it’s just Ajax enabled server controls collection.

  13. I prefer CTPs vs Betas, and a CTP every 30/40 days is ok for me, what I’d like is a better install/removal tools so that i don’t have to recreate a new machine every time, sometimes this process takes a long time (in special case if you don’t/can’t use a virtual machine)

  14. Personally I prefer CTPs to be more frequent than quarterly, in an ideal world every 6 weeks would be pretty good.

    Certainly the Whidbey beta process was MUCH better than the VS2003 process which felt like it was being driven by the need to get the Compact Framework in place and the bug triage process was being pretty brutal.

    Possible the most important bit of having CTP builds available is that if you report a bug that someone marks as "fixed in next release" you can at least test again on the next CTP rather than waiting for many months for the next full "beta" version.

    Most of us have PCs that can run a virtual PC for testing with so even the "uninstall" pain of a CTP version goes away.

  15. Brad Abrams, a Group Program Manager at Microsoft, asks for feedback regarding customers’ preference…

  16. I like CTP’s as they give us more looks at the progress of software.

  17. Peter Ritchie says:

    To be honest, they’re just terms.  Sure, they’re handled differently internally to MSFT; but to customers they’re the same thing: software distributed pre-release in the hopes of feedback about defects and usability.  With no real warranties/guarantees there’s no other reason to install/use these things.

    I’ve tried to be involved a several betas, and the impression I get from MSFT is they don’t want to hear any feedback.  IE 7 beta 1, for example, I could find nowhere to provide feedback directly to Microsoft.

    Call it "Beta" or call it "CTP" but be organized and make it REALLY easy to gather that priceless feedback about defects and usability.  And be consistent with gathering feedback; if it’s a "Beta" then always collect feedback and always in the same places–don’t have one beta that has a place to give feedback and another that doesn’t.  You’ve got an army of people willing to work without pay, in their spare time, with a specific focus and emotional commitment to the product–Microsoft should be falling over itself trying to use these people.

    Have a look at some of the bugs logged in Product Feedback.  FDBK44025, for example, has 13 people validating the bug, 18 people voting for it, and an average rating of 4.72 yet it’s resolved as "Not Reproduced".  That comes across as "we don’t want to hear it", reducing people’s motivation to give feedback.  There’s no way for  customers to query stats; but, a quick search for validated, "Not reproduced" maxes out at 100 where all have a rating of 5.

  18. Wagahai says:

    Go with CTPs. The product is unfinished, so the "variable quality" is to be expected. There is little that I can or expect to do in a beta over a CTP.

    Too many issues were postponed during the Whidbey cycle due to beta schedules. I do not mind frequent (montly or whatnot) CTPs. It is crucial to verify in a timely manner if specific reported issues have been fixed, and if not to redirect attention there. Maybe a beta a few months before the product is to be released is in order, but please focus on CTPs for most of the development cycle.

  19. Bob says:

    I am all for CTPs, but I might not be the best person to look at as an example.  I typically flatten my laptop every other weekend anyhow, so I am all about having the newest semi working bits installed 😉

  20. zzz says:

    If you ask me a 3 month delay between CTP is fine, however it might be nice if these CTPs got a week or two of "pre-ctp" testing incase there is some blocking install or other major issues.

    But what I’d most like to see is ability to log on to some virtual machine that is updated much more frequently and automatically. These could be used to see whether a bug fix actually did the job, by allowing the customer to upload their project to the virtual machine. After logoff the machine would reset to the state before the customer was there.

    That would be very good in testing whether for example a customer solution/project has had it’s problems fixed in latest build, without waiting for the next CTP to see that.

  21. Sessha says:

    In an ideal world, beta software (including CTPs) should not be installed on critical systems. However, many individuals do not have extra systems hanging around to utilize just for testing. That is one reason I think some people would rather wait for a beta than a CTP.

    In that regard, the news is reporting that Virtual Server 2005 is going to be offered for free. Maybe Virtual PC would be better, but perhaps the CTPs can be installed there. If that could work, it would provide an excellent means to encourage end-users to test the CTPs, as they could be removed easily without risking the rest of the system. This may be an avenue to explore.

  22. Julian Kay says:

    I think CTPs are really great. I learn so much more about a product by using CTPs and seeing the product evolve.

    Though I also think strong beta milestones are very important for people who don’t want to get ‘CTP Madness’.

  23. says:

    An number of people have commented on the install story. I’d like to see the uninstall story improved. That way, those of us who don’t have the luxury of machines that we can flatten periodically can play along more easily.  

  24. Dan Bartels says:

    Show me the tests…

    The CTP cycle is fine, but I would love to know what to expect when I take it out of the box (so to speak)  I would love to know in advance areas to expect to work and what will not…  

    So even if its not the unit test data, maybe just a feature matrix with a red / yellow / green; at least so I know when I am in a bug vs not done code…


  25. I’ll weigh in with support for the people that say that both are good. Frequent CTP releases of variable and unpredictable quality, and somewhat less frequent betas with a rather higher quality bar and more coordination with other teams to make sure that a given set of beta products all work together.

    Perhaps 2-3 CTPs per beta. Or more. Or less. Whatever works 🙂

    For me personally, I’ll try CTPs on products I *really* care about and want to see what’s going on as soon as I possibly can (at the moment that’s pretty much WPF and C#3/LINQ). Products that I’m *fairly* interested in and want to get a general idea of what’s happening, I’ll wait for a beta, assuming the option’s available (this, incidentally, is part of the reason I didn’t try SQL2005 prior to its release – they stopped doing betas in favor of CTPs only).

    And I agree with sam.jack about uninstall. Clean uninstallation is tremendously important for prerelease products. If it’s too hard to get that right for CTPs, that’s another reason for having betas as well.

  26. Sean Chase says:

    Go with having 2-3 betas. The CTPs were just a huge pain and I’ll probably just use betas from now on to try out new version of Visual Studio.

  27. Christoph richter says:

    for the most times, i install ctp’s on a test system (or virtual servers) and betas on my machines, or even my(privatE) live environment (as with asp.net2.0, yukon). the hardest problem for me is the upgrade process. when theres every month a new release, and it always takes hours to compelete (since update to another beta is allways not easy) its too much time for me. i like ctp’s very much, but there are also some bugs in it, that shouldn’t be there (and in most cases the "anoying" bugs, when you use it daily. not really the "big" bugs)

  28. Daniel Moth says:

    What you call them is a separate issue to what types there are.

    Unstable = play around for short(er) periods of time to uncover new features.

    Stable = as above + port existing applications, make sure they run, test/report bugs

    Stable + goLive = as above + start enhancing existing applications

    Released = as above + start brand new applications

    The frequency at which you provide the above and which type you provide at each drop is entirely up to you (given that you know what we do with each).

    Whatever you drop *please* allow make it uninstall cleanly.

  29. GRiNSER says:

    ctps are the best in my opinion because with this shorter release cycle i get versions with important bugfixes faster than need to wait for the next beta – specifically with winfx it’s great with ctps…

  30. I prefer CTPs, but if the GoLive licenses apply only to Betas, then I want a really good final beta.

  31. Oran says:

    I really like the CTPs and wish I had more time to play with them.  But unless there is a go-live license, I don’t feel it’s worth investing too much of my time.  Once there’s a go-live, I’m much more willing to spend time because I know I could use whatever I learn in production immediately.  Go-lives help me visualize _really_ using the product.

    More frequent CTPs get you closer to the open source model, but until users have the (usually unwise) option of using bleeding edge releases in deployment, you won’t get the same level of community involvement.  Just knowing that "I could use that if I wanted to" causes people to pay more attention.

    On a related note, our team is dealing with the same dilemma of "do we go with shorter releases or longer?"  Ironically, quality is the argument for both directions.  I’ve come to the same conclusion that you have about the importance of keeping quality consistently higher with shorter release cycles.  Not only does this keep us closer to a shippable product at any point in time, but it reduces the schedule risk of loose-ends-cleanup that happens with our longer releases.

    For us, the key is automating more of our rather costly system tests.  Those darn installer/uninstaller/upgrade/compatibility issues have really been killing us lately.

  32. This probably reflects poorly on me, but I don’t know the difference between Beta and CTP release.  When I am downloading either, I usually look for the latest release date and go with that.