Adding to, but not Perverting the Beta Process


Today someone forwarded me a post entitled “Beta Perversion” written by Rick Strahl, one of our MVPs.  I’m going to have to disagree with some of the assertions he made in this post about our abilities to react to broad customer bug feedback, but at the same time admit that I can see how he may have gained this perception through our actions as we learn how to become open earlier in our development cycles.

In the opening paragraph he states…

“In the past, beta testing was the idea of final testing of a product by end users (who in this case are developers) – usually a selective sample of the end users along with people who have been selected to provide quality feedback and have a deep understanding of the tool. By the time a beta rolled around the product generally was pretty much feature complete and in a fairly stable state of operation, and approaching the release cycle.”

Your right, in the past Microsoft betas have worked this way. This is not the case across the industry and especially not the case in the open source world. I’ll also make the claim that as products become more complex and the stabilization phase of software increases the old model starts to break down and the old model no longer affords developers enough time to make substantial changes if they need to occur.  I personally saw this with several features I owned as a tester in the VS 2002/2003 release cycles. It was very frustrating to see bugs come in after we released a beta and not be able to do anything about most of them. In 2003, for example, we didn’t provide any response to over 90% of issues raised by our Beta customers. The old model was officially broken.

“With Visual Studio .NET 2005 …Microsoft has turned the beta process into little more than a public marketing program. Microsoft’s beta/marketing process starts as soon as a product ships and immediately pushes the next technology coming.”

Ouch. There is something to be said about hyping future products too soon. If someone has this perception then I don’t believe we struck the right balance between gathering feedback from Day 1 of a product cycle and making sure customers are focused on the existing products. Most of the criticism I’ve seen that relates to this has to do with all of the preview material that was posted to existing product developer centers.  Thus confusing customers who would go to MSDN to look for a VS 2003 answer. Yup, I think we goofed there and have worked to correct it be trying to limit the 2005 related news to the vs2005 developer center.

The early alpha/beta/CTP program information and preview information needs to be isolated better in the future.  Not made private again, but more obviously separated.  Though I honestly can’t blame the product team members for hyping whidbey on their blogs. It’s what they are excited about! The funny thing is that I think marketing had less to do with this perception than the product teams being told they could be more open and then being publicly excited about what they where working on. It’s a period of adjustment for us and customers as we learn how to be more open.

“Beta is now a year long or longer process of a product that is still in heavy flux. This means that the purpose of beta – to test a fairly complete version of a product – is not really being served at all.”

No reason there still can’t be an end game beta that serves this purpose. We’ll do this too.

“Instead Microsoft uses the beta to see what features need to be added, removed or modified due to public consensus while not really handling the process of bug fixing based on customer feedback.”

This is the first time I’ve heard someone say it’s a bad thing that we are finally giving people a chance to play with a product early enough to effect the end result. I would also disagree with the statement that we “don’t really handle the process of bug fixing based on customer feedback”.  The statistics from the early betaplace and Product feedback centers tell a different story. On the feedback center alone almost 6000 bugs from customers have been filed against us and these bugs are being fixed at the same rate with similar numbers to bugs that are filled internally. Customers now have a way to raise bugs against our pre-release software with the same voice as the internal developers in testers.  I’ll go as far as to say that customers are getting better responses than internal testers on the issues they raise… and that’s a good thing. Also, 6000 bugs doesn’t generate as much noise internally as you might imagine.

“A while back when the public beta was released Microsoft actually closed the ‘official’ beta newsgroups and opened up public general access newsgroups for this process. While the public groups are a good thing to allow the greater public access to information Microsoft also stopped taking responsibility for anything posted there. In fact, Microsoft never really has done even a remotely decent job of dealing with feedback that has been posted on the beta newsgroups and message boards.”

The official beta groups you where referring to where closed because they represented duplication. The “private” testers (even with more recent bits) where asking the same exact questions as the public testers. We made a choice and that choice was to lean on the more public groups because the information was relevant to everyone. Regarding our poor levels of participation in these groups.  I think we are guilty as charged in some areas, but the numbers (that I’ll post next week), aren’t any worse in the semi-public whidbey groups than they were in the private ones. There are some newsgroups that get good responses and there are some where we are 0 for 26 in the last 30 days. In those cases… we suck right now. Its that simple.

“Making a beta public and then removing direct beta involvement for ‘official’ beta testers is also a bad idea. While a public beta provides potentially many more testing sites, it also increases the signal to noise ratio dramatically with unqualified feedback or feedback that is so obvious or previously handled that it has no bearing.”

You’d be right… if our only feedback mechanisms where public betas. The fact is that we still have a very large early adopter program with individuals and companies that have even more direct communication with the teams building the product. The 1:1 lines of communication here ensure that their feedback is heard about the “noise” of the public feedback.  Our usability teams have also stepped up their efforts to connect directly with a smaller group of customers to attain their feedback.  Its just that these programs are just as you expect them to be… more private.

The MVPs used to get early releases that made them part of a more elite club. We have diluted this. There is not currently as much “special attention” paid to MVP feedback compared to public feedback. What needs to happen, in the future, is that more teams need to act like the Foxpro team and treat the MVPs (or best customers) as part of the product team.  These customers should have the opportunity to be reading our specs for Orcas soon after they have been written and ever before any code has been produced. These documents and the opportunity to provide this extremely early feedback should not be publicly available, but our first working code drop might be. :-)

“For a model of what a beta process should look like the VS.NET teams should review how the Visual FoxPro team runs its betas. VFP is handling most bug reports directly through the newsgroups rather than a ‘bug reporting’ system that sucks.”

I’ll disagree here. The foxpro model works because of the limited scope of the product, the customer base, and small internal team size.  Yes, the Foxpro team all have their hearts in the right location (to make customers feel connected to the process), but some of the methodologies don’t scale well outside of the small focused product and team. It was becoming a nightmare to track bug requests and responses in the newsgroups across the product.  I might read one, forward it to the right team, but then as the problem is shuffled from developer to developer (it may take several teams making a change to fix), the response to the customer would ultimately be lost. There is no additional meta-data attribution on the newsgroups to know when a bug (disguised as a newsgroup post) is reactivated, or what version of the product it may have been fixed in for verification. The process falls apart when scaled out (I’m also making the assumption that we are going to continue to scale our feedback out to catch a wider net of scenarios.)

The product feedback center has some issues that, if addressed, could make it 100x better. If the firefox team can handle their bug influx from 10 million customers in bugzilla, then I imagine we’ll be able to scale to our Visual Studio developers with the right tool. I make no claims that the Feedback Center is perfect today or that it will be tomorrow. Only that we will be dedicated to improving the process of handling customer bug submissions.  While I can’t give exact numbers… trust me, the number of customer bugs submitted is not anywhere near the number of bugs that are submitted internally. We will be able to keep up for a long time.

Again, this is not to say that newsgroups and forums do not have a purpose. I just don’t believe that purpose is to be a public (or even large private) bug database. They are simply a mechanism to carry on discussions with customers.

“It’s kind of ironic how Microsoft is advocating distributed, atomic systems, yet VS.NET of all things is the epidemy of a monolithic Windows application that is tightly coupled to every part of the operating system, IIS, SQL Server 2005, heck even the .NET version. If there’s one application that’s most likely to hose your system on install or uninstall it is Visual Studio!!!”

I won’t disagree with your statements here. Sure, if we started fresh we would do things differently, but you always have to weigh that against the cost of simply scrapping a lot of legacy code in the IDE that has worked just fine for a very long time. Currently there are teams working on how to improve our engineering excellence so that we can start to make progress on this front in the years to come. It is not a switch one can simply throw.  I’m personally scared to attempt an uninstall and re-install of a daily build without re-imagine my machine. This is something that needs to be addressed. Unfortunately it won’t happen tomorrow.

“I’m all for Microsoft taking their time getting it right and making sure the product is as good and stable as it can be. This is priority One. But I can’t agree with the policy of over-hyping the new technology so early on when it distracts developers so significantly with what at this point is little more than vaporware that’s diverting valuable resources (time!) from developers who should be getting work done … “

I’m not sure which build of the product you are using, but if we simply only labeled the later releases as a “beta” it seems that a lot of your concerns would go away.  In the end it really sounds like your concerns are ones that could primarily be addressed with proper messaging rather than serious changes to how we are opening up the process earlier in the release cycle.

Comments (16)

  1. I’m not quite sure what this means:

    "The early alpha/beta/CTP program information and preview information needs to be isolated better in the future. Not made private again, but more obviously separated."

    Isolated as in, still 100% publically available, just not blasted out to everyone? If so, right on! Rick’s not the first person to complain about all the "vNext" content out there (mainly regarding Longhorn).

    I think part of the "problem" are that a lot of people are now reading blogs, and of course, people are gonna talk about new products (or about what they’re currently working on).

    Just so long there’s not much "mainstream" publication of *really* early features, I think it’s fine. I think the people who can give the best feedback early on are also the people who can cut thru a bit of _insulation_ to get to the CTPs.

  2. Josh Ledgard says:

    You translated my intentions correctly. Yes. There is a need to improve our communication about alpa/ctp/beta releases. It needs to be more seperated from existing communications but also richers in context so that it is easier for customer who do want to be involved to understand what they are getting into.

  3. CTP is fine. Calling a CTP a beta when it is really still a CTP does cause confusion.

    You say that in the past you called things betas when you thought they were ready, but in the past some versions of Windows were called betas when they should have been called CTPs.

    When a product really does reach beta and you think it’s ready, but customers find problems you didn’t find, the facts are the exact opposite of "can’t do anything about it", the facts are "better do something about it". (Except of course for monopolies who don’t have to do anything about it.)

    Sure if you want to release a version on schedule even after latent bugs become known bugs due to feedback, you can still release it, just call it a beta instead of calling it 1.0. If you spend the next year developing a service pack to fix a beta instead of a service pack to fix 1.0, fine. (When your company charges upgrade fees for things that are really bugfix packs, of course your victims scream bloody murder, and only monopolies can get away with it.)

  4. Joku says:

    I think you are on the right tracks here. What comes to the "insulation" proposed, my suggestion would be some sort of extension to blog categories perhaps. When you are making a new blog post, you would choose that this is about a future product (category) and then the post would be go to similar place as blogs.msdn.com, but on the appropriate product center. Essentially gathering the talks and information about upcoming product in the vs2005 center for example. There could also be additional blogs.msdn.com feed that would include all the future product "hype", for those who want it.. However I would believe everyone would move to using this feed then.

  5. I think my problem with the current process, especially with how external the whole process has become, is that there are too many public builds available and too many public builds that you can report bugs on (There are what look like 7 Visual Studios you can report on, not including the Express Versions). And I think that encourages the testers to sign off with "Fixed in a Later Build" when the issue may be "Can’t Replicate"

    For example, I’ve recently started working with the Nov CTP. (I tried Dec CTP, but it was pointed out that it was an older Framework, and I wanted newer framework, even if it meant some older tools, or not the Architect VS) Since then, I’ve reported four bugs, two of which have been closed as "does not reproduce" or "You will see this behavior in a later build." One of them was a pretty serious bug that I’m suprised made it into the Dec CTP, was already better into the Nov CTP, so I’m almost not suprised if it’s fixed (http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=8c6eab30-28f1-4996-8376-18978c436e4d), but the other one is almost esoteric. It’s a Windows Forms issue that only effects Multi-monitor setup, and Form Startup (http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=ade38c35-50e1-45b0-858e-e539cd23624e) It’s been there since Beta 1 of 1.0 (Dec of 2000, which I should’ve reported it), and I find it hard almost hard to beleive that it’s been fixed in the last two months (although it would be nice) as opposed to not happening on on certain setups.

    I guess that part of the more open process should be a better statement on position. Should Microsoft accept bugs on every "beta" version they’ve released, or close off some of the older ones and only accept bugs on newer ones since there always is a chance an issue has been fixed. Also, when a bug can’t be replicated on the "latest" internal build, it would be nice to know that it had been replicated on the older build under the same circumstances (or that there was an actual internal bug fixed.) I’d like to be a part of the beta process, but it’s difficult when I’m shooting a moving target, and I’ve now no idea where it is…

  6. AT says:

    I think that there must be applied tiny correction to Josh statements:

    Making Product Feedback database publicy accessible is not a very big deal.

    The biggest problem is how this feedback will be used and replies provided.

    I know that Microsoft VS.NET teams did a _good_ job in the past (I was in Everett 7.0, dunno about more old) handling private feedback.

    This were improved a little bit since Everett, but I feel that this is no way related to PF database. I _feel_ that this was possible to change something internaly – without making everything public.

    Several years ago I’ve proposed Windows teams to open bug report content to same limited beta group only. Unfortunately – Microsoft decided to make them open to _all_ the people.

    Instead of test-driving Product Feedback _website_ on limited number of participants – website still in "beta" quality for about a 9+ months. This is not good.

  7. I recently joined Developer Division as a Program Manager.  One of my goals is to improve the customer…

  8. I recently joined Developer Division as a Program Manager.  One of my goals is to improve the customer…

  9. I recently joined Developer Division as a Program Manager.  One of my goals is to improve the customer…

  10. Billy Tan says:

    Hi board,

    Q1. Any idea how to do integrates software configuration management (SCM) with bug-tracking.

    For example, glue MS Visual Source Safe with any bug-tracking system (such as Bugzilla and Mantis or MS product – which i dun think MS have it now). Please help me this….. (email me at billytcj@tm.net.my)

    I just thinking, if user have the change to glue above mentioned items, they will have change involving the initial "drive" of software design mind-set. May be not much influence to original design, but at least they have a "way" to understand the difficultly or effort behind what they seen. From my exeperiences is … people tend to raise thier "thought" when they not know how it working or design on it. Well… i believe "open_to_all_people" is a way to make the entire product more "ideal" during the release day, don’t u think that?