Today someone forwarded me a post entitled “Beta Perversion” written by Rick Strahl, one of our MVPs. I’m going to have to disagree with some of the assertions he made in this post about our abilities to react to broad customer bug feedback, but at the same time admit that I can see how he may have gained this perception through our actions as we learn how to become open earlier in our development cycles.
In the opening paragraph he states…
“In the past, beta testing was the idea of final testing of a product by end users (who in this case are developers) – usually a selective sample of the end users along with people who have been selected to provide quality feedback and have a deep understanding of the tool. By the time a beta rolled around the product generally was pretty much feature complete and in a fairly stable state of operation, and approaching the release cycle.”
Your right, in the past Microsoft betas have worked this way. This is not the case across the industry and especially not the case in the open source world. I’ll also make the claim that as products become more complex and the stabilization phase of software increases the old model starts to break down and the old model no longer affords developers enough time to make substantial changes if they need to occur. I personally saw this with several features I owned as a tester in the VS 2002/2003 release cycles. It was very frustrating to see bugs come in after we released a beta and not be able to do anything about most of them. In 2003, for example, we didn’t provide any response to over 90% of issues raised by our Beta customers. The old model was officially broken.
“With Visual Studio .NET 2005 …Microsoft has turned the beta process into little more than a public marketing program. Microsoft’s beta/marketing process starts as soon as a product ships and immediately pushes the next technology coming.”
Ouch. There is something to be said about hyping future products too soon. If someone has this perception then I don’t believe we struck the right balance between gathering feedback from Day 1 of a product cycle and making sure customers are focused on the existing products. Most of the criticism I’ve seen that relates to this has to do with all of the preview material that was posted to existing product developer centers. Thus confusing customers who would go to MSDN to look for a VS 2003 answer. Yup, I think we goofed there and have worked to correct it be trying to limit the 2005 related news to the vs2005 developer center.
The early alpha/beta/CTP program information and preview information needs to be isolated better in the future. Not made private again, but more obviously separated. Though I honestly can’t blame the product team members for hyping whidbey on their blogs. It’s what they are excited about! The funny thing is that I think marketing had less to do with this perception than the product teams being told they could be more open and then being publicly excited about what they where working on. It’s a period of adjustment for us and customers as we learn how to be more open.
“Beta is now a year long or longer process of a product that is still in heavy flux. This means that the purpose of beta – to test a fairly complete version of a product – is not really being served at all.”
No reason there still can’t be an end game beta that serves this purpose. We’ll do this too.
“Instead Microsoft uses the beta to see what features need to be added, removed or modified due to public consensus while not really handling the process of bug fixing based on customer feedback.”
This is the first time I’ve heard someone say it’s a bad thing that we are finally giving people a chance to play with a product early enough to effect the end result. I would also disagree with the statement that we “don’t really handle the process of bug fixing based on customer feedback”. The statistics from the early betaplace and Product feedback centers tell a different story. On the feedback center alone almost 6000 bugs from customers have been filed against us and these bugs are being fixed at the same rate with similar numbers to bugs that are filled internally. Customers now have a way to raise bugs against our pre-release software with the same voice as the internal developers in testers. I’ll go as far as to say that customers are getting better responses than internal testers on the issues they raise… and that’s a good thing. Also, 6000 bugs doesn’t generate as much noise internally as you might imagine.
“A while back when the public beta was released Microsoft actually closed the ‘official’ beta newsgroups and opened up public general access newsgroups for this process. While the public groups are a good thing to allow the greater public access to information Microsoft also stopped taking responsibility for anything posted there. In fact, Microsoft never really has done even a remotely decent job of dealing with feedback that has been posted on the beta newsgroups and message boards.”
The official beta groups you where referring to where closed because they represented duplication. The “private” testers (even with more recent bits) where asking the same exact questions as the public testers. We made a choice and that choice was to lean on the more public groups because the information was relevant to everyone. Regarding our poor levels of participation in these groups. I think we are guilty as charged in some areas, but the numbers (that I’ll post next week), aren’t any worse in the semi-public whidbey groups than they were in the private ones. There are some newsgroups that get good responses and there are some where we are 0 for 26 in the last 30 days. In those cases… we suck right now. Its that simple.
“Making a beta public and then removing direct beta involvement for ‘official’ beta testers is also a bad idea. While a public beta provides potentially many more testing sites, it also increases the signal to noise ratio dramatically with unqualified feedback or feedback that is so obvious or previously handled that it has no bearing.”
You’d be right… if our only feedback mechanisms where public betas. The fact is that we still have a very large early adopter program with individuals and companies that have even more direct communication with the teams building the product. The 1:1 lines of communication here ensure that their feedback is heard about the “noise” of the public feedback. Our usability teams have also stepped up their efforts to connect directly with a smaller group of customers to attain their feedback. Its just that these programs are just as you expect them to be… more private.
The MVPs used to get early releases that made them part of a more elite club. We have diluted this. There is not currently as much “special attention” paid to MVP feedback compared to public feedback. What needs to happen, in the future, is that more teams need to act like the Foxpro team and treat the MVPs (or best customers) as part of the product team. These customers should have the opportunity to be reading our specs for Orcas soon after they have been written and ever before any code has been produced. These documents and the opportunity to provide this extremely early feedback should not be publicly available, but our first working code drop might be. 🙂
“For a model of what a beta process should look like the VS.NET teams should review how the Visual FoxPro team runs its betas. VFP is handling most bug reports directly through the newsgroups rather than a ‘bug reporting’ system that sucks.”
I’ll disagree here. The foxpro model works because of the limited scope of the product, the customer base, and small internal team size. Yes, the Foxpro team all have their hearts in the right location (to make customers feel connected to the process), but some of the methodologies don’t scale well outside of the small focused product and team. It was becoming a nightmare to track bug requests and responses in the newsgroups across the product. I might read one, forward it to the right team, but then as the problem is shuffled from developer to developer (it may take several teams making a change to fix), the response to the customer would ultimately be lost. There is no additional meta-data attribution on the newsgroups to know when a bug (disguised as a newsgroup post) is reactivated, or what version of the product it may have been fixed in for verification. The process falls apart when scaled out (I’m also making the assumption that we are going to continue to scale our feedback out to catch a wider net of scenarios.)
The product feedback center has some issues that, if addressed, could make it 100x better. If the firefox team can handle their bug influx from 10 million customers in bugzilla, then I imagine we’ll be able to scale to our Visual Studio developers with the right tool. I make no claims that the Feedback Center is perfect today or that it will be tomorrow. Only that we will be dedicated to improving the process of handling customer bug submissions. While I can’t give exact numbers… trust me, the number of customer bugs submitted is not anywhere near the number of bugs that are submitted internally. We will be able to keep up for a long time.
Again, this is not to say that newsgroups and forums do not have a purpose. I just don’t believe that purpose is to be a public (or even large private) bug database. They are simply a mechanism to carry on discussions with customers.
“It’s kind of ironic how Microsoft is advocating distributed, atomic systems, yet VS.NET of all things is the epidemy of a monolithic Windows application that is tightly coupled to every part of the operating system, IIS, SQL Server 2005, heck even the .NET version. If there’s one application that’s most likely to hose your system on install or uninstall it is Visual Studio!!!”
I won’t disagree with your statements here. Sure, if we started fresh we would do things differently, but you always have to weigh that against the cost of simply scrapping a lot of legacy code in the IDE that has worked just fine for a very long time. Currently there are teams working on how to improve our engineering excellence so that we can start to make progress on this front in the years to come. It is not a switch one can simply throw. I’m personally scared to attempt an uninstall and re-install of a daily build without re-imagine my machine. This is something that needs to be addressed. Unfortunately it won’t happen tomorrow.
“I’m all for Microsoft taking their time getting it right and making sure the product is as good and stable as it can be. This is priority One. But I can’t agree with the policy of over-hyping the new technology so early on when it distracts developers so significantly with what at this point is little more than vaporware that’s diverting valuable resources (time!) from developers who should be getting work done … “
I’m not sure which build of the product you are using, but if we simply only labeled the later releases as a “beta” it seems that a lot of your concerns would go away. In the end it really sounds like your concerns are ones that could primarily be addressed with proper messaging rather than serious changes to how we are opening up the process earlier in the release cycle.