Long-term goals/challenges of community tech previews


I was forwarded a great email thread from VB customers.  Here are some excerpts (from multiple people):


 


“I’m beginning to think “Community Editions” of products as large and complicated as Whidbey are NOT a good idea. Yes I said, NOT.


Reason? I believe the average non-NDA, Joe Blow developer (esp. those


new to .NET or those who’ve not upgraded yet) out there get the wrong


impression. These editions are just not up to the quality-bar that the major betas. And *I* understand that. Most of us do. I just wonder if they don’t hurt the products.”


 


“I’m disappointed in how Microsoft is handling this “release”. Each time they’ve done it (MVP summit, Dev conf in Orlando, Whidbey preview


workshop this week), I’ve counseled them to tell the developers that


anyone installing this software must be prepared to format their drive


and re-image. Invariably, they agree but say nothing.


This can only cause hard feelings as more “Technology Previews” arrive.  Fewer developers will dare to try them–simply because they don’t know how to protect themselves.”


 


“Why not offer these drops in the form of a VPC image with the build


installed?  Not only would this give the community a safe way to work


with the builds, it would also let you get up and running in 2 minutes


instead of the 20-40 it takes to get a preview installed.”


 


“I agree that VPC images would be a _lot_ more compelling.  Unfortunately, there are lots of issues with distributing VPC images.” 


 


It’s great feedback, partly because it shows that we need to explain our long-term intent better.  I’ve touched on some of this before, but I’ll try to elaborate here. 


 


Our ideal is that every build is publicly available.  Another way to say it is that we want to enable a scenario where a customer finds a bug, reports it, the team/developer gets the report essentially immediately, sees it’s an easy fix, fixes it, resolves the bug, the customer gets notified, and with no extra effort from anyone (beyond the customer installing) the customer can install the fixed version within a few days.  I have tried to do that within our existing processes and it requires a significant amount of extra effort to get the fixed build distributed to even one customer.  There are similar (but different) scenarios around getting feedback on features before the features are so baked that it will be a whole product cycle (!) before the feedback can be addressed.  Frequent public builds are a means to the end of enabling us to be much more responsive to some kinds of customer input. 


 


We’re in a very significant transition phase that will be challenging due to people’s expectations (both internally and externally). For a long time we’ve released builds very infrequently and gone to great lengths to make sure “builds work as well as possible” by having long lockdown/test periods before each beta.  (More on “builds work as well as possible” below)  This March Tech Preview is a step towards much more frequent builds, but the transition is causing turbulence because of past expectations.  Once we’ve been putting builds out frequently (think weekly or every day) for a while, there will be less confusion about which ones have been through comprehensive testing – many people will learn to ignore most and look for the Beta indicator of quality, while bleeding-edge customers have the option of being more involved.  I’d love input on how to get through the transition phase better.


 


I put “builds work as well as possible” in quotes above because the downside to the long lockdown periods is that we’d regularly get customer feedback several weeks prior to the release and decide not to fix it simply to avoid the risk of destabilization.  That has been a source of great frustration for many of us for prior releases – we’d get builds out to some customers at the earliest possible moment but we’d already be well into lockdown so feedback we got prior to beta N often didn’t get addressed until a beta N+1 many months later.  This is the tension – minimizing change so we don’t break things that worked for customers yesterday, or addressing the feedback they gave us based on their experience yesterday.  Community drops are about getting feedback in time to address it. 


 


We’ll be releasing another community technology preview in a few weeks.  You will probably find that the quality will be significantly better than the March on.  This is a function of where we are in the release cycle.  The March one happened very shortly after a significant number of changes went into the build.  Since then we’ve fixed a huge number of bugs, done extensive testing and not added/changed functionality.  We will explain this to people.  There will still be significant bugs – otherwise we’d call it a beta. 


 


The feedback above on messaging at conferences, etc., is great – I’m following up on that.  We did put a fairly dramatic warning on the cd envelope – I know that had some effect because someone in my management chain asked a VSLive attendee whether they’ve installed the build, and the person said “Are you kidding?!?  Have you read the warning?”  We’ve also had relatively limited downloads on MSDN (in the thousands, vs. many times that for betas).  Some obviously got the message but again, given expectations, we have to do stellar here to avoid problems and we’re not stellar yet.


 


In the long run we need to do five things well for this to really come together: 


 


First, we need to solve the distribution problem.  We’re looking at several different options – shipping media, VPC images, the p2p solution suggested earlier.  The one I’m most intrigued by (and probably the farthest from reality today) is incremental updates.  Anyway, we need to figure out how to get a large quantity of bits to customers in a quick, reliable, and cheap way, otherwise it’s impossible to do this frequently.   


 


Second, we have to expose what we know about the build’s quality.  Internally we survive by running build verification tests on every build.  At a minimum we can start exposing that level of information for these builds.  Part of the problem is that the information is very hard to digest, e.g. it comes out as 83% of the tests passed, but what you really want to know is that you can’t open a project or debug because there are significant failures in those areas, i.e. the test results can look good when any real user would say it’s broken.   


 


Third, we need to be smart about letting the community help in communicating build quality.  I described this in an earlier post, but next to the “Download this build” button, I would love to show how people who’ve installed so far feel about the build.  If I’m a customer about to download the bits and I see that 21 out of 23 people who’ve installed so far say “It’s great” (or “it sucks” as builds sometimes do), I can make much better decisions about using my time to download and install.  There’s an opportunity here to promote and benefit from community self-organization.


 


Fourth, we need to be more transparent about where we are in the dev cycle, what features are coming in now, etc..  We’ve made some progress here, but we need to do more.  One thing about the March tech preview was that it happened pretty quickly and we missed opportunities to explain where we were in advance, to set expectations better.  In the coming months expect us to do things like cover the project status on Channel 9 and be even more explicit in blogs.  We’re doing a review of the March tech preview now to figure out what to change and what specific steps we can take today to help with the next one.   


 


Lastly, we need to give customers a dependable, effective mechanism for submitting feedback and finding out what we’ve done with it.  Few customers have this today, and historically we’ve done a poor job responding (although we’ve done much better with feedback on the Whidbey alpha from last summer).  If customers can submit feedback, get a response within a few days, and see the results a few days later, my guess is they’ll be more interested in tracking our progress more closely. 


 


I think if we do those things – make the logistics easy, make it easier to assess quality, make it easy to understand whether the builds have features/bug fixes that are interesting enough to me – then much of today’s pain will go away (or be offset by benefits).  The problem right now is that none of these things are automatic.  The many many month release cycle has gotten us in the habit of doing little of it as we go, and then trying to pull a bunch of stuff together at the last minute – it’s pretty classic for us to send out a beta with 0 guidance on what new features there are or what has been fixed.  The current community drop approach is still too much like infrequent betas – but we are starting to figure out how to build it in and make it part of what we do every day. 


 


In the end, if we do all this and 99% of installs still happen with betas (but 1% continue to make the effort to get more frequent drops), we’ll be satisfied because two things will have changed:


        We will have shifted much of the control/decision-making over how involved customers can be from us to our customers. 


        It will enable every person within our division to be much more responsive to customer feedback, without a lot of incremental extra effort.


 


That’s the big picture.  I’m very excited about frequent community/public builds not because people are clamoring for them, but because of the implications of making them work well for us and for our customers.  There is some skepticism internally about these – no one will install, etc..  And there are stages in our product cycle where builds generally are bad.  But even scoping this down to just the time between a beta1 and RTM would be a huge improvement on what we do today.  The great thing is that that period is coming up soon and it’s a great chance to figure it out, so that when we get ready for the next product cycle we have enough of the infrastructure/processes/philosophy in place that we can continue. 


 


Thoughts? 


 


 


 


 


 


 


Comments (17)

  1. Sean McLellan says:

    Great article! hit the head on a number of questions and thought processes I had on this issue.

    Some thoughts I have which may not be valid since I don’t know the whole picture, but here goes…

    You noted that more customer feedback both for the dev team and for other customers wishing to try out a new build is definately needed. I think the flip side of that is if open comments are allowed (rather than just a rating system) there also should be a way to filter down the responses so that only "good" information is relayed to my peers (as a customer) and devs. I don’t know how you’d do this well (slashdot-post model, amazon-comment model?) but being able to limit other comments to those from particularly MVP’s, MCP’s or just other good users would delightful.

    Being a participant in a number of betas on betaplace I’ve seen this numerous times — in using a beta someone encounters a nasty bug in some product, which isn’t listed in the beta readme, and they head over to the beta newsgroups and post a lengthy detailed message on what happened, why, where, how to reproduce, screenshots, diagrams, slideshows and free beer tokens for all those who got to the end of the presentation. And then the response from the devs is something akin to "Oh yeah, that, knew about that. That’s fixed in the next beta." ..

    Could there be public bugtracking pages to go along with public betas? Updated project-by-project status page complete with fancy barcharts indicating completion? 😉

    Some of the things you listed would be great if they were implemented in right in the MSDN subscriber downloads area. Seeing a 1-5 build rating, internal dev. quality assessment and comments would be nice to see just before hitting that download button. It would be great if MSDN had other pertinant "INFO" associated with each download, but that’s nother topic..

    Anyhow, just some of my thoughts..

    oh, and "We’ll be releasing another community technology preview in a few weeks" + TechEd is in a few weeks + I’m going to TechEd = Whoohoo! 😉

  2. Sean says:

    Had another thought before the fact that another CP is around the corner totally overwhelmed the thought..

    Could VPC be advertised a little more and/or made to be accessable for devs along with CP releases? (Licensing issues abound, but I’m just thinking here)

    p.s. Virtual Server rocks..

  3. Mark Cliggett says:

    I hate it when customers go to a huge amount of effort to report a bug and then we say "oh yeah, we knew about that one". While we have significant improvements planned in the near term for being much more transparent about feedback we get from customers (which addresses to some extent your interest in filtering), I don’t have a great answer today about internal bug status – there’s always been a concern about revealing customer confidential information. (Plus some other concerns which could be addressed through, um, professionalism…)

    One thing I didn’t say in the post above is that this is a many month (if not years) process. If we waited until we got the process/transparency 100% right before we started, it would never happen – there’s simply too much internal inertia. Thanks to some very strong management support, we’ve gone from releasing builds very infrequently to somewhat more often. We have lots of gaps in what we need to make it really work, but by taking the first step we’re now getting very concrete feedback from customers on what is missing.

    The suggestion about MSDN download ratings is a good one – I was thinking just about Whidbey but you’re right, this would generally useful. I’ll follow up with them.

  4. Bill Vaughn says:

    I think the VPC approach would be best. I would also suggest a whitepaper (which I can write for you) that outlines how to test beta software and not have to buy a new hard disk each time (which actually works pretty well) 😉

    Of course, the VPC approach assumes a great deal. I have moved VPC images around on my systems but they’re all pretty healthy. We would need a minimum testbed system to host the VPC or the performance would be "challenged".

  5. Joku says:

    I am using vmware, is these VPC images even compatible with that? vmware allows to just load the .iso file into virtual drive, or to point to a host drive like a virtual dvd drive such as daemon-tools’s which can mount all types of .iso’s.

    About the incremental updates, I thought of one way to do it:

    1) distribute the "master" image, which would install the base for incremental updates.

    2) create a program that runs as a peer to a p2p incremental update distribution network, which still verifies the file integrity through some good hash from MS server.

    3) this p2p program would also act as a server for LAN based adresses, so the virtual pc could connect to this and use it as a proxy to getting the incremental updates

    4) a client for the p2p update proxy which would run inside the virtual pc and do the actual updating

    There you have it, I think this could be feasible, secure and not hog too much MS bandwidth.

  6. Joku says:

    .. OTOH it could be that people would just shut down the p2p part of the thing after being done with their daily update, making the updates mostly being fed from MS servers.

  7. Chris Garty says:

    First, I have to say that it is great to see someone at Microsoft so dedicated to promoting the benefits of increased customer feedback and collaboration. Keep up the great work!

    "Once we’ve been putting builds out frequently (think weekly or every day) for a while, there will be less confusion about which ones have been through comprehensive testing – many people will learn to ignore most and look for the Beta indicator of quality, while bleeding-edge customers have the option of being more involved."

    "Community drops are about getting feedback in time to address it."

    Shorter iterations that allow you to incorporate customer feedback quicker and respond to change faster are an excellent goal. I think you are right… most of the friction and grief caused by this community release is probably because the customers expectation is currently out of sync with what you are trying to achieve. This is a risk in any project and a challenge to put right. I can only imagine how big a challenge this is for a project/product like Visual Studio with its massive customer base. Keep the warnings big and blatant. Just ensure that you scare off the majority of developers ;).

    "Part of the problem is that the information is very hard to digest, e.g. it comes out as 83% of the tests passed, but what you really want to know is that you can’t open a project or debug because there are significant failures in those areas, i.e. the test results can look good when any real user would say it’s broken."

    Have you got any automated acceptance tests? If you define a limited set of regression tests to run each time, you’ll be able to see if anything has been broken. Perhaps you could create a group of ‘these must always pass’ tests that can tell you whether an incremental (community preview) release can take place.

    "I would love to show how people who’ve installed so far feel about the build"

    A public rating system like Amazon… I love it ;). Use the power of the community! You’d need to require registration, or else you’d have all the slash.dot community pounding you for a laugh.

    "what specific steps we can take today to help with the next one"

    I think a rating system with comments would be one of the easiest, simplest and most effective things you could try. Giving some regular dates (e.g. community preview drops at least once every two weeks) and sticking to them (even if it means not getting a specific feature in) would encourage a greater trust and a larger amount of feedback. Providing a bug tracking system, like Sean suggested, would be an excellent way to ensure that only a single (ideally) amount of hair is pulled on a single bug :).

    Shift of involvement control to customers -> excellent news for those leading edge shops and excellent news if you are looking to increase the volume of feedback you receive with little effort.

    Responsive to customer feedback -> If you can succeed in making Microsoft more responsive than the smaller players through better process and better communication with customers, then you’ll gain another area of competitive advantage.

    "Some skepticism internally about these"

    Encourage them to give it a go, and then show them the download counter and the increased stream of feedback. Get the skeptics to estimate the cost of changing and then plot this against the tangible and intangible benefits.

    – Chris

  8. One thought I’ve had would be to provide drive imaging software. (Ghost or Drive Image, etc) Once a good master image is created for the install machine, that means it’s easy to rollback to a good known state, even if the CP or beta, or whatever, doesn’t have the best uninstall. This might foster more people being willing to try interim builds on secondary or primary machines.

  9. Amr Essam says:

    I develop with both C# and VB.NET, but I like VB.NET to be better because I am old VB developer.

    I just installed VS.NET 2005 (Technology Preview), and this my prompt feedback

    ——————————————————————————

    I surprised that I found productive feature in C# and not

    in VB.NET, this features:

    1) Refactoring

    I tried Refactoring in C#, It is really very productive

    feature.

    2) IntelliSense/Auto Completion (Keywords)

    Now only in C# debugger feel the Language keywords like

    (private, public, foreach …), and auto complete it to

    you.

    However VB.NET have longer Keywords such as (MustInherit,

    NotInheritable,.)

    —————————————————————————-

    I dreamt that I can find these features in VS.NET, but

    unfortunately I frustrated when I found this feature in C#

    not VB.NET ?

    Microsoft always say that it concentrate on Productivity

    features in VB.NET, How does VB.NET 2005 lack these

    very productive features ?!

    Looking forward for your reply.

    Amr Essam

    http://www.Verizon.com

    Consultant & Team Lead

    MCSDT + MCT

    Dallas, Texas

    amr_essam@yahoo.com

  10. >> Anyway, we need to figure out how to get a large quantity of bits to customers in a quick, reliable, and cheap way, otherwise it’s impossible to do this frequently.

    What about using BitTorrent?