‘Avalanche’ of information/bits


The frequent Community Technology Previews (CTPs), Betas and other kinds of information across the different products often end up causing a virtual ‘avalanche’ of information flow for our customers.  These information ‘avalanches’ pose problems related to congested servers, network traffic and download speeds.


 


To address these issues, Microsoft Research (MSR) is working on innovative technologies with the codename “Avalanche”.  Avalanche provides a PC-based, fast, internet-scale software distribution solution.  The Avalanche model addresses internet software-distribution problems using network coding which ensures that any piece uploaded by a given peer can be of use to any other peer.  This research paper and these slides provide more information on network coding and Avalanche.


 


The Developer Division has a pilot project under way using Avalanche to download CTP bits for our Most Valuable Professionals.


 


Namaste!


Comments (11)

  1. Marc Brooks says:

    Ahem… why are you reinventing BitTorrent? It works VERY well, and it is VERY mature. Please don’t waste our shareholder capital building something just because of NIH syndrome.

  2. Matthew says:

    You misspelled "Avalanche" in the title: ‘Avalance’ of information/bits

  3. Alan Stevens says:

    Soma,

    I would like to second Marc’s comment. BitTorrent is very effective at solving the problems you cite. Microsoft is very capable of innovating in this space, I am sure, but I don’t see any evidence of it here.

    Regards,

    ++Alan

  4. I recommend reading the research paper, or at least flipping through the powerpoint – Avalanche solves several problems that BitTorrent suffers from.

  5. File Swarming is a great way to distribute large files, and BitTorrent is a cool file swarming implementation. We’re evaluating a different approach to file swarming problem that will have benefits in certain scenarios.

    As Jonathan suggests, the slides & papers Soma point to on Pablo’s research site provide motivation for this research, as well as some of the research results. If you’re interested, take a peek at them. If you have any questions or comments you can drop me a line at johnmil@microsoft.com.

  6. loc says:

    It’s pretty clever the more I think about NC. I’ve thought long and hard to figure if/where Avalanche can be improved. The down side is that this is not ideal for media streaming (for example, on-demand and interactive TV via broadband), because all encoded blocks must be collected before reconstructing the file from the DNA, but I think this can be solved.

  7. TAG says:

    Wow.

    You have spend more then 1.5 years on this idea and still in pilot stage.

    It was discussed long time ago by Mark (Ex-DevDiv Customer Connection Lead)

    http://blogs.msdn.com/markcli/archive/2004/03/29/101448.aspx

    I’m very disappointed the way Microsoft create/use "internal" tools for running betas and collecting feedback.

    ProductFeedback / BetaPlace / WindowsBeta – all of them sucks.

    I hope Avalanche will be more secure than old File Transfer Manager ( http://news.zdnet.com/2100-1009_22-954590.html )



    Andriy Tereshchenko / TAG

  8. Keen Browne says:

    Hi TAG,

    I’m running the Avalanche pilot and Mark was my Manager.

    You make a few interesting points–

    Why is it that it took us so long to use this technology in a pilot, much less than in a complete product?

    When I joined the customer connection team this past summer, Mark asked me to take a look at Avalanche. I wanted to get it into customer’s hands as soon as possible–a worldwide customer release of an MSR project was deemed (understandably) too risky. Instead, I decided to run a pilot. Now I’m working on the next phase of the project.

    You might ask, why is it that with Microsoft having so much money, that someone wasn’t hired until this summer to work on this problem?

    The answer has two parts. At first, Microsoft did not perceive file distribution to be the most important priority. Specifically my team had to pick between the product feedback center, visual studio registration, forums, and a new mechanism for distributing CTP’s. The rest of our division was busy building the product. We picked the feedback center, registration, and forums as the highest priorities. When my team got me, we picked up ctp distribution as the next priority on the list. We have a long list and have room for more employees if you know anyone who’s interested in interviewing. Secondly, it took time to fill the open positions on Marks team.

    Will Avalanche be secure?

    As you [TAG] know the threat surface area for a file distribution system is relatively large compared to its functionality. The surface area is large because of it’s public network connectivity, the amount of information about hosts moving back and forth, requisit disk, network, and cpu access on the end user computer, and vulnerability of the server that hosts the file. Not to mention the standard suseptability to the various flavors of buffer overruns, integer overflow attacks, character encoding, etc.–it’s much like other network servers of which many (IIS comes to mind) also have a large surface area.

    We’re working on a variety of mechanisms to make Avalanche more secure than other file distribution technologies. Specifically, we’re working on ways to verify what content can be distributed as well as enable the end user control which hosts participate in the swarm. We will also look at mechanisms for decreasing the time a node will be present, and, therefore, potentially vulnerable.The code will also be constructed using our trustworthy computing processes, we require complete threat models, static analysis, and review.

    In my opinion, it will be more secure.

    I’m sorry that you are disappointed in the functionality of our beta testing technology. We’re working to fix that. Have you entered your specific complaints into the feedback center or talked with my teammate, Marie? She’s been managing the Product Feedback Center from our side.

  9. Chris Bick says:

    My company Wurld Media which develops Peer Impact(http://www.peerimpact.com) uses a transport protocol similar to "Avalanche" that I’ve been working on for the past 5 years. As you know, BitTorrent is inefficient in how it works, and our core protocol was built before BitTorrent and addressed those issues from the beginning. We have built an application that uses our protocol which the U.S. Army has been using for the last three years to distribute their 900 MB video game called America’s Army: Special Forces. Over the last three years we have saved the Army an unbelievable amount of bandwidth and resources. We have distributed the game to users where other sites could not because their servers were congestion with network traffic. Our protocol is in its third generation and solves many of the problems faced with content distribution over the Internet.

    I’m not sure who to address this too, but I have read your papers about “Avalanche” and would very interest in talking with the "Avalanche Team" on how we might be able to work together. I think we could learn a lot from each other and talk about the pending/issued patents we both have filed.

    More information on our technology can be found at: http://www.lxsystems.com and on my blog at http://cbick.blogspot.com

    Chris Bick (cbick@wurldmedia.com)

    Directory of Research & Development

  10. Frustrated End-User says:

    Is there anyone in Microsoft saying hey lets look at this from the end-user’s perspective? The college student with no money, the college grad with no job, the typical working man shelling out a hundred dollars a week just to drive to and from work, people living overseas. Does anyone think about lowering the cost and price while improving the product? I’ve participated in the beta programs and aside from the finished product not resolving my complaints I hate the download process. Hey guys get with the program I can use a torrent and get the same thing in an hour as it takes a day or more to get from your server and it doesn’t cost you a thing. Modify your business model. Apple and linux aren’t your competitors the so called pirates are. Compete or die thats how it works in this business as you well know.