Enclosure Download


A while ago I posted details about the RSS Platform Download Engine. That post focused on downloading of feeds, but did not include additional details on enclosure downloads.


 

Enclosures are, as most readers know, files that are “attached” to items in an RSS feed. Typically, a publisher will include a reference to a binary file, which an RSS aggregator can optionally download when the feed content is downloaded. The most common example of enclosure use in RSS feeds is for podcasting, where the attached (or “enclosed”) files are audio files.


 

As with feed download, we designed the enclosure download with server and client bandwidth in mind since feed as well as enclosure downloads also happen in the background. Their impact on foreground applications should be limited. Similarly, the impact of large enclosure downloads on servers should be limited.


 

Let me sketch how the enclosure download process works:

  1. Every time the feed download engine runs it processes feeds that have the “Automatically Download Enclosures” setting set to true it. If it comes across a new item with an enclosure it adds the URL of the enclosure to a FIFO queue.
  2. Before the enclosure is added to the queue, the URL is checked with Attachment Execution Service API (AES) to assure the enclosure file type is one of the permitted types. If it’s not, the enclosure download is failed (IFeedEnclosure.LastDownloadError = FDE_DOWNLOAD_BLOCKED).
  3. The first 4 enclosures in the queue are then handed off to the Background Intelligent Transfer Service (BITS). BITS is a background download service that ships in Windows and which enables downloading of files in the background while limiting its affects on network usage. In particular, BITS uses HTTP RANGE requests to download files in chunks. BITS also monitors whether foreground applications (like email or browser) are using the network, and if so, it throttles back its own network usage to limit its impact on those applications.
  4. Once BITS completes downloading an enclosure, the Download Engine uses AES to save the enclosure to the folder corresponding to the feed. Saving via AES associates zone information with the file. The zone information is used when the file is launched at a later time.
  5. If there are more enclosures waiting to be downloaded and there are less than 4 enclosure downloads active, the next enclosure is handed off to BITS as in step #3.
  6. If, however, the server of the enclosure does not support HTTP RANGE requests, the Platform Download Engine falls back to downloading the enclosure via a regular HTTP GET request. If this attempt fails as well, then the enclosure download is fails and will not be attempted again automatically.


     

Note that the enclosure fall-back download (HTTP GET) is size limited to 15MB to limit the impact of denial of service (DoS) attacks against the RSS Platform Download Engine. Since the RSS Platform Download Engine runs in the background, a malicious server could consume all of the client’s download bandwidth without the user having any idea. Enclosure download via BITS (HTTP RANGE requests) is less impacted by such an attack and is consequently not size limited.


 

In other words, if you are an enclosure publisher that wants to serve enclosures larger then 15MB to IE7 users, then you should use HTTP servers that support HTTP RANGE requests. Most popular web servers support HTTP RANGE requests.


 

It’s also worth noting, that when a server does not support HTTP RANGE requests, the RSS Platform Download Engine will issue two requests for each file (the first testing for HTTP RANGE support, and the second to download the file without range support).

For more details on the security measures used to protect applications and users from potentially malicious enclosures, see Miladin’s enclosure security post.


 

I hope that this description of the enclosure download process explains the “multiple-requests” that some publishers have seen, as well the security restrictions associated with enclosure downloads.


 

-Walter vonKoch

Program Manager

Comments (42)

  1. rickw says:

    I’m using IE7 and the feed list to manage the podcast I listen to and the auto-downloads work well.  Except for trying to get WMP to autodiscover the new media.  Right now I’m using a manual process.

    1) Setup the feed to auto-download

    2) Click the "view files" button and copy the folder to the clipboard

    3) Go to WMP and add a new folder to monitor.  Paste the path from the clipboard.

    4) Setup a Auto-Playlist to include the new podcast

    5) Setup the sync to my portable player (a Sprint PPC-6700)

    Any suggestions for simplifying this process?  

    Are there events that an app could respond to notice that new feed is downloading attachments?

  2. Is there a way to define the directory for downloads? It would be so much better for findnig the files later.

  3. Hi guys,

    Is there perhaps a registry hack to extend the 15MB limit? We have work-related podcasts we’re trying to download that are just outside that range, and so we have to manually download them.

  4. ptvGuy says:

    That will place a serious limitation on the Internet Explorer browser for feeds that contain video since it doesn’t take much video to pass 15MB.  This will adversly impact not only video bloggers, but all public media content producers.

    Just as more and more public broadcasting stations are adopting RSS feeds for sharing their local content, you’re capping the file size in way that prevents most video and nearly any full-length audio program.  There has to be a better alternative than HTTP-Range checking.

    The point of RSS is to make this kind of content syndication simple.  RSS is an XML format and should be totally independant of the server.

  5. A number of companies including mine absolutley hate that you all decided to continue using BITS it completly destroys stats reporting of downloads and puts ridiculous loads on servers that are serving the data.

    Here is why instead of having to deal with one file request and serving it at once thus one call to our databases servers that are managing file distribution for load purposes you all send 100’s of request for one single file.

    One service provider is so pissed at the way Bits is abusive to those of use serving the data that they have BLOCKED BITS releated transfers.

    All of consider the protocol to not be friendly to us serving up data and you continue to not win friends in your utilization of bits. You should have talked to the people really serving data!

  6. Ari Pernick says:

    It’s intresting what you did with the security model, but I think you should allow "infinte" sized simple HTTP GET downloads for enclosures that come from the same server who supplied the feed.

  7. Rob Walch says:

    So if I am reading all of this correctly.  IE7 is really not an aggragator in that it does not actually download all of the shows and then place them with your other Rich Media.  

    The real reason podcasting took off – was not that you could add an enclosure and have some one subscribe and download to it.  Rather it went a crucial step further (or at least Adam Curry did) it was setup such that the files after being downloaded were placed with all of your other rich media files so the subscriber could easily find and consume the media.  

    With out that crucial step – it is not podcasting.  It is just downloading via RSS.  At least if the file size is less than 15 Meg.  

    Which by the way most podcasts are well over 15 Meg in size.

    So when is MS coming out with a truly intergrated Podcasting solution for the subscriber?

    Rob @ podCast411

  8. Wilbur says:

    How about a new year’s resolution for the RSS team:

    Allow users to choose the maximum size of enclosure to download and let them decide whether or not to trust the server.

    If you are feeling really virtuous, how about another one: Allow users to set the download location for enclosures.

    I know that Microsoft is always being lambasted for not catering to the lowest common denonminator but these two things are pretty vital to achieving the functionality needed to make IE7 achieve its objectives.

  9. Wilbur says:

    How clever to aviod DoS attacks the size of the enclosure is limited to 15MB. Nobody would ever have a podcast bigger than that surely?

    Check out: http://msdn.microsoft.com/msdnmag/podcast/default.aspx

    8 podcasts, 4 of which would not be automatically downloaded by IE7. What internet vandal would host such large files?

  10. rss says:

    @Wilbur,

    You might have misunderstood the 15MB limit. It’s a fallback limit which *only* applies if the webserver does *not* support HTTP RANGE requests. The enclosure server download.microsoft.com does support HTTP RANGE requests. So if you try it and turn on automatic enclosure download for MSDN podcast feed you will see all enclosures downloaded.

    I hope this clears up the confusion.

    – Walter [MSFT}

  11. Tom Gleeson says:

    Does the feed manager recognise links with the micro format rel=enclosure as enclosures or does it look for the <enclosure> tag? What about Atom enclosures?

  12. James says:

    Please make it possible to set the download location of enclosures!

  13. Dann Veldkamp says:

    One more request for the ability to specify a download folder for each feed. Without this I am left with iTunes or manually downloading my podcasts so I can burn them to CD for listening in the car.

    Dann

  14. ali says:

    Walter  –

    You’re not being misunderstood. You need to make it so that this does not bother users by not allowing them to download podcasts. Here’s a better way of preventing bandwidth wastage: Monitor the feeds bandwidth usage pattern and inform the user of any abnormalities that appear. This will actually turn a stupid restriction into a useful feature.

  15. DavidWT says:

    I agree with Dann, being able to specify (different) download folders for different auto downloaded  feeds would be a great advantage. Also feeds that are different auto downloaded go to Temporary Internet Files and so could easily be deleted forgetting that there are feeds downloaded.

  16. Lanad says:

    لوتسمح هل من الممكن توضع رابط تنزيل لا الشرح وحدة لا يكفي ولك مني جزيل الشكر .

  17. LinkDir says:

    It’s intresting what you did with the security model, but I think you should allow "infinte" sized simple HTTP GET downloads for enclosures that come from the same server who supplied the feed.

  18. Thanks Walter, Good to know. 15mb limit would be useless for our guys. I see what you’ve done now, it’s starting to make sense. Thanks again.

  19. Tweak Vista says:

    A number of companies including mine absolutley hate that you all decided to continue using BITS it completly destroys stats reporting of downloads and puts ridiculous loads on servers that are serving the data.

    Here is why instead of having to deal with one file request and serving it at once thus one call to our databases servers that are managing file distribution for load purposes you all send 100’s of request for one single file.

    One service provider is so pissed at the way Bits is abusive to those of use serving the data that they have BLOCKED BITS releated transfers.

    All of consider the protocol to not be friendly to us serving up data and you continue to not win friends in your utilization of bits. You should have talked to the people really serving data!

  20. Veridom says:

    How about a new year’s resolution for the RSS team:

    Allow users to choose the maximum size of enclosure to download and let them decide whether or not to trust the server.

    If you are feeling really virtuous, how about another one: Allow users to set the download location for enclosures.

    I know that Microsoft is always being lambasted for not catering to the lowest common denonminator but these two things are pretty vital to achieving the functionality needed to make IE7 achieve its objectives.

  21. TheCableGuy says:

    That will place a serious limitation on the Internet Explorer browser for feeds that contain video since it doesn’t take much video to pass 15MB.  This will adversly impact not only video bloggers, but all public media content producers.

    Just as more and more public broadcasting stations are adopting RSS feeds for sharing their local content, you’re capping the file size in way that prevents most video and nearly any full-length audio program.  There has to be a better alternative than HTTP-Range checking.

    The point of RSS is to make this kind of content syndication simple.  RSS is an XML format and should be totally independant of the server.

  22. LFERC says:

    I’m using IE7 and the feed list to manage the podcast I listen to and the auto-downloads work well.  Except for trying to get WMP to autodiscover the new media.  Right now I’m using a manual process.

    1) Setup the feed to auto-download

    2) Click the "view files" button and copy the folder to the clipboard

    3) Go to WMP and add a new folder to monitor.  Paste the path from the clipboard.

    4) Setup a Auto-Playlist to include the new podcast

    5) Setup the sync to my portable player (a Sprint PPC-6700)

    Any suggestions for simplifying this process?  

    Are there events that an app could respond to notice that new feed is downloading attachments?

  23. How do I change the default directory for downloads?

  24. enciclopedia says:

    You cant! You must wait for the soft update.

  25. Thanks for very interesting article. btw. I really enjoyed reading all of your posts. It’s interesting to read.

  26. it is really good article, I found the reason for my problem!!

    Thanks

  27. Thanks for the article. I really enjoyed reading all of your posts.

  28. Thanks for very interesting article.

  29. Padova says:

    It’s easy to overcome the 15 MB limit – just set the registry value HKCUSoftwareMicrosoftFeedsMaxEnclosureSize (DWORD) to a suitable value (in kB). There is a similar value for the maximum XML size (also 15 MB by default). If you are writing a podcast app that uses the common feed list, this is easy enough to adjust. More details are on MSDN.

  30. Padova says:

    @LFERC – Here’s what I do:

    1. From the command line, type

    mklink /j %userprofile%MusicPodcasts %localappdata%MicrosoftWindowsTemporary Internet FilesEnclosure"

    This will make the podcasts visible to WMP.

    2. In WMP, create an auto playlist with a filename that contains "MusicPodcasts".

    Now all you have to do is sync that playlist to your audio player. It works great!

  31. rzeszow says:

    now I get the multiple-requests subject. thanks! 😉

  32. Great article. It`s realy worth reading. I wish you further successes

  33. omaha rules says:

    It’s intresting what you did with the security model, but I think you should allow "infinte" sized simple HTTP GET downloads for enclosures that come from the same server who supplied the feed.

  34. Sukienki says:

    Very interesting article.

    Good work!

  35. Gili says:

    <a href= http://index1.cagew.com >long island teen murdered in neighbors basement and 1987</a> <a href= http://index2.cagew.com >richerd glassman lcsw</a> <a href= http://index3.cagew.com >psjaclass of 1982</a> <a href= http://index4.cagew.com >jewerly box for little girl</a> <a href= http://index5.cagew.com >western mass bass trail tournaments</a>

  36. 15 MB limit – just set the registry value HKCUSoftwareMicrosoftFeedsMaxEnclosureSize (DWORD) to a suitable value (in kB). There is a similar value for the maximum XML size (also 15 MB by default)

    anyway thanks

  37. Lsmwutxc says:

    <a href= SADDDDDDDDDDDDDDDDDDDD >DDDDDDDDDDDDDDDDDDD</a>, %-DDD,

  38. phax says:

    Let’s make one thing clear… people want to do Podcasts, and Windows users HATE iTunes – it’s OK on Macs but sucks on Windows. MS have a 90% solution to make it OK and native in Windows. It just boggles my mind that you actually build obstacles for it to become the solution everyone is waiting for.

    1. get rid of the file size limitation. Too many interesting podcasts can’t be downloaded today.

    2. let users decide where to put the media data

    I also use the My MusicPodcast link to the downloads directory and the Auto Playlist in Mediaplayer, but I found another issue… The auto playlist does not seem to refresh when I expect it. Both on MP11 XP and on VISTA. It does, eventually, but after it failed syncing my mp3 player a couple of times, and I hack around it for a few minutes.

    How does one get auto playlists to refresh reliably and always before a sync? If it should it doesn’t on any of my boxes.

  39. hoobuba says:

    @Padova – you are genious! it works! but WMP should be able to do this out of the box

  40. Sukienki says:

    Nice and very useful article. Thanks!

  41. dj says:

    leave me along pooopoooppooooidooooooooooppppooodooooppooo head

  42. Rob Smeed says:

    It would be very helpful for finding files later if there was a method of defining the directory for downloads.