A WinINet Chunked-Encoding Story

  • Article

A couple of months back, Nick Bradbury of Homesite and FeedDemon fame posted a blog entitled, "Microsoft, Please fix this WinINet bug!" where he mentioned some users of FeedDemon 2.0 were experiencing a significant CPU spike when downloading RSS feeds. Nick discovered that the spike occurred when downloading feeds from particular sites via WinINet InternetReadFile. The problem turned out to be a combination of issues; first, a handful of servers have non-compliant chunked-encoding transfer implementations and secondly, WinINet was poorly handling these situations.

At a basic level, chunked-encoding allows servers to begin sending responses without first having to determine the overall size of the content. When the server knows the size of the content beforehand, as the case with static content, this is less of a big deal. Things become more difficult and time-consuming when the server is responding with dynamic data, such as from a database or a live multimedia event.

With chunked-encoding, the web server responds with chunked in the Transfer-Encoding header instead of using the Content-Length header. Clients receiving chunked know that each chunk is only a piece of the overall content response. Each chunk is preceded by the size of the chunk (in hexadecimal) before being sent by the server:

Transfer-Encoding: chunked
5DC
    <…1500 bytes of chunked data>
57D
    <…1405 bytes of chunked data>

There is no set limit to the number of chunks that can be returned from the server. Clients simply process the chunks as they arrive depending on the type of content and the scenario. A live multimedia stream may begin playback as soon as enough buffered content has arrived, while a downloaded ISO file may write the bytes directly to a local file until the last chunk is received. The server indicates the end of content by returning a chunk indicator of 0 (zero):

Server: Microsoft-IIS/6.0
Date: Wed, 02 Jul 2006 08:15:00 GMT
Transfer-Encoding: chunked
Content-Type: text/xml
5DC
    <…1500 bytes of chunked data>
57D
    <…1405 bytes of chunked data>
0

This final zero-length chunk is where the problematic servers were breaking from RFC 2616, and where WinINet was failing not so gracefully.

The problematic servers were not ending their chunked responses with the zero-length chunk and instead were simply terminating the TCP connection. The WinINet routine that processed these chunks was overlooking the possibility of an abrupt termination and was steadfastly waiting for the illusive zero-length chunk. Of course, this issue has been fixed in WinINet and is now publicly available with IE 7 Beta 3 builds and newer. We have also engaged with the owners of the known sites, so that they could update their servers to be compliant. I recommend testing your web servers so that you are certain that your site is properly ending the chunked transfer.

As always, I encourage everyone to continue to report potential bugs with Windows Core Networking technologies. The easiest and most direct way to submit bugs is via the Microsoft Networking Connect site at https://connect.microsoft.com/networking. Issues submitted via Microsoft Connect make it directly to our use in minutes. Nick Bradbury, thank you for bringing this issue to our attention.

- Billy Anders