Content-Length in the Real World

Earlier in IE9, we tried to change the WinINET networking component to reject as incomplete any HTTP responses for which the Content-Length header specified more bytes than the server actually sent back. It turns out that some sites and applications expect to be able to specify an incorrect Content-Length without the client raising a fuss.

This was a pretty surprising finding, and we spent a lot of time looking at it. It turns out that all common browsers are accommodating to this violation of the HTTP protocol.

I built a script for Meddler which tests the browser’s behavior in a variety of Incorrect Content-Length circumstances; the results of running this script are shown below:

Scenario: IFrame Navigation

In this test, the browser navigates a frame to a page which sends an incorrect content length. The scenario where the server sends fewer bytes than promised is the “Underrun” case, and the server closes the connection either gracefully (“FIN”) or abruptly (“RST”). The scenario where the server sends more bytes than promised is the “Overrun” case.

As you can see, all browsers simply ignore the additional content in the “Overrun” case and render the page as if nothing is amiss. In the case of underruns, however, browsers differ in behavior. In the RST cases, browser behaviors differ (and Safari 5’s behavior is utterly bizarre). However, all browsers except for IE9 Beta accommodate the case where the server returns less content than promised, so long as the connection is gracefully closed with FIN.

We ended up reverting IE9 RC to IE8’s behavior here for compatibility reasons. We found at least one major ecommerce site and one major streaming product which always triggered an underrun with a graceful close, and we elected to keep compatibility by accepting the invalid content.

Scenario: File Download

In the File Download scenario, we were particularly expecting browsers to be strict about Content-Length, as an underrun seems like a good clue that the transfer is incomplete.

Here too, we found that all browsers agree on the behavior of an overrun—the downloaded file is simply truncated at the Content-Length specified value. In the case of an Underrun ended by RST, only Chrome treats the file as correctly completed. Safari simply shows a blank download manager window, and all other browsers display an error.

In the gracefully-closed Underrun case, we found that only Opera and IE9 Beta treated the file as incomplete. During testing of the beta, we encountered some internal line-of-business applications that send an invalid Content-Length header, and again, concerned about the compatibility impact of the change, we reverted to IE8 behavior for the IE9 Release Candidate.

Ultimately, this was a surprising and disappointing exercise; Content-Length is one of the most fundamental aspects of HTTP, and it’s one of the most important things to get right in order to ensure reliable operation of the protocol across myriad server, proxy, and client products. Alas, the “real world web” is polluted by buggy behavior and accommodating implementations, and hence it will likely be some time before products can increase their strictness without significant compatibility risk.

Fiddler will show a HTTP Protocol Violation warning if it encounters content with an invalid Content-Length (or missing the proper message length indications altogether). If you encounter such a warning on a site or service that you maintain, please help clean up the web by fixing the issue.

Thanks!

-Eric Lawrence

Update: IE10 updated the behavior here somewhat; see my post Content-Length Validation in IE10.

Update: Firefox tried to get stricter about length validation, but ended up backing it out. https://daniel.haxx.se/blog/2014/10/26/stricter-http-1-1-framing-good-bye/