Guest TCP psychic debugging: Why the remote server keeps RSTing the connection


My colleague Keith Moore (who occasionally comments on this site) shared with me one of his recent triumphs of psychic debugging. First the question:

The customer is getting an RST response from IIS and they would like to know why. Here is a fragment from a network capture that illustrates the problem. (Fragment deleted.) The full capture is available on ...

Keith didn't look at the full capture; he barely even glanced at the fragment. Because his psychic powers told him the answer:

The amount of data they are sending does not match the Content-Length header.

He goes on to explain:

If there's unread data pending when a connection is closed, it is automatically reset.

Sure enough, the customer was uploading data to the IIS server, specifying a Content-Length of 1289 but actually sending 1291 bytes. The server stopped reading after 1289 bytes (respecting the Content-Length), and the client was upset because Hey, you forgot two bytes!

As is occasionally the case with these types of problems, the misunderstanding goes deeper than the question itself. The customer replied, "I don't understand why the server sees 1291 bytes. If you look at the network capture, the client machine sends two frames, one of size 1289 and one of size 2. Both frames have the correct size specified in the frame header. The size of the frames is correct; I don't see what the problem is. I mean, sure, if the first frame had a header specifying 1289 bytes and the payload contained 1291 bytes, then that would be a problem, but that is not the case here."

The problem is not with the frames; it's at a higher level. The client machine promised via the HTTP protocol to send 1289 bytes, and it sent 1289 bytes, and then sent two more bytes. The reset occurs because the client machine lied about how many bytes it was going to send. The frames themselves are fine; the problem is that they are the wrong frames.

Comments (14)
  1. Karellen says:

    And the extra two bytes were …?

    My psychic debugging skills tell me it’s likely to be a trailing rn

  2. Sean says:

    @Karellen: I thought the same thing, "What about rn?"

  3. ton says:

    Sounds like his software is creating a post request setting the content-length correctly for that and then attaching a new frame for the rn characters when it should have been in the original frame.

  4. @ton: The code in question is on a level of abstraction that does not know about TCP segments at all. If the two bytes were intended to be part of the posted data, the client should have included them in the HTTP-level content-length.

    Possibly the RST packet only gets sent if the client sends the extra bytes after the client-side TCP/IP stack has already shipped off the actual body, AND the server is quick enough to read the 1289 bytes, process them and shut down the reading side of its socket before the straggler segment arrives. This does not mean that the error is at the TCP layer; it’s just an accident at the TCP layer that controls whether or not a bug higher in the hierarchy gets triggered.

    In any case, the lying client is in good company. There’s a popular web browser which used to (and may still, for all I know) append rn to its POSTs without counting them in the content-length. Some HTTP server code will explicitly try to flush the reading side of the socket before it closes it, in the spirit of be-liberal-in-what-you-accept.

  5. Alexandre Grigoriev says:

    From RFC2616:

    "Certain buggy HTTP/1.0 client implementations generate extra CRLF’s after a POST request. To restate what is explicitly forbidden by the BNF, an HTTP/1.1 client MUST NOT preface or follow a request with an extra CRLF."

  6. (And here’s to me managing to completely overlook the explanation in the third quoted statement of the post, leading to a long bogus speculation about the precise triggering of the bug. Cheers!)

  7. Jeff Walden says:

    Actually, it seems a number of HTTP clients were buggy sending a CRLF after POST requests, some including it in the Content-Length, some not.  See the comments in the following Mozilla bug:

    https://bugzilla.mozilla.org/show_bug.cgi?id=87817

    The practice apparently was a response to a CERN webserver bug, which then likely propagated to clients.  This bug then resulted in a note in the HTTP/1.1 RFC (at least 2616, didn’t check earlier ones), and, perhaps less obviously, in RFC 2616 section 4.1’s note that "In the interest of robustness, servers SHOULD ignore any empty line(s) received where a Request-Line is expected.", which, notably, is important when you consider how a trailing CRLF interacts with keep-alive connections.

  8. John says:

    @Henning

    The code in question is on a level of abstraction that does not know about TCP segments at all.

    You wish. There was a certain webserver that expected request headers to be wholly in the first TCP segment received, otherwise it’d trigger an invalid request error. As a consequence it would always fail if the headers were longer than the network MSS, or if you tried to manually type in a request using telnet (instant failure the moment you pressed the first ‘H’).

    Fortunately it got fixed, but some browsers still have workaround code for it (in any case sending the whole request at once it’s a good thing, so something good came out of it). That said it was hardly the only application that assumes TCP is packetized. Lots of stuff break if you lower the network MTU enough.

  9. dave says:

    re: Henning

    The code in question is on a level of abstraction

    that does not know about TCP segments at all.

    Unfortunately, I frequently see code written by people who think that because they wrote one send() call in their client code, exactly those bytes and no others will show up in the one recv() call they wrote in their server code.

    And unfortunately, when they ‘tested’ it on a nice quiet LAN, it did work like that.

    Homework: write out 5000 times "TCP is a byte-stream protocol".

  10. Okay, my mind wasn’t depraved enough to imagine that somebody would think recv() calls on a stream socket would correspond directly to TCP segments, or to the other side’s send() calls. Yes, on reflection I can see it’s a misassumption that holds just often enough to trap the unwary.

  11. dsn says:

    My first assumption was that they were on comcast :P

  12. strik says:

    @dave:

    > Unfortunately, I frequently see code written by

    > people who think that because they wrote one

    > send() call in their client code, exactly those

    > bytes and no others will show up in the one

    > recv() call they wrote in their server code.

    You frequently see this? I *only* see this. At my work, I try to convince my collegues that this assumption is not good. Until now, I could not convince them, as in the field, everything works fine. There was no bug report ever for this, so, they still use this.

Comments are closed.

Skip to main content