HOWTO: Address random 403 errors that cause images to not display

Question:

We are running IIS 5.0 on Win XP Pro, with dual core processors and 1 GByte of memory.

Many of the HTML pages being served reference lots of small jpgs (between 20 and 50). We have had our site up for a few months, with no problems. Recently, (after Christmas), about half of the jpgs are not served on an initial page hit -- with a "hyperlink" shown instead... If the page is refreshed, then some of these undisplayed jpgs will display. After refreshing the page two or three times, all the jpgs are displayed.

The log file says that the undisplayed jpgs caused a 403 error. Yet, the 403 error for any given jpg seems random, as they will display after refreshing, or on a different page hit. In other words, IIS is randomly showing 403 errors.

Additional fact: this behavior is way more pronounced with IE than with Firefox.

We sure would like to know why this problem started (was it a Window's update), and how we can fix it (is there a tuning parameter in IIS that would solve our problem?)...

Many thanks to whoever can give us a clue...

Answer:

You are describing the classic by-design symptom of IIS on a non-Server SKU (like Windows 2000 Professional, XP Professional, XP Professional x64) hitting its 10 connection limit at the OS level and not serving out any more open requests, leading to images and other dereferenced hyperlinks to not display.

The reason it appears random is simply due to timing and relative size of each dereferenced resource (like the image link). If you have larger images that happen to keep the browser's connection open while it uses other connections to retrieve the remaining image links, you may randomly hit the 10 connection limit. This issue also gets more pronounced if you have Integrated Authentication enabled, the browser uses HTTP Pipelining, but it is not very smart about the pipelining...

The reason that after you hit refresh a couple of times all the images display is due to client-side caching. Suppose some random number of images are retrieved successfully with status 200 and remainder return 403.9 Connection-Limit (unfortunately, IIS 5.1 on XP Professional does not log the substatus code which would aid troubleshooting in general). With each refresh, the successfully retrieved images return "304 Not Modified" really fast, and more of the 403.9 images retrieve successfully now... until you hit the 10-connection limit again. With each successive refresh, you incrementally retrieve more and more images successfully as more and more images return "304 Not Modified", until all are basically cached client-side, at which point you see the web page with all linked pictures composed.

And what is a "304 Not Modified"? It is a standard part of HTTP optimization that happens on the initial retrieve of images that returned "200 OK", where IIS gives a hash to the client saying "on future requests to this resource, return this hash and I'll quickly tell you whether you have the latest already and no file download needed (via "304 Not Modified") or if I need to update your value (via another "200 OK" download of the updated content).

This 10 connection limit is by-design for a non-Server SKU. Thus, the "problem" has always existed by-design, is not due to a Windows Update patch, and there is nothing to "fix".

Ways to work-around the limitation include:

  1. Turn Off KeepAlives ( cscript %SYSTEMDRIVE%\Inetpub\AdminScripts\adsutil.vbs SET W3SVC/AllowKeepAlive 0 ). This has other ramifications such as breaking Integrated Authentication, but you probably do not care about it for your anonymous website.
  2. Bump up the soft limit to 40 ( cscript %SYSTEMDRIVE%\Inetpub\AdminScripts\adsutil.vbs SET W3SVC/MaxConnections 40 ). It won't go higher on non-Server SKU, but 40 buys you a little more headroom.

You can also search for "403.9 Connections" and find tons of related articles on this very topic.

//David