VectorSendThrottleLimit in IIS, shall we… shan’t we?


Okay, first of all a disclaimer... don't try this on a production box. I am lucky enough to have a spare Win 2k3 box to play with and do the changes required. So without any further ado, let me point out a KB which is often misread... 916984. Recently, I noticed that there were a few support calls where customers  installed this hotfix and made the appropriate changes. Instead of fixing the issue, they started facing IIS Hangs and High CPU issues! Naturally, something went wrong... In this post, I will try to explain what exactly went wrong and how to avoid such mishaps.

<excerpt>

SYMPTOMS

You experience high memory usage in the W3wp.exe process on a Microsoft Windows Server 2003-based computer that has Microsoft Internet Information Services (IIS) 6.0 installed.

This problem occurs during the transfer of a large HTTP response that is generated by an Active Server Pages (ASP) application to a client computer. An example of an ASP method that can generate this large HTTP response is the Response.BinaryWrite method. The memory usage in the W3wp.exe process may be as high as the size of the whole HTTP response that is being transferred. Typically, this problem occurs when the client computer uses a slow connection link.

</excerpt>

The problem arises if you don't read the warning section...

<excerpt>

Warning If you set the VectorSendThrottleLimit subkey to a value that is less than the amount of data that an ASP page sends to a client, IIS stops responding (hangs). For example, if you set the VectorSendThrottleLimit subkey to 1000, any ASP page that transfers more than 1000 bytes is not dispayed in the client browser. Additionally, the IIS thread that handles the request is blocked until you restart the IIS service.

</excerpt>

Before I try to explain what all this means in plain English...

First of all, I have created a page called VectorThrottle.asp with the following content

<%
      Response.Buffer = TRUE
      Server.ScriptTimeout = 1200
      Response.AddHeader "Content-Disposition", "attachment; filename=test.txt"
      Response.ContentType = "text/plain"
      'Create a variable called thing which is 1 KB in size
      Dim thing
      thing = String(1024, "A")
      dim Limit
      Limit = 1024 * 2    'This will create a 2 MB file
      For i = 1 To Limit - 1
           Response.Write thing
           Response.Flush
      Next 
%>

You will notice that the file goes through pretty easily. Try saving the file and you will see that the size is as follows...

image

Response.BinaryWrite should have been a better example but I wanted to show that the issue which I am going to talk about happens with Response.Write as well. Although, it seems like everything is fine... it isn't actually a very robust code if you think about it.

Let's do a simple change, and that is to change the Limit in the asp page to 1024 * 100, so that this page starts sending 100 MB files to the client. Once again, everything will appear fine and the file will pass through. Now just open your Task Manager and repeat the exercise. You will find the CPU spike and will see the memory climb up like crazy!!! Naturally, if you have multiple such requests... you server will cry sooner or later.

BEFORE the request>

image

AFTER the request>

image

If you click on Save, you will be able to save the file and the Memory usage will go down as soon as the download is complete. But the main Question here is... "Is this a good approach" ?? The answer would be a straight-forward NO. Simply because, you are able to send the file at the cost of high memory and this solution is not scalable at all.

This is the time when this guy VectorSendThrottleLimit chips in. You can set it as per the KB 916984. I have set it to 2097152 (= 2 MB). Let's have a look at what happens next.

This time after the request>

image

I am sure, you might have already noticed the Mem Usage column above. Seems pretty okay, right? Well... see what happens next (after I click on that Save button)...

image

This time, the CPU is busted!!! Once again, as soon as the file is sent to the client, CPU will drop down. It is pretty easy to guess that the For Loop is causing the High CPU issue here...

      Limit = 1024 * 100    'This will create a 100 MB file
      For i = 1 To Limit - 1
           Response.Write thing
           Response.Flush
      Next 

The worst, is yet to come... modify your ASP file as follows...

      Limit = 1024 * 3    'This will create a 3 MB file but we have set our VectorSendThrottleLimit to 2 MB Surprised
      For i = 1 To Limit - 1
           Response.Write thing
      Next 

Now, open 4 different browser instances and browse to this file, you will notice that the pages will never come back. You won't see any prompt at all. If you like you can even close the browser. Normally, from the server's perspective you would guess that everything is fine. But in fact, it is far from it... read that warning again!

If you create a dump of w3wp.exe and check the threads you will find 4 threads (because you made 4 requests) doing the following even after the browsers are closed. I will show how the threads look like internally using WinDBG.

0:027> kL200
ChildEBP RetAddr 
03fcf730 7c827d0b ntdll!KiFastSystemCallRet
03fcf734 77e61d1e ntdll!NtWaitForSingleObject+0xc
03fcf7a4 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03fcf7b8 70a1f1be kernel32!WaitForSingleObject+0x12
03fcf7e0 70a06ae0 asp!CThrottleSection::Enter+0x8e
03fcf820 709e440c asp!CIsapiReqInfo::AddToPendingList+0xf6
03fcfa14 709e419f asp!CIsapiReqInfo::SendClientResponse+0x2bd
03fcfb28 709e4035 asp!CResponse::WriteResponse+0x1d3
03fcfb50 709e3f5d asp!CResponse::FinalFlush+0x268
03fcfb98 709e2753 asp!CHitObj::ViperAsyncCallback+0x43f
03fcfbb4 4a77b5ea asp!CViperAsyncRequest::OnCall+0x92
03fcfbd0 77720d30 comsvcs!CSTAActivityWork::STAActivityWorkHelper+0x32
03fcfc1c 777217dc ole32!EnterForCallback+0xc4
03fcfd7c 776f03b4 ole32!SwitchForCallback+0x1a3
03fcfda8 7769c194 ole32!PerformCallback+0x54
03fcfe40 7772433a ole32!CObjectContext::InternalContextCallback+0x159
03fcfe60 4a77b78c ole32!CObjectContext::DoCallback+0x1c
03fcfecc 4a77bcf2 comsvcs!CSTAActivityWork::DoWork+0x12d
03fcfee4 4a77c7de comsvcs!CSTAThread::DoWork+0x18
03fcff04 4a77cabf comsvcs!CSTAThread::ProcessQueueWork+0x37
03fcff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x190
03fcffb8 77e64829 msvcrt!_threadstartex+0x74
03fcffec 00000000 kernel32!BaseThreadStart+0x34

With that said (and shown), let me proceed to the summary...

SUMMARY
=========
1. VectorSendThrottleLimit is not bad in itself, but if you use it without a clear understanding of what it does, you may end up in IIS Hang scenarios.
2. If you have to set VectorSendThrottleLimit, don't think of a value to set to. Because 2 MB, 5 MB, 20 MB is all application dependent. Take a step back and think....... why are you taxing your ASP pages by sending such a big response in the first place. IMHO, using a custom 3rd party component (or you can create your own) which handles such requests in a better way can be better than using classic ASP trying to achieve this feat. You can also use Response.TransmitFile feature of ASP.NET. OR, you may like to have an FTP solution for those large files.
3. In case you think that you are good with 2 MB, please ensure that none of the pages ever outputs a page greater than that. How can you find it?? Use IIS Logs and parser them using LogParser. Check for the maximum sc-bytes and decide accordingly.

Hope this helps,
Rahul


Share this post :

Quote of the day:
In three words I can sum up everything I've learned about life: it goes on. - Robert Frost

Comments (5)
  1. Mitch says:

    Dear,

    We have exactly the same issue with an application. We changed set the registry key but there is no change.

    Did you configure the key in hexa or decimal?

    Thank you beforehand for your help.

    Regards,

    Mitch

  2. imRahulSoni says:

    Hi Mitch,

    I have added the value in Decimal.

    By the way, in case you are getting into the same issue, an easy way to confirm it would be to take a dump and analyze it accordingly.

    Debug Diagnostics has an analyzer which might catch the problem and show you in black or white.

    HTH,

    Rahul

  3. Mitch says:

    Rahul,

    Many thanks for your quick answer.

    As the registry key seems not to solve our issue and as we don’t have any access on the application code, we will as you told, enter in a debugging process to try to fingerpoint the issue.

    Best Regards,

    Mitch

  4. Balu says:

    When i am running the Asp.Net Website Application , i am getting the error like "Un able to connect Visual Studio". can u please give the solution for that

  5. Barry says:

    I’m having similar issues, although once the memory starts to rise the w3wp processes start to hog the CPU time followed by mysqld, and neither drop back down after, causing a server hang – any thoughts???

Comments are closed.

Skip to main content