Hangs and how to solve them - part 2 - Queuing
So in a previous post, we began discussing hangs, Hangs and how to solve them - part 1 - Deadlocks. So we will now move on to another common problem we see with ASP.NET... Request Queuing.
How to tell Queuing is happening
So the first step is to see if this is really your problem. One way to see this is to look at PerfMon. There is a performance object for ASP.NET Applications (be sure to pick the one for the version of the framework you are using). This is where you will find the Requests in Application Queue counter. This one is what you want to use prior to 2.0 to monitor your queuing.
In 2.0 and later, you want to look until the ASP.NET v2.0.50727 object (again, best to pick the versioned one) and look at the Requests Queued counter.
Alternatively, you can look at a dump of the process to determine if queuing is happening. In 2.0 you can just load the 2.0 sos and run !threadpool.
For earlier versions, you need to do a bit more, you first run !dumpheap -stat
And then !dumpheap -mt <address of System.Web.RequestQueue>
Then dump out the RequestQueue objects using !dumpobj <object>
This now gives you the _localQueue and _externQueue. Dumping them out will show if anything is in them
You can see that the _head, _tail, _size are all 0. If there was data in here, we would then dump out the _array and we could look at the items in the array. Here we are using !do -v <queue> to print out the items in the array.
If you aren't getting this data from the source, maybe you should check out the original: https://blogs.msdn.com/tom
How to fix queuing
So if we have determined that queuing is happening, what should we do about it? Well, obviously we don’t want that. It can cause request times to be larger and can cause requests to time out. The first thing to check is to make sure our threadpool settings are set according to what the recommended settings are.
Note: In 2.0 and later, if you leave the default of autoconfig=true, you will have these settings
The recommended settings are, from: https://msdn2.microsoft.com/en-us/library/ms998583.aspx
Table 17.7: Recommended Threading Settings for Reducing Contention
Configuration setting | Default (.NET 1.1) | Recommended value |
maxconnection | 2 | 12* #CPUs |
maxIoThreads | 20 | 100 |
maxWorkerThreads | 20 | 100 |
minFreeThreads | 8 | 88 * #CPUs |
minLocalRequestFreeThreads | 4 | 76 * #CPUs |
Please note that the default and recommended values are shown, and that if hyperthreading is enabled, you want to treat a hyperthreaded CPU as two. So if you look at the Performance tab of TaskManager, however many CPU’s it is showing is the number you want to use. For example, if I have a dual-proc box that is hyperthreaded, I will use 4 CPU’s and the numbers become:
maxconnection | 48 |
maxIoThreads | 100 |
maxWorkerThreads | 100 |
minFreeThreads | 352 |
minLocalRequestFreeThreads | 304 |
This will generally fix the problem if the settings haven’t already been adjusted in versions prior to 2.0. If you have already made this change or are using 2.0 with the default settings, then you will have to do some more tuning. You may want to see if there is a call to a database that is taking a long time. Or a WebService call. If so, you can try to cache the results if they are used by multiple users. Another option is to balance your load across two webservers.
You can get more information about load balancing at: