Hangs and how to solve them - part 2 - Queuing

So in a previous post, we began discussing hangs, Hangs and how to solve them - part 1 - Deadlocks. So we will now move on to another common problem we see with ASP.NET...  Request Queuing.

How to tell Queuing is happening

So the first step is to see if this is really your problem.  One way to see this is to look at PerfMon.  There is a performance object for ASP.NET Applications (be sure to pick the one for the version of the framework you are using).  This is where you will find the Requests in Application Queue counter.  This one is what you want to use prior to 2.0 to monitor your queuing.

RequestInAppQueue

In 2.0 and later, you want to look until the ASP.NET v2.0.50727 object (again, best to pick the versioned one) and look at the Requests Queued counter.

RequestQueue

Alternatively, you can look at a dump of the process to determine if queuing is happening.  In 2.0 you can just load the 2.0 sos and run !threadpool.

20Queue

For earlier versions, you need to do a bit more, you first run !dumpheap -stat

dumpheap

And then !dumpheap -mt <address of System.Web.RequestQueue>

queues

Then dump out the RequestQueue objects using !dumpobj <object>

AQueue

This now gives you the _localQueue and _externQueue.  Dumping them out will show if anything is in them

LocalQueue

You can see that the _head, _tail, _size are all 0.  If there was data in here, we would then dump out the _array and we could look at the items in the array.  Here we are using !do -v <queue> to print out the items in the array.

queue

If you aren't getting this data from the source, maybe you should check out the original: https://blogs.msdn.com/tom

How to fix queuing

So if we have determined that queuing is happening, what should we do about it?  Well, obviously we don’t want that.  It can cause request times to be larger and can cause requests to time out.   The first thing to check is to make sure our threadpool settings are set according to what the recommended settings are.

Note: In 2.0 and later, if you leave the default of autoconfig=true, you will have these settings

The recommended settings are, from: https://msdn2.microsoft.com/en-us/library/ms998583.aspx

Table 17.7: Recommended Threading Settings for Reducing Contention

Configuration setting Default (.NET 1.1) Recommended value
maxconnection 2 12* #CPUs
maxIoThreads 20 100
maxWorkerThreads 20 100
minFreeThreads 8 88 * #CPUs
minLocalRequestFreeThreads 4 76 * #CPUs

Please note that the default and recommended values are shown, and that if hyperthreading is enabled, you want to treat a hyperthreaded CPU as two.  So if you look at the Performance tab of TaskManager, however many CPU’s it is showing is the number you want to use.  For example, if I have a dual-proc box that is hyperthreaded, I will use 4 CPU’s and the numbers become:

maxconnection 48
maxIoThreads 100
maxWorkerThreads 100
minFreeThreads 352
minLocalRequestFreeThreads 304

This will generally fix the problem if the settings haven’t already been adjusted in versions prior to 2.0.  If you have already made this change or are using 2.0 with the default settings, then you will have to do some more tuning.  You may want to see if there is a call to a database that is taking a long time.  Or a WebService call.  If so, you can try to cache the results if they are used by multiple users.  Another option is to balance your load across two webservers.

You can get more information about load balancing at:

kick it on DotNetKicks.com