Hangs and how to solve them – part 2 – Queuing


So in a previous post, we began discussing hangs, Hangs and how to solve them – part 1 – Deadlocks. So we will now move on to another common problem we see with ASP.NET…  Request Queuing.


How to tell Queuing is happening


So the first step is to see if this is really your problem.  One way to see this is to look at PerfMon.  There is a performance object for ASP.NET Applications (be sure to pick the one for the version of the framework you are using).  This is where you will find the Requests in Application Queue counter.  This one is what you want to use prior to 2.0 to monitor your queuing.


RequestInAppQueue


In 2.0 and later, you want to look until the ASP.NET v2.0.50727 object (again, best to pick the versioned one) and look at the Requests Queued counter.


RequestQueue


Alternatively, you can look at a dump of the process to determine if queuing is happening.  In 2.0 you can just load the 2.0 sos and run !threadpool.


20Queue


For earlier versions, you need to do a bit more, you first run !dumpheap -stat


dumpheap


And then !dumpheap -mt <address of System.Web.RequestQueue>


queues


Then dump out the RequestQueue objects using !dumpobj <object>


AQueue


This now gives you the _localQueue and _externQueue.  Dumping them out will show if anything is in them


LocalQueue


You can see that the _head, _tail, _size are all 0.  If there was data in here, we would then dump out the _array and we could look at the items in the array.  Here we are using !do -v <queue> to print out the items in the array.


queue



If you aren’t getting this data from the source, maybe you should check out the original: http://blogs.msdn.com/tom

How to fix queuing


So if we have determined that queuing is happening, what should we do about it?  Well, obviously we don’t want that.  It can cause request times to be larger and can cause requests to time out.   The first thing to check is to make sure our threadpool settings are set according to what the recommended settings are.



Note: In 2.0 and later, if you leave the default of autoconfig=true, you will have these settings


The recommended settings are, from: http://msdn2.microsoft.com/en-us/library/ms998583.aspx


Table 17.7: Recommended Threading Settings for Reducing Contention



























Configuration setting Default (.NET 1.1) Recommended value
maxconnection 2 12* #CPUs
maxIoThreads 20 100
maxWorkerThreads 20 100
minFreeThreads 8 88 * #CPUs
minLocalRequestFreeThreads 4 76 * #CPUs

Please note that the default and recommended values are shown, and that if hyperthreading is enabled, you want to treat a hyperthreaded CPU as two.  So if you look at the Performance tab of TaskManager, however many CPU’s it is showing is the number you want to use.  For example, if I have a dual-proc box that is hyperthreaded, I will use 4 CPU’s and the numbers become:


















maxconnection 48
maxIoThreads 100
maxWorkerThreads 100
minFreeThreads 352
minLocalRequestFreeThreads 304

This will generally fix the problem if the settings haven’t already been adjusted in versions prior to 2.0.  If you have already made this change or are using 2.0 with the default settings, then you will have to do some more tuning.  You may want to see if there is a call to a database that is taking a long time.  Or a WebService call.  If so, you can try to cache the results if they are used by multiple users.  Another option is to balance your load across two webservers.


You can get more information about load balancing at:



kick it on DotNetKicks.com

Comments (9)

  1. You’ve been kicked (a good thing) – Trackback from DotNetKicks.com

  2. Yuanjian says:

    .NET: How to Build a REST app in .NET (with WCF) IIS7 – Mobile Admin What is wrong with the ASP.NET Community

  3. Josh Coswell says:

    Insightfull and I imagine queuing would happen when there are a lot of concurrent requests.

    Thanks

    Josh Coswell

    http://riverasp.net

  4. tomchris says:

    That is correct.  If more requests come in then we have threads to handle the requests, we will queue them up until we have a thread to execute the request on.

  5. 【原文地址】 March 28th Links: ASP.NET, ASP.NET AJAX, ASP.NET MVC, Visual Studio, Silverlight, .NET 【原文发表日期

  6. Collegamenti del 28 Marzo, ASP.NET, ASP.NET AJAX, ASP.NET MVC, Visual Studio, Silverlight, .NET

  7. Aqui está a última versão da minha lista de links . Verifique também minha página de Dicas, Truques e

  8. So I was hoping to get some more comments to this before posting the answer but I guess that isn’t going

  9. Mike Volodarsky says:

    Probably the best writeup about troubleshooting queueing that I've seen!

    Unfortunately, these queues are no longer used in Integrated mode on IIS 7 and later.  If you are looking for more info on IIS & ASP.NET request queueing, this may also help:

    blog.leansentry.com/…/all-about-iis-asp-net-request-queues

    This also goes over how to get the requests stuck in the queue without debugging (not as precise as this post, but should be good enough in most cases).

    Best,

    Mike