Server too busy exception

  • Article

The "Server too busy" error is a common one that causes a lot of confusion when related to WCF. It is possible for WCF to respond with a server too busy fault when a quota is exceeded in the security layer. However you wouldn't get this when you hit a WCF throttle limit. Server too busy typically comes from ASP.Net.

As a case in point, I recently looked into an issue where a user was running a web-hosted WCF service in a performance test. The test was to push 10,000 simultaneous requests onto the server. They hit the server too busy problem at around 5,000 requests. To diagnose the issue, I asked for a dump file. When examining the dump file, the first thing I checked was the WCF throttles. For information on how to do that, check out this post: https://blogs.msdn.com/b/dmetzgar/archive/2011/02/01/checking-a-dump-file-for-wcf-throttles.aspx

All the counts I saw were 0 so we didn't have anything laying around in the WCF layer. The next command I used was !threads to see how many threads were in the process. It turned out there were only 147 threads and only a handful were IO completion threads (the kind WCF uses). This would generally point to ASP.Net being the culprit. Just to be sure I opened a log file like this: .logopen ThreadStacks.log and then ran ~*k to dump the stacks for every thread. I then closed the log with .logclose. Using notepad, I opened the log file and searched for ServiceModel. If there are any stacks with ServiceModel in them, then WCF is involved. However, the find did not turn up a match.

Usually one of the things I revert to is to look for anything with Queue in the name in the heap. So I ran !dumpheap -type Queue -stat and found an interesting object called System.Web.RequestQueue. There was only one instance of this class and it looks like this:

 0:018> !do 02954c74 
Name:        System.Web.RequestQueue
MethodTable: 6436d35c
EEClass:     64178a2c
Size:        64(0x40) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_32\System.Web\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Web.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6de72978  4001082       14         System.Int32  1 instance      352 _minExternFreeThreads
6de72978  4001083       18         System.Int32  1 instance      304 _minLocalFreeThreads
6de72978  4001084       1c         System.Int32  1 instance     5000 _queueLimit
6de6aedc  4001085       2c      System.TimeSpan  1 instance 02954ca0 _clientConnectedTime
6de7662c  4001086       28       System.Boolean  1 instance        0 _iis6
6de69f24  4001087        4 ...Collections.Queue  0 instance 02954cb4 _localQueue
6de69f24  4001088        8 ...Collections.Queue  0 instance 02954cd8 _externQueue
6de72978  4001089       20         System.Int32  1 instance     5000 _count
6de627b8  400108a        c ...ding.WaitCallback  0 instance 02954d8c _workItemCallback
6de72978  400108b       24         System.Int32  1 instance        0 _workItemCount
6de7662c  400108c       29       System.Boolean  1 instance        0 _draining
6de6aedc  400108d       34      System.TimeSpan  1 instance 02954ca8 _timerPeriod
6de6ab08  400108e       10 ...m.Threading.Timer  0 instance 02954dcc _timer

As you can see, there is a _queueLimit of 5000 and the current _count is 5000. That would make sense why we're getting the server too busy exception. Going on this, I found this TechNet article: https://technet.microsoft.com/en-us/library/dd425294(office.13).aspx

The request queue limit is therefore something that can be controlled by the <processModel> tag in the machine.config. Note that when you change something in processModel, you need to turn autoConfig off. The customer indicated that in addition to changing the requestQueueLimit they also had to run the following:

 appcmd.exe set config /section:serverRuntime /appConcurrentRequestLimit:10000

Note that this is for IIS7. IIS6 is set up differently and you should be able to get more information from the TechNet article linked above.