How to debug WCF service hang due to ServiceThrottle.

One of the reasons why a WCF service may not be responding to requests (in other words hang) might be that we have hit the ServiceThrottle limit for either number of concurrent calls, instances or sessions.

https://msdn.microsoft.com/en-us/library/system.servicemodel.description.servicethrottlingbehavior.aspx talks about this.
Use the ServiceThrottlingBehavior class to control various throughput settings that help prevent your application from running out of memory.
The MaxConcurrentCalls property specifies the maximum number of messages actively processing across all the dispatcher objects in a ServiceHost object.
The MaxConcurrentInstances property specifies the maximum number of InstanceContext objects in the service.
The MaxConcurrentSessions property specifies the maximum number of sessions a ServiceHost can accept.

The easiest way to configure the ServiceThrottle for a service is to use the ServiceThrottlingBehavior especially from a configuration file.
<system.serviceModel>
    <services>
      <service name="SampleService" behaviorConfiguration="Throttled" >
….
      </service>
    </services>
    <behaviors>
      <serviceBehaviors>
        <behavior  name="Throttled">
          <serviceThrottling
            maxConcurrentCalls="16"
            maxConcurrentSessions="100"
            maxConcurrentInstances="116"/>
…         
        </behavior>
      </serviceBehaviors>
    </behaviors>
</system.serviceModel>

You can find out if you are running into this, by capturing a dump of the process hosting the WCF service and examining the serviceHost object.
If the WCF service is hosted in IIS 6.0, the default process would be w3wp.exe.

Here are the steps:
Capture a dump of the process
========================
Dump is a snapshot of the process, so make sure you capture it when the process is in the hung state, you have a few options to capture the dump.
A. Windows7 and Vista lets you create the dump directly from task manger.
https://support.microsoft.com/kb/931673.
B. Adplus: This is part of debugging tools for windows and can be installed from
https://www.microsoft.com/whdc/devtools/debugging/default.mspx.
You can run the following command to capture a hang dump
      adplus –hang –pn w3wp.exe
or
adplus –hang –p <PID of the w3wp.exe>
C. DebugDiag: This is a graphical interface tool and can be installed from
https://www.microsoft.com/downloads/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en)
After you install debugdiag, you can go to the processes tabàright click on the process and click on create Full user Dump.

Debugging
=========
Open the dump using windbg.exe(part of debugging tools for windows) and load sos using the following command:
               .loadby sos mscorwks

Tess's blog below talks about how the threads look like in this hung condition.
https://blogs.msdn.com/tess/archive/2009/01/09/net-hang-my-application-hangs-after-i-called-my-wcf-service-a-couple-of-times.aspx.

In our case since we are looking for throttle values and these are related to the ServiceHost object, our first task is to find the ServiceHost objects in the dump.
a. Find the method Table of ServiceHost class.
You can get a list of methodTable of all objects in a process and their total memory consumption by using the “!dumpheap –stat” command and looking through the list for the ServiceHost object.

      0:000> !dumpheap –stat
      …….removed data for brevity.
MT Count TotalSize Class Name
       000007fef29692d8    1 248 System.ServiceModel.ServiceHost

b. Now that you have the methodTable for the ServiceHost object, you can obtain the address of the ServiceHost object by using the dumpheap command with –mt option and passing in the methodTable you got from the above command.
Note: You may have multiple ServiceHost objects depending on your application, so you will need to repeat this step for each of the ServiceHost objects. (I have a short-cut at the end to dump out these values directly)

                       0:000> !dumpheap -mt 000007fef29692d8
Address MT Size
      000000017f157d48 000007fef29692d8 248    

2) Now, that you have the ServiceHost object, let’s start examining it. You can dump the object using the “!do“ command and passing in the address of the object from above.
       0:000> !do 000000017f157d48
Name: System.ServiceModel.ServiceHost
MethodTable: 000007fef29692d8
EEClass: 000007fef2332dd0
Size: 248(0xf8) bytes
(C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef5dc5928 400098a 4c System.Boolean 1 instance 0 aborted
000007fef5dc5928 400098b 4d System.Boolean 1 instance 0 closeCalled
000007fef29194d8 400098c 8 ...ct+ExceptionQueue 0 instance 0000000000000000 exceptionQueue
000007fef5dc6048 400098d 10 System.Object 0 instance 000000017f157f40 mutex
000007fef5dc5928 400098e 4e System.Boolean 1 instance 0 onClosingCalled
000007fef5dc5928 400098f 4f System.Boolean 1 instance 0 onClosedCalled
000007fef5dc5928 4000990 50 System.Boolean 1 instance 1 onOpeningCalled
000007fef5dc5928 4000991 51 System.Boolean 1 instance 1 onOpenedCalled
000007fef5dc5928 4000992 52 System.Boolean 1 instance 0 raisedClosed
000007fef5dc5928 4000993 53 System.Boolean 1 instance 0 raisedClosing
000007fef5dc5928 4000994 54 System.Boolean 1 instance 0 raisedFaulted
000007fef5dc5928 4000995 55 System.Boolean 1 instance 0 traceOpenAndClose
000007fef5dc6048 4000996 18 System.Object 0 instance 000000017f157d48 eventSender
000007fef2908538 4000997 48 System.Int32 1 instance 2 state
000007fef5dbe168 4000998 20 System.EventHandler 0 instance 00000001bf234d90 Closed
000007fef5dbe168 4000999 28 System.EventHandler 0 instance 0000000000000000 Closing
000007fef5dbe168 400099a 30 System.EventHandler 0 instance 000000017f1583f8 Faulted
000007fef5dbe168 400099b 38 System.EventHandler 0 instance 0000000000000000 Opened
000007fef5dbe168 400099c 40 System.EventHandler 0 instance 0000000000000000 Opening
000007fef5dc5928 4002e7e 56 System.Boolean 1 instance 1 initializeDescriptionHasFinished
000007fef295fde8 4002e7f 58 ...meKeyedCollection 0 instance 000000017f157f58 baseAddresses
000007fef295ffe8 4002e80 60 ...patcherCollection 0 instance 000000017f157fb8 channelDispatchers
000007fef5e06b90 4002e81 c0 System.TimeSpan 1 instance 000000017f157e08 closeTimeout
000007fef295f958 4002e82 68 ...erviceDescription 0 instance 000000017f158690 description
0000000000000000 4002e83 70 0 instance 000000017f158028 extensions
0000000000000000 4002e84 78 0 instance 0000000000000000 externalBaseAddresses
0000000000000000 4002e85 80 0 instance 000000017f160580 implementedContracts
000007fef2968258 4002e86 88 ...nceContextManager 0 instance 000000017f158098 instances
000007fef5e06b90 4002e87 c8 System.TimeSpan 1 instance 000000017f157e10 openTimeout
000007fef28c8368 4002e88 90 ...rformanceCounters 0 instance 0000000000000000 servicePerformanceCounters
000007fef296a1f0 4002e89 98 ...rformanceCounters 0 instance 00000001bf266660 defaultPerformanceCounters
      000007fef2960500 4002e8a a0 ...r.ServiceThrottle 0 instance 000000017f1580d8 serviceThrottle
000007fef295f708 4002e8b a8 ...erviceCredentials 0 instance 0000000000000000 readOnlyCredentials
000007fef295f840 4002e8c b0 ...orizationBehavior 0 instance 00000001bf2fdf30 readOnlyAuthorization
0000000000000000 4002e8d b8 0 instance 0000000000000000 UnknownMessageReceived
000007fef526c640 4002e7d f80 System.Uri 0 shared static EmptyUri
>> Domain:Value 000000000204e5d0:NotInit 00000000020e8080:000000017f157e98 <<
000007fef5dc6048 40032d6 d0 System.Object 0 instance 0000000000000000 singletonInstance
000007fef5dc8430 40032d7 d8 System.Type 0 instance 000000017f157d20 serviceType
000007fef29609c8 40032d8 e0 ...ontractCollection 0 instance 000000017f159ad8 reflectedContracts
000007fef5dc1010 40032d9 e8 System.IDisposable 0 instance 0000000000000000 disposableInstance

Looking through the list of members for the ServiceHost object, the serviceThrottle object seems to be the one we are, so lets dump that out next.
      0:000> !do 000000017f1580d8
Name: System.ServiceModel.Dispatcher.ServiceThrottle
MethodTable: 000007fef2960500
EEClass: 000007fef232c348
Size: 72(0x48) bytes
(C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef2960578 4003727 8 ...cher.FlowThrottle 0 instance 000000017f158240 calls
      000007fef2960578 4003728 10 ...cher.FlowThrottle 0 instance 000000017f158370 sessions
000007fef28e0c18 4003729 18 ...her.QuotaThrottle 0 instance 0000000000000000 dynamic
000007fef2960578 400372a 20 ...cher.FlowThrottle 0 instance 0000000000000000 instanceContexts
000007fef295f3b0 400372b 28 ...l.ServiceHostBase 0 instance 000000017f157d48 host
000007fef5dc5928 400372c 38 System.Boolean 1 instance 1 isActive
000007fef5dc6048 400372d 30 System.Object 0 instance 000000017f1581a8 thisLock

Since we are investigating the number of sessions, we can examine the sessions object in the ServiceThrottle object.
You would follow similar steps if you want to investigate concurrentcalls or concurrentInstances by going after calls or instancecontexts respectively. Each of these throttles is an instance of System.ServiceModel.Dispatcher.FlowThrottle.
      0:000> !do 000000017f158370
Name: System.ServiceModel.Dispatcher.FlowThrottle
MethodTable: 000007fef2960578
EEClass: 000007fef232c400
Size: 64(0x40) bytes
(C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
MT Field Offset Type VT Attr Value Name
      000007fef5dcd9c8 40034ff 30 System.Int32 1 instance 10 capacity
000007fef5dcd9c8 4003500 34 System.Int32 1 instance 10 count

000007fef5dc6048 4003501 8 System.Object 0 instance 000000017f1583b0 mutex
000007fef5db4d80 4003502 10 ...ding.WaitCallback 0 instance 000000017f158330 release
0000000000000000 4003503 18 0 instance 000000017f1583c8 waiters
000007fef5dc6758 4003504 20 System.String 0 instance 000000017f158160 propertyName <<<MaxConcurrentSessions
000007fef5dc6758 4003505 28 System.String 0 instance 000000017f1582e8 configName

Let us examine where we are with the sessions capacity. Capacity above indicates the value set by the maxConcurrentSessions ServiceThrottle and count indicates the current sessions that are active. As you can see above we have maxed out the capacity and thus hit the throttle limit.

Short-Cut:
=========
There could be several instances of ServiceHost based on your application , so we need to check for all ServiceHost objects to see if we are blocked by the throttles. Here is a command to do that
            .foreach (Var {!dumpheap -mt 000007fef4a80a48 -short}) {.echo FlowThrottle: Var;!do Var }

Again, 000007fef4a80a48 is the Method Table of System.ServiceModel.Dispatcher.FlowThrottle, which you can easily find from !dumheap –stat command.
Scan through the output to examine the values of ‘capacity’ and ‘count’. If they are equal, then some throttle is getting maxed.