Things to ignore when debugging an ASP.NET hang

When looking at a dump, a lot of the art of debugging (I like to call it an art because it makes me feel more important:)) is knowing what you can ignore so you can get to the goodies.

Especially if you are looking at a hang, it’s nice to know what the most common threads are so you can just scan them and say not that one, not that one, well you get the idea.

Since I work mostly with I’m going to dissect a w3wp.exe dump (IIS 6) and show you some of the more common stacks. Common for all of these is that they are all in a sleeping or waiting state, waiting for more work. So if you see them like this in a hang dump, they are not the cause of the hang… if you see them in any other state, they are executing work and then of course they become far more interesting:)

I’ll divide them up into a few different categories

CLR Threads
W3WP Threads
RPC Threads
Other COM Threads

CLR Threads

The ThreadPool manager has five main thread types

Completion Port/IO Threads, and Worker Threads which execute your managed/.net code.
A gate thread that creates or destroys worker threads and IO threads based on CPU utilization, GC frequency and worker queue size.
Wait threads, used for waiting on synchronization objects. I.e. if a developer uses the WaitHandle class, it will internally cause this thread to “waitforsingleobject”.
And finally the timer threads which are used to implement timer callback functionality

This is how they look in their idle states in .net 1.1.

Idle CLR WorkerThread

  11  Id: fb8.268 Suspend: 0 Teb: 7ffaa000 Unfrozen
ChildEBP RetAddr Args to Child
0199fdf8 7c822124 77e6baa8 00000234 00000000 ntdll!KiFastSystemCallRet
0199fdfc 77e6baa8 00000234 00000000 0199fe40 ntdll!NtWaitForSingleObject+0xc
0199fe6c 77e6ba12 00000234 00009c40 00000000 kernel32!WaitForSingleObjectEx+0xac
0199fe80 791d401f 00000234 00009c40 00000000 kernel32!WaitForSingleObject+0x12
0199fea4 791fdacc 00000000 00000000 80a56bcc mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x3a
0199ffb8 77e66063 000b6da8 00000000 00000000 mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44
0199ffec 00000000 791fda8b 000b6da8 00000000 kernel32!BaseThreadStart+0x34

Idle CLR Completion Port / IO Thread

   9  Id: fb8.ef8 Suspend: 0 Teb: 7ffac000 Unfrozen
ChildEBP RetAddr Args to Child
0191fec0 7c821bf4 77e6611a 0000023c 0191ff1c ntdll!KiFastSystemCallRet
0191fec4 77e6611a 0000023c 0191ff1c 0191ff08 ntdll!NtRemoveIoCompletion+0xc
0191fef0 791fdb22 0000023c 0191ff18 0191ff1c kernel32!GetQueuedCompletionStatus+0x29
0191ff24 791fdacc 00000001 809a05fe 7ffac000 mscorsvr!ThreadpoolMgr::CompletionPortThreadStart+0x49
0191ffb8 77e66063 000b6da8 00000000 00000000 mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44
0191ffec 00000000 791fda8b 000b6da8 00000000 kernel32!BaseThreadStart+0x34

Idle CLR Gate Thread

  10  Id: fb8.bdc Suspend: 0 Teb: 7ffab000 Unfrozen
ChildEBP RetAddr Args to Child
0195fe50 7c821364 77e42439 00000000 0195fe94 ntdll!KiFastSystemCallRet
0195fe54 77e42439 00000000 0195fe94 00000000 ntdll!NtDelayExecution+0xc
0195febc 77e424b7 000001f4 00000000 0195ffb8 kernel32!SleepEx+0x68
0195fecc 791bf4f9 000001f4 0b2646e6 00000939 kernel32!Sleep+0xf
0195ffb8 77e66063 00000000 00000000 00000000 mscorsvr!ThreadpoolMgr::GateThreadStart+0x54
0195ffec 00000000 791bf4a5 00000000 00000000 kernel32!BaseThreadStart+0x34

Idle CLR Wait Thread

  25  Id: 230.ac0 Suspend: 0 Teb: 7ff4c000 Unfrozen
ChildEBP RetAddr Args to Child
1b88ff6c 7c573a4e 00000001 1b88ff84 00000000 NTDLL!ZwDelayExecution+0xb
1b88ff8c 7923558c ffffffff 00000001 198ba828 KERNEL32!SleepEx+0x32
1b88ffb4 7c57438b 00000000 198ba828 197f7498 mscorsvr!ThreadpoolMgr::WaitThreadStart+0x45
1b88ffec 00000000 7923556a 1973b3d0 00000000 KERNEL32!BaseThreadStart+0x52

Idle CLR Timer Thread

  18  Id: fb8.914 Suspend: 0 Teb: 7ff5f000 Unfrozen
ChildEBP RetAddr Args to Child
0224ff38 7c821364 77e42439 00000001 0224ff7c ntdll!KiFastSystemCallRet
0224ff3c 77e42439 00000001 0224ff7c 000003e8 ntdll!NtDelayExecution+0xc
0224ffa4 791cc578 000003e8 00000001 00000000 kernel32!SleepEx+0x68
0224ffb8 77e66063 00000000 00000000 00000000 mscorsvr!ThreadpoolMgr::TimerThreadStart+0x30
0224ffec 00000000 791cc548 00000000 00000000 kernel32!BaseThreadStart+0x34

Apart from these we also have some other threads created by the CLR:

On a multi-proc box, in a process that hosts the CLR (server version of GC) we have one Garbage Collector thread per processor or two per proc if running on a hyper threaded system. These take care of garbage collecting the managed heaps.

We have one finalizer thread per process, and this is where all your finalize methods run. In both hang and high memory situations, watch out for this thread. If it is blocked in any way when finalizing an object your managed variables will not be properly cleaned up and you may end up with a memory leak, and/or a high cpu situation since the GC will have to work over time to keep your memory usage up.

Finally, we have the DebuggerThread used for interfacing with the Debugging API’s

Idle CLR GC Thread

  13  Id: fb8.c10 Suspend: 0 Teb: 7ffa8000 Unfrozen
ChildEBP RetAddr Args to Child
01d6fefc 7c822124 77e6baa8 000002b4 00000000 ntdll!KiFastSystemCallRet
01d6ff00 77e6baa8 000002b4 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
01d6ff70 77e6ba12 000002b4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
01d6ff84 791fe6b0 000002b4 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
01d6ffac 792356be 00000000 01d6ffec 77e66063 mscorsvr!gc_heap::gc_thread_function+0x2f
01d6ffb8 77e66063 000d2ed8 00000000 00000000 mscorsvr!gc_heap::gc_thread_stub+0x1e
01d6ffec 00000000 792356a0 000d2ed8 00000000 kernel32!BaseThreadStart+0x34

Idle CLR Finalizer Thread

  17  Id: fb8.7b0 Suspend: 0 Teb: 7ffa4000 Unfrozen
ChildEBP RetAddr Args to Child
01e6fdf8 7c822114 77e6711b 00000002 01e6fe48 ntdll!KiFastSystemCallRet
01e6fdfc 77e6711b 00000002 01e6fe48 00000001 ntdll!NtWaitForMultipleObjects+0xc
01e6fea4 77e61075 00000002 793eee08 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
01e6fec0 7927826b 00000002 793eee08 00000000 kernel32!WaitForMultipleObjects+0x18
01e6fee0 791fecf4 000002dc 00000000 792376a4 mscorsvr!WaitForFinalizerEvent+0x5a
01e6ff24 79245681 00000000 809a05fe 7ffa4000 mscorsvr!GCHeap::FinalizerThreadStart+0x96
01e6ffb8 77e66063 000d5500 00000000 00000000 mscorsvr!Thread::intermediateThreadProc+0x44
01e6ffec 00000000 79245640 000d5500 00000000 kernel32!BaseThreadStart+0x34

Idle CLR Debugger Thread

  12  Id: fb8.b30 Suspend: 0 Teb: 7ffa9000 Unfrozen
ChildEBP RetAddr Args to Child
01adfe70 7c822114 77e6711b 00000003 01adfec0 ntdll!KiFastSystemCallRet
01adfe74 77e6711b 00000003 01adfec0 00000001 ntdll!NtWaitForMultipleObjects+0xc
01adff1c 77e61075 00000003 01adff5c 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
01adff38 79238fd6 00000003 01adff5c 00000000 kernel32!WaitForMultipleObjects+0x18
01adffa0 79238f4d 00000000 00000000 00000000 mscorsvr!DebuggerRCThread::MainLoop+0x90
01adffb0 7923a714 01adffec 77e66063 019d1eb0 mscorsvr!DebuggerRCThread::ThreadProc+0x68
01adffb8 77e66063 019d1eb0 00000000 00000000 mscorsvr!DebuggerRCThread::ThreadProcStatic+0xb
01adffec 00000000 7923a709 019d1eb0 00000000 kernel32!BaseThreadStart+0x34

W3wp Threads

The basic threads in w3wp are the main thread, the Thread Pool Threads and the Compression thread.

The main Thread is responsible for starting and shutting down the process. The thread pool threads handle incoming requests and the compression thread just like the name suggests handles compression.

W3wp main thread

.  0  Id: fb8.fd8 Suspend: 0 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr Args to Child
0006fc08 7c822124 77e6baa8 00000184 00000000 ntdll!KiFastSystemCallRet
0006fc0c 77e6baa8 00000184 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
0006fc7c 77e6ba12 00000184 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
0006fc90 5a36467a 00000184 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
0006fca0 5a366e63 00254ff8 5a3af41d 00000000 w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
0006fca8 5a3af41d 00000000 64711da9 00000000 w3dt!UlAtqStartListen+0x2d
0006fcb8 5a3bc259 0100141c 010013e4 010012d0 w3core!W3_SERVER::StartListen+0xbd
0006ff0c 0100187c 00000007 002538e0 00000000 w3core!UlW3Start+0x26e
0006ff44 01001a23 00000007 002538e0 002543d0 w3wp!wmain+0x22a
0006ffc0 77e523cd 00000000 00000000 7ffd7000 w3wp!wmainCRTStartup+0x12b
0006fff0 00000000 010018f8 00000000 78746341 kernel32!BaseProcessStart+0x23

Idle w3wp ThreadPoolThread

   2  Id: fb8.c5c Suspend: 0 Teb: 7ffd9000 Unfrozen
ChildEBP RetAddr Args to Child
00c8ff24 7c821bf4 77e6611a 00000170 00c8ff80 ntdll!KiFastSystemCallRet
00c8ff28 77e6611a 00000170 00c8ff80 00c8ff6c ntdll!NtRemoveIoCompletion+0xc
00c8ff54 5a30249e 00000170 00c8ff7c 00c8ff80 kernel32!GetQueuedCompletionStatus+0x29
00c8ff8c 5a3026bc 00000000 00258580 5a300000 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x33
00c8ffa0 5a301db9 00000102 00000000 00000000 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x24
00c8ffb8 77e66063 00258580 00000000 00000000 w3tp!THREAD_MANAGER::ThreadManagerThread+0x39
00c8ffec 00000000 5a301d80 00258580 00000000 kernel32!BaseThreadStart+0x34

Idle w3wp Compression Thread

   7  Id: fb8.b20 Suspend: 0 Teb: 7ffaf000 Unfrozen
ChildEBP RetAddr Args to Child
017dfa84 7c822124 77e6baa8 000001d0 00000000 ntdll!KiFastSystemCallRet
017dfa88 77e6baa8 000001d0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
017dfaf8 77e6ba12 000001d0 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
017dfb0c 5a3b8147 000001d0 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
017dffb8 77e66063 00000000 00000000 00000000 w3core!HTTP_COMPRESSION::CompressionThread+0x126
017dffec 00000000 5a3b8021 00000000 00000000 kernel32!BaseThreadStart+0x34

RPC Threads

Having idle LRPC worker threads means that we can handle more local and remote COM activation and local COM method calls.

Having idle RPC worker threads means that we can manage more remote COM method calls.

See this doc for more info on the RPC Threading Model.

Idle RPC Worker Thread

  29  Id: fb8.7fc Suspend: 0 Teb: 7ffa1000 Unfrozen
ChildEBP RetAddr Args to Child
02ddfeac 7c821bf4 77e6611a 00000078 02ddff04 ntdll!KiFastSystemCallRet
02ddfeb0 77e6611a 00000078 02ddff04 02ddfef4 ntdll!NtRemoveIoCompletion+0xc
02ddfedc 77c604c3 00000078 02ddff14 02ddff04 kernel32!GetQueuedCompletionStatus+0x29
02ddff18 77c60655 ffffffff 02ddff6c 02ddff70 rpcrt4!COMMON_ProcessCalls+0xa1
02ddff84 77c5f9f1 02ddffac 77c5f7dd 00090250 rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+0x117
02ddff8c 77c5f7dd 00090250 00000000 00000000 rpcrt4!ProcessIOEventsWrapper+0xd
02ddffac 77c5de88 0008e610 02ddffec 77e66063 rpcrt4!BaseCachedThreadRoutine+0x9d
02ddffb8 77e66063 029515e8 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b
02ddffec 00000000 77c5de6d 029515e8 00000000 kernel32!BaseThreadStart+0x34

Idle LRPC Worker Thread

  33  Id: fb8.b24 Suspend: 0 Teb: 7ffdb000 Unfrozen
ChildEBP RetAddr Args to Child
02e1fe18 7c821c54 77c7538c 00000138 02e1ff74 ntdll!KiFastSystemCallRet
02e1fe1c 77c7538c 00000138 02e1ff74 00000000 ntdll!ZwReplyWaitReceivePortEx+0xc
02e1ff84 77c5778f 02e1ffac 77c5f7dd 0009d660 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
02e1ff8c 77c5f7dd 0009d660 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd
02e1ffac 77c5de88 0008e610 02e1ffec 77e66063 rpcrt4!BaseCachedThreadRoutine+0x9d
02e1ffb8 77e66063 05057c08 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b
02e1ffec 00000000 77c5de6d 05057c08 00000000 kernel32!BaseThreadStart+0x34

Other COM Threads

Single threaded components must be handled by the Host (Main) STA Thread which is the first COM STA Thread created.

Having an idle STA Thread means that we can handle more STA activation and method calls.

For more info see

Host STA Thread

31  Id: fb8.b3c Suspend: 0 Teb: 7ff4e000 Unfrozen
ChildEBP RetAddr Args to Child
049cff14 7739c78d 7739c7c0 049cff58 00000000 ntdll!KiFastSystemCallRet
049cff34 77694ff1 049cff58 00000000 00000000 user32!NtUserGetMessage+0xc
049cff74 776cf35b 00007530 77e6ba20 02a40fe0 ole32!CDllHost::STAWorkerLoop+0x72
049cff90 776cf2a3 049cffb8 776b2307 77790438 ole32!CDllHost::WorkerThread+0xc8
049cff98 776b2307 77790438 00000000 02a40fe0 ole32!DLLHostThreadEntry+0xd
049cffac 776b2374 00000000 049cffec 77e66063 ole32!CRpcThread::WorkerLoop+0x1e
049cffb8 77e66063 02a40fe0 00000000 00000000 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x20
049cffec 00000000 776b2354 02a40fe0 00000000 kernel32!BaseThreadStart+0x34

Idle STA Thread

14  Id: e5c.d90 Suspend: 1 Teb: 7ffac000 Unfrozen
ChildEBP RetAddr Args to Child
0140fdcc 77f4372d 77e41bfa 00000003 0140fe1c SharedUserData!SystemCallStub+0x4
0140fdd0 77e41bfa 00000003 0140fe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
0140fe78 77d076f5 00000003 0140fea0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
0140fed4 77d077f5 00000002 0140ff74 ffffffff USER32!RealMsgWaitForMultipleObjectsEx+0x13f
0140fef0 7563439d 00000002 0140ff74 00000000 USER32!MsgWaitForMultipleObjects+0x1d
0140ff84 77bc91ed 000b6eb8 00000000 00000000 COMSVCS!CSTAThread::WorkerLoop+0x1e3
0140ffb8 77e4a990 00268a58 00000000 00000000 msvcrt!_threadstartex+0x6f
0140ffec 00000000 77bc917e 00268a58 00000000 kernel32!BaseThreadStart+0x34

Note: This is by no means a complete list of the types of threads you can see but it should give you a good idea of what you are looking at and what threads you can immediately ignore when searching for the culprit of your hang.

Merry X-mas to all  


Comments (14)

  1. Alex Pronin says:

    Very useful article. But what about .Net 2.0?

  2. Know what is normal so you can recognize the abnormal.

  3. Scott says:

    Thanks for keeping me from wasting my time…this was news to me!  Enjoy the weekend.  Scott

  4. I have talked about a number of different hang/performance issues in my posts. This post is a generic

  5. Michael says:

    Hi Tess,

    Thanks very much for the useful article.  I’m just getting my feet wet in WinDbg  (I’ve been doing .NET for several years but am currently doing C++ in v6.0, and spent a long time trying to get proper symbol resolution in 6.0, only to figure out that it probably won’t work).  All Hail WinDbg.

    Really liked your intro, too;  I also fell in love with assembly language early in my studies/career and do still admire it’s "black and white"ness.

    Interesting; in the hang I’m debugging now, all of the 34 threads are "unfrozen" at ntdll!KiFastSystemCallRet (the search term I used to find this page), except for the debugger thread.  Any suggestions for figuring out what thread’s hung?

    Thanks again,


  6. Tess says:

    Hi Michael,

    It depends a bit on what the debugger thread is doing.  This thread is used to provide info to a managed debugger debugging the process, so if it is not in an idle state (similar to what is shown above) perhaps there is a debugger attached to the process.

    In general for hangs in native code, for high cpu hangs, look for active threads  (doing something other than wait for single or multiple objects), for low CPU hangs, look for threads stuck in critical sections, waiting on socket calls or com calls or db calls.


  7. Michael says:

    Thanks.  From what I can tell, the debugger thread’s not doing much, i.e.

    # 10  Id: 17c4.e94 Suspend: 1 Teb: 7ffaf000 Unfrozen

    ChildEBP RetAddr  Args to Child              

    02ddffc8 7c9507a8 00000005 00000004 00000001 ntdll!DbgBreakPoint

    02ddfff4 00000000 00000000 00000000 00000000 ntdll!DbgUiRemoteBreakin+0x2d

    and all other threads are at ntdll!KiFastSystemCallRet, most of which have several frames of ntdll!ZwWaitForSingleObjector ntdll!ZwWaitForMultipleObjects below them.  

    This is a very low CPU hang, so I’ll focus on the threads that seem to be waiting on sockets or com.

    I’ll also be checking out some of your other posts; I’m hoping that WinDbg can help me fix this thing!

    Thanks again!


  8. Tess says:

    Did you take this dump in crash mode? it looks like you got a dump on a breakpoint which means one of a couple of things

    1. another debugger is attached and stopped at a breakpoint

    2. you have manually enabled gflags and broke at a suspected heap corruption, but typically in that case it is not the debug thread but another thread calling a breakpoint

    3. something in the code calls a breakpoint, but again, in this case it should not be the debugger thread, but code calling into dbgbreakpoint

    So, from what it looks like there is actually another debugger attached to the process, stopping at a bp which would of course make the process "hang"

  9. Michael says:

    Thanks for the reply, Tess.

    (I of course read your polite request that people don’t contact you directly for help, and I really appreciate your taking the time to provide additional info).

    The problem I’m trying to resolve is an application hang in my DirectShow application.  I got the stack trace as follows:

    1. Open up WinDbg and did "Open Executable" of the Debug version of my app

    2. Create the hang condition in the app

    3. Use WinDbg’s Break command to interrupt execution

    4. Issue the "~*kb" command to dump all thread stacks (34 of ’em).

    I could send you the dump of the 34 stacks, but don’t want to tke advantage of your helpfulness.

    Thanks again for the guidance, I’ll keep at it.


  10. Michael says:

    Hi Tess,

    I’m very happy to say that with your WinDbg guidance and some excellent help from Geraint over in the DirectShow forums, I’ve been able to code a workaround to the web cam driver hangs that were driving me crazy.

    Thanks again for your help!


  11. I often get questions like "what is this thread doing?". A lot of the time it is about threads that are

  12. abhinawsharma says:

    Very good article.

    I was analysing a hang dump file on Win2003 Sp2 COM+ application (VB6).

    appreciate any help on this . Here is the stack and



    00000000 ??              ???

    EXCEPTION_RECORD:  ffffffff — (.exr 0xffffffffffffffff)

    ExceptionAddress: 00000000

      ExceptionCode: 80000007 (Wake debugger)

     ExceptionFlags: 00000000

    NumberParameters: 0

    BUGCHECK_STR:  80000007

    PROCESS_NAME:  dllhost.exe

    ERROR_CODE: (NTSTATUS) 0x80000007 – {Kernel Debugger Awakened}  the system debugger was awakened by an interrupt.




    Dl Eid Cid     WaitType

    — — ——- ————————–

      0   f34.f40 Event                  

    WAIT_CHAIN_COMMAND:  ~0s;k;;

    BLOCKING_THREAD:  00000f40



    LAST_CONTROL_TRANSFER:  from 7c827d0b to 7c8285ec

    FAULTING_THREAD:  00000f40


    0007fcd4 7c827d0b 77e61d1e 00000078 00000000 ntdll!KiFastSystemCallRet

    0007fcd8 77e61d1e 00000078 00000000 00000000 ntdll!NtWaitForSingleObject+0xc

    0007fd48 77e61c8d 00000078 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac

    0007fd5c 77727b21 00000078 ffffffff ffffffff kernel32!WaitForSingleObject+0x12

    0007fd78 77728683 0009a6b0 0007fdc7 00000000 ole32!CSurrogateProcessActivator::WaitForSurrogateTimeout+0x55

    0007fd90 01001383 0007fdac 00000000 00000000 ole32!CoRegisterSurrogateEx+0x1dd

    0007ff1c 010016a1 01000000 00000000 000924b0 dllhost!WinMain+0xe3

    0007ffc0 77e6f23b 00000000 00000000 7ffdc000 dllhost!WinMainCRTStartup+0x182

    0007fff0 00000000 0100151f 00000000 78746341 kernel32!BaseProcessStart+0x23



    01001383 ff158c100001    call    dword ptr [dllhost!_imp__CoUninitialize (0100108c)]


    FOLLOWUP_NAME:  MachineOwner

    MODULE_NAME: dllhost

    IMAGE_NAME:  dllhost.exe


    SYMBOL_NAME:  dllhost!WinMain+e3

    STACK_COMMAND:  ~0s ; kb

    FAILURE_BUCKET_ID:  80000007_dllhost!WinMain+e3

    BUCKET_ID:  80000007_dllhost!WinMain+e3

    Followup: MachineOwner

  13. If I were to pick out ten keywords for my blog I would pick, in no particular order,, windbg,