COM+ Application sometime hangs after moving to Windows 2003 SP2
Symptom
============
Customer has one COM+ application running in Windows 2000 well for long time. After moving this COM+ application to Windows 2003 SP2, they noticed that the COM+ application easily hangs under heavy load.
Captured dump files for analysis. When issue happens, multiple STA activities stuck on one STA thread under heavy load:
0:008> kL 100
ChildEBP RetAddr
01fd8124 7c827cfb ntdll!KiFastSystemCallRet
01fd8128 77e6202c ntdll!NtWaitForMultipleObjects+0xc
01fd81d0 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
01fd822c 776c202a user32!RealMsgWaitForMultipleObjectsEx+0x141
01fd8254 776c1da4 ole32!CCliModalLoop::BlockFn+0x7d
01fd827c 776e24b6 ole32!ModalLoop+0x5b
01fd828c 7778b5bd ole32!SwitchSTA+0x21
01fd82ac 7778c38b ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xc0
01fd838c 776c1fcb ole32!CRpcChannelBuffer::SendReceive2+0xd3
…….
01fdb07c 7778cfc8 ole32!SyncStubInvoke+0x37
01fdb0c4 776c123b ole32!StubInvoke+0xa7
01fdb1a0 776c0c25 ole32!CCtxComChnl::ContextInvoke+0xec
01fdb1bc 776bc484 ole32!MTAInvoke+0x1a
01fdb1e8 7778ced5 ole32!STAInvoke+0x48
01fdb21c 7778cd66 ole32!AppInvoke+0xa3
01fdb2f0 7778c24d ole32!ComInvokeWithLockAndIPID+0x2c5
…….
01fdb574 776e24b6 ole32!ModalLoop+0x5b
01fdb584 7778b5bd ole32!SwitchSTA+0x21
01fdb5a4 7778c38b ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xc0
….
01fdfae8 776c123b ole32!StubInvoke+0xa7
01fdfbc4 776c0c25 ole32!CCtxComChnl::ContextInvoke+0xec
01fdfbe0 776bc484 ole32!MTAInvoke+0x1a
01fdfc0c 7778ced5 ole32!STAInvoke+0x48
01fdfc40 7778cd66 ole32!AppInvoke+0xa3
…..
01fdfecc 4a77bc4a comsvcs!CSTAQueueLessMessageWork::DoWork+0x4e
01fdfee4 4a77c736 comsvcs!CSTAThread::DoWork+0x18
01fdff04 4a77ca17 comsvcs!CSTAThread::ProcessQueueWork+0x37
01fdff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x190
01fdffb8 77e64829 msvcrt!_threadstartex+0x74
01fdffec 00000000 kernel32!BaseThreadStart+0x34
Root Cause
============
There is one EnableStrictBinding value that determines whether the STA thread pool takes a hard lock when incoming activities are assigned to STA threads. This behavior was the default behavior in Windows NT 4.0 and in Windows 2000. In Windows XP and later versions, the thread pool takes a read lock instead of a hard lock to improve performance. Therefore, simultaneous incoming activities can simultaneously "select" the same STA thread under load even when other STA threads are idle. This may impact some COM+ application performance when application load is high, just as we see in this case.
In this situation, we can consider to turn on the EnableStrictBinding value, the behavior will be similar to Windows 2000. When the EnableStrictBinding value is turned on, the value makes sure that activities are serially bound to STA threads. The benefit is that simultaneous incoming activities are not bound to the same "least-busy" thread under load. This benefit will help to prevent "activity stacking." The drawback is that the activity scheduling will be slower, so we recommend that you enable the EnableStrictBinding value for only long blocking calls where the cost of activity stacking outweighs the increased cost of scheduling time.
In most situations, if we want to modify the EnableStrictBinding value on Windows 2003, we just need to set EmulateMTSBehavior=1, it is equivalent to setting the following:
MaxThreads=100 ActivitiesPerThread=1 EnableStrictBinding=1, CPUMetricEnabled=0
The hard lock is re-enabled when the EmulateMTSBehavior value is turned on. Also note that the hard lock is only available in Windows Server 2003 Post-Service Pack 1 COM+ Hotfix Rollup Package 5 and later versions of Windows 2003. The hard lock is currently not available in Windows XP.
Restarting the COM+ application is enough for the application after modifying the EmulateMTSBehavior value.
More Information
================
303071 Registry key for tuning COM+ thread and activity
https://support.microsoft.com/default.aspx?scid=kb;EN-US;303071
Regards,
Freist Li