Induced GC happened frequently and caused High CPU when creating Serviced Component in ASP.NET 2.0
Customer reported one High CPU issue while one ASP.NET application is under normal load.
Regarding ASP.NET application, besides the multiple threads busy, tighten looping code defect some general possibilities, looking at the memory usage is quite important because heavy memory pressure may lead to frequent Garbage collection, and this is a kind of expensive operation for CPU resource. If GC often occurs (we verify this through %Time in GC and Induced GC performance counters under .NET CLR Memory Object), then we need to dig out how it happens in this pattern.
Check the Performance monitor, the Induced GC increased quickly and turns to be the culprit of the high CPU issue:
Usually, Induced GC is caused by explicit GC.Collect calls in customer's code, however, after double confirmed with customer's application team and reviewing the code, the code doesn't do such a explicit call at all.
We noticed this is one application which uses Enterprise Serviced Components, thus recommend customer using Dispose instead of DisposeObject:
After implementing this, the performance didn't get better.
By analyzing memory dump thoroughly, we finally found out how the Induced GC happened in this way. Here is the call stack which triggered GC:
Unmanaged Call Stack:
Child-SP RetAddr Call Site
00000000`0f59bbe8 00000000`7704ce00 ntdll!ZwWaitForSingleObject+0xa
00000000`0f59bbf0 000007fe`f52fa74a kernel32!WaitForSingleObjectEx+0x9c
00000000`0f59bcb0 000007fe`f52fa83b mscorwks!CLREventWaitHelper+0x42
00000000`0f59bd10 000007fe`f53cee38 mscorwks!CLREvent::WaitEx+0x63
00000000`0f59bdc0 000007fe`f53c1cb7 mscorwks!SVR::gc_heap::wait_for_gc_done+0x88
00000000`0f59be00 000007fe`f542913b mscorwks!SVR::GCHeap::GarbageCollectGeneration+0x147
00000000`0f59be90 000007fe`f5412ac3 mscorwks!SVR::GCHeap::GarbageCollect+0x5b
00000000`0f59bee0 000007fe`f57301a5 mscorwks!GCInterface::AddMemoryPressure+0x13b
00000000`0f59bf70 000007fe`f57b9beb mscorwks!RCW::AddMemoryPressure+0x15
00000000`0f59bfb0 000007fe`f57dbbf4 mscorwks!RCW::CreateRCW+0x17b
00000000`0f59c030 000007fe`f57dbdea mscorwks!COMInterfaceMarshaler::CreateObjectRef+0x74
00000000`0f59c130 000007fe`f58843f4 mscorwks!COMInterfaceMarshaler::WrapWithComObject+0x3a
00000000`0f59c1a0 000007fe`f0268995 mscorwks!MarshalNative::WrapIUnknownWithComObject+0x134
00000000`0f59c3c0 000007fe`f0270933 System_EnterpriseServices_ni!System.EnterpriseServices.RemoteServicedComponentProxy..ctor(System.Type, IntPtr, Boolean)+0xd5
00000000`0f59c420 000007fe`f0270213 System_EnterpriseServices_ni!System.EnterpriseServices.FastRSCPObjRef.GetRealObject(System.Runtime.Serialization.StreamingContext)+0x33
00000000`0f59c470 000007fe`f37254db System_EnterpriseServices_ni!System.EnterpriseServices.ServicedComponentProxyAttribute.CreateProxy(System.Runtime.Remoting.ObjRef, System.Type, System.Object, System.Runtime.Remoting.Contexts.Context)+0x133
00000000`0f59c500 000007fe`f37253ff mscorlib_ni!System.Runtime.Remoting.RemotingServices.SetOrCreateProxy(System.Runtime.Remoting.Identity, System.Type, System.Object)+0x9b
00000000`0f59c560 000007fe`f372466f mscorlib_ni!System.Runtime.Remoting.RemotingServices.GetOrCreateProxy(System.Runtime.Remoting.Identity, System.Object, Boolean)+0xbf
00000000`0f59c5c0 000007fe`f027050a mscorlib_ni!System.Runtime.Remoting.RemotingServices.InternalUnmarshal(System.Runtime.Remoting.ObjRef, System.Object, Boolean)+0x12f
00000000`0f59c650 000007fe`f40e3edf System_EnterpriseServices_ni!System.EnterpriseServices.ServicedComponentProxyAttribute.CreateInstance(System.Type)+0x2ba
00000000`0f59c7c0 000007fe`f5416e61 mscorlib_ni!System.Runtime.Remoting.Activation.ActivationServices.IsCurrentContextOK(System.Type, System.Object[], Boolean)+0x9ba81f
Managed call Stack:
Child-SP RetAddr Call Site
000000000f59c3c0 000007fef0270933 System.EnterpriseServices.RemoteServicedComponentProxy..ctor(System.Type, IntPtr, Boolean)
000000000f59c420 000007fef0270213 System.EnterpriseServices.FastRSCPObjRef.GetRealObject(System.Runtime.Serialization.StreamingContext)
000000000f59c470 000007fef37254db System.EnterpriseServices.ServicedComponentProxyAttribute.CreateProxy(System.Runtime.Remoting.ObjRef, System.Type, System.Object, System.Runtime.Remoting.Contexts.Context)
000000000f59c500 000007fef37253ff System.Runtime.Remoting.RemotingServices.SetOrCreateProxy(System.Runtime.Remoting.Identity, System.Type, System.Object)
000000000f59c560 000007fef372466f System.Runtime.Remoting.RemotingServices.GetOrCreateProxy(System.Runtime.Remoting.Identity, System.Object, Boolean)
000000000f59c5c0 000007fef027050a System.Runtime.Remoting.RemotingServices.InternalUnmarshal(System.Runtime.Remoting.ObjRef, System.Object, Boolean)
000000000f59c650 000007fef40e3edf System.EnterpriseServices.ServicedComponentProxyAttribute.CreateInstance(System.Type)
When Serviced Component is being created, .NET RCW layer will call GC.AddMemoryPressure in the mscorwks!RCW::CreateRCW call. When the memory resource is under pressure, it will add the Induced GC counter, and then call GarbageCollect. this means the increased Induced GC is from this part.
To resolve this issue, we need to reduce the frequency of creating Serviced Component instances in the application. Suggested customer to create a pool of the Serviced Component, to reduce the frequency of component creation. And based on our testing, before creating Serviced Component, add RemoveMemoryPressure(4004) will also help this issue as a workaround. 4004 is a defined size for local serviced component call used by AddMemoryPressure in the above logic.
Customer finally created a pool of the Serviced Component, and the application runs very well under the heavy load.
Regards,
Freist Li from APGC DSI Team