Process Affinity Again


 


Three months ago, I have a blog talking about worker process affinity. Now, we have a similar issue again, but it is more complex this time.


My customer has SQL reporting service installed on an 8 CPU windows 2008 system. And he experienced severe performance issue as soon as the application started. IE keep show blank page for long time when he tried to generate a report, no matter the report is small or huge.


As you may or may not know, reporting service installs two web applications on IIS, one is “Reports” and another is “ReportServer”. Usually, “Reports” call the web service provided by “ReportServer”. Of course, you can call the web service provided by “ReportServer” directly. And, by default these two applications are hosted in same worker process.


Same, we collected dumps to analysis. From dump, we could see only one “Reports” page was running and this page was calling a web service under “Reportserver” which hosted in the same worker process. Then why we could not see the request for ASMX web service? Looking at the dumps carefully, we found the request was in the application’s queue. Please notice that the HTTPRuntime minFreeThreads for “Reports” and “ReportServer” were different. “ReportSever” was configured for 8 CPUs and “Reports” was set for 4 CPUs.


0:031> !dumprequestqueues


   RequestQueue 0x000000015f766920:


   _minExternFreeThreads: 704


   _minLocalFreeThreads: 608


   _count: 7


   _workItemCount: 0


   _localQueue 0x000000015f766978 size: 3


   _externQueue 0x000000015f766ad0 size: 4


—————–


   RequestQueue 0x000000019f7ac0c0:


   _minExternFreeThreads: 352


   _minLocalFreeThreads: 304


   _workItemCount: 0


   _localQueue 0x000000019f7ac118 size: 0


   _externQueue 0x000000019f7ac270 size: 0


 


Max worker threads are 4 CPU settings. The default settings is 100*#CPU.


0:031> !threadpool


Worker Thread: Total: 8 Running: 1 Idle: 7 MaxLimit: 400 MinLimit: 4


Completion Port Thread:Total: 1 Free: 1 MaxFree: 16 CurrentLimit: 0 MaxLimit: 400 MinLimit: 4


 


By default, MinFreeThreads is set to 88*#CPU, and MinLocalFreeThread is set to 76*#CPU.


Here is the MSDN suggested settings and its details description. As you can see, customer’s setting is exactly what we suggested.




























Configuration Setting


Recommended Value


maxconnection


12 * #CPUs


maxIoThreads


100


maxWorkerThreads


100


minFreeThreads


88 * #CPUs


minLocalRequestFreeThreads


76 * #CPUs


 


If you are not familiar with this, please refer to follow KB.


821268  Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications


http://support.microsoft.com/default.aspx?scid=kb;EN-US;821268


Chapter 10 — Improving Web Services Performance


http://msdn.microsoft.com/en-us/library/ms998562.aspx#scalenetchapt10_topic10


 


As I said, this is an 8 CPU system.


0:031> vertarget


Windows Server 2008/Windows Vista SP1 Version 6001 (Service Pack 1) MP (8 procs) Free x64


 


Then why this setting blocked “ReportServer” accept new requests? Same again, we got the answer via check code in reflector. The incorrect _minExternFreeThreads and _minLocalFreeThreads stop ASP.Net to dequeue a request as the dequeue condition can never be satisfied.



            if (freeThreads >= _minExternFreeThreads) {


                wr = DequeueRequest(false); // enough threads to process even external requests


            }


            else if (freeThreads >= _minLocalFreeThreads) {


                wr = DequeueRequest(true);  // enough threads to process only local requests


            }



            int workerThreads, ioThreads;


            ThreadPool.GetAvailableThreads(out workerThreads, out ioThreads);


 


            // not enough worker threads to do anything


            if (workerThreads < _minLocalFreeThreads)


                return;


 


            // pick up request from the queue


            HttpWorkerRequest wr = DequeueRequest(workerThreads < _minExternFreeThreads);


 


My first think is customer uses customized configuration for ProcessModel, and used different HTTPRumtime configuration for “Reports” and “ReportServer”. However, this is not true. By review customer’s configuration file, we found that customer is using auto configure. Here is the “auto” setting for an 8 CPUs system.


MaxWorkerThreads 100, total = 8*100


MinFreeThreads = 88*8 = 704


MinLocalFreeThreads = 76*8 = 608.


 


Then, why ProcessMode and HTTPRuntime for “Reports” set to 4 CPU values? Look into the dump again, and we found a crazing issue. The CPU counts for “Reports” was 4, and for “ReportServer was 8.


0:031> !do 0x000000019f790380


Name: System.Web.Configuration.ProcessModelSection


MethodTable: 00000642bcf33970


EEClass: 00000642bcf55a50


Size: 120(0x78) bytes


00000642788ca730  4001946      998         System.Int32  1   shared           static cpuCount


                                 >> Domain:Value  0000000001bb2c50:NotInit  0000000001c2f740:8 0000000008221f30:4 <<


 


0:031> !dumpdomain -stat


    Domain        Num Assemblies        Size Assemblies         Name


000006427fc82130                  0                       0        System Domain


000006427fc82ac0                 18            119,934,976            Shared Domain


0000000001bb2c50                10             72,175,616             DefaultDomain


0000000001c2f740                240           161,005,056            /LM/W3SVC/1/ROOT/ReportServer-1-128885691260563502


0000000008221f30                 73            122,132,992            /LM/W3SVC/1/ROOT/Reports-2-128885692243612024


 


And current process affinity is 0xf, which means only 4 CPUs were available to this process.


0:000> dt ntdll!_PEB 000007fffffdf000


   +0x0b8 NumberOfProcessors : 8


   +0x138 ActiveProcessAffinityMask : 0xf


 


This is crazy, right? So, the only reasonable explanation is affinity. A simple test shows that the process affinity of worker process which hosted Reporting Service changed to 4 CPU as long as it is started. However, all other worker processes on this server didn’t experience this issue. We are suspecting “ReportServer” changes the worker process’s affinity itself. Then, we found out the problem by:


1.    Created a test web application which contains simple pages only


2.    Assign Reporting Service’s application pool to the test application.


3.    Start the Reporting Services application pool by access the test application.


4.    Attach a debugger to the worker process, and set breakpoint at Kernel32!SetProcessAffinityMask.


5.    Access “ReportServer”, and then the breakpoint was hit.


Here is the call stack:


0:028> !clrstack


OS Thread Id: 0x61c (28)


Child-SP         RetAddr          Call Site


000000000572e5b0 0000064280713f9d DomainBoundILStubClass.IL_STUB(Void*, UInt64)


000000000572e680 0000064280713a48 Microsoft.ReportingServices.Utilities.ProcessUtilities.SetProcessAffinity(UInt64)


000000000572e6c0 000006428070791a Microsoft.ReportingServices.Diagnostics.ResourceUtilities.ThrottleCpu()


000000000572e730 0000064280706c3b Microsoft.ReportingServices.WebServer.Global.RunOnlyOnceStartReportServer()


000000000572e7a0 00000642807069af Microsoft.ReportingServices.WebServer.Global.StartApp()


000000000572e800 00000642bcb24d10 Microsoft.ReportingServices.WebServer.Global.Application_BeginRequest(System.Object, System.EventArgs)


 


Here is the code via Reflector. From the code, we can see that “ReportServer” changes the affinity if it is not Dev version, ENT version and EVAL version.


public static void InitializeProcessParameters()


{


    if (!Sku.IsFeatureEnabled(ConfigManager.Configuration.InstanceID, RestrictedFeatures.NoCpuThrottling))


    {


        ResourceUtilities.ThrottleCpu();


    }


}


 


IsFeatureEnabled()


{


   case RestrictedFeatures.NoCpuThrottling:


            return SkuUtil.IsEnterpriseDevOrEval(sku);


 


}


 


public static bool IsEnterpriseDevOrEval(SkuType sku)


{


    if ((sku != SkuType.Developer) && (sku != SkuType.Enterprise))


    {


        return (sku == SkuType.Evaluation);


    }


    return true;


}


 


Here comes the cause, customer is using a standard version of SQL. And it support up to 4 CPUs only. Here is the full link:


http://www.microsoft.com/sqlserver/2005/en/us/compare-features.aspx


 


Scalability and Performance


















Feature


Express


Workgroup


Standard


Enterprise


Comments


Number of CPUs


1


2


4


Max OS supported


Includes support for multicore processors


 


Now, we are able to reproduce this problem easily on my 2 CPU system.


1.    Create two simple asp.net applications contains page like helloworld.aspx


2.    Create a test application pool and assign this application pool to these two applications.


3.    Access application 1.


4.    Open task manager and select “set affinity”, un-check one CPU.


5.    Access application 2.


6.    Now, back to IE, refresh application 1, it never load.


7.    Refresh application 2, it works.


8.    Now if we break into the process, we could see follow results. Application 1 set to 2 CPUs setting, and application 2 set to 1 CPU. ProcessMode(max worker threads) set to 1 CPU system.


*This is for application 1*


   RequestQueue 0x060438c0:


   _minExternFreeThreads: 176


   _minLocalFreeThreads: 152


 


*this is for application 2*


   RequestQueue 0x0612d874:


   _minExternFreeThreads: 88


   _minLocalFreeThreads: 76


 


0:029> !threadpool


Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 100 MinLimit: 1


Now, come to the last but most important section, what’s the solution? Based on the cause, definitely using a 4CPU machine can resolve the problem. Another solution is customizing HTTPRuntime for “Reporting Service”. Unfortunately, this is not working. The HTTPRuntime settings are reverted to default setting as long as the application started. It just ignores the customized settings in web.config. I don’t have an opportunity to find out why Reports Service overwrites the customized HTTPRuntime settings. But, at last, we resolved the problem by separate them into different application pools.


 


See you next time.


Best Regards,


Zhao Wei

Comments (1)

  1. Alik Levin says:

    Very informative!

    Thanks for putting this all together.

    I am having similar issue with a customer and I am going to test your solution right away

Skip to main content