Process Affinity Again


Three months ago, I have a blog talking about worker process affinity. Now, we have a similar issue again, but it is more complex this time.

My customer has SQL reporting service installed on an 8 CPU windows 2008 system. And he experienced severe performance issue as soon as the application started. IE keep show blank page for long time when he tried to generate a report, no matter the report is small or huge.

As you may or may not know, reporting service installs two web applications on IIS, one is “Reports” and another is “ReportServer”. Usually, “Reports” call the web service provided by “ReportServer”. Of course, you can call the web service provided by “ReportServer” directly. And, by default these two applications are hosted in same worker process.

Same, we collected dumps to analysis. From dump, we could see only one “Reports” page was running and this page was calling a web service under “Reportserver” which hosted in the same worker process. Then why we could not see the request for ASMX web service? Looking at the dumps carefully, we found the request was in the application’s queue. Please notice that the HTTPRuntime minFreeThreads for “Reports” and “ReportServer” were different. “ReportSever” was configured for 8 CPUs and “Reports” was set for 4 CPUs.

0:031> !dumprequestqueues

   RequestQueue 0x000000015f766920:

   _minExternFreeThreads: 704

   _minLocalFreeThreads: 608

   _count: 7

   _workItemCount: 0

   _localQueue 0x000000015f766978 size: 3

   _externQueue 0x000000015f766ad0 size: 4


   RequestQueue 0x000000019f7ac0c0:

   _minExternFreeThreads: 352

   _minLocalFreeThreads: 304

   _workItemCount: 0

   _localQueue 0x000000019f7ac118 size: 0

   _externQueue 0x000000019f7ac270 size: 0


Max worker threads are 4 CPU settings. The default settings is 100*#CPU.

0:031> !threadpool

Worker Thread: Total: 8 Running: 1 Idle: 7 MaxLimit: 400 MinLimit: 4

Completion Port Thread:Total: 1 Free: 1 MaxFree: 16 CurrentLimit: 0 MaxLimit: 400 MinLimit: 4


By default, MinFreeThreads is set to 88*#CPU, and MinLocalFreeThread is set to 76*#CPU.

Here is the MSDN suggested settings and its details description. As you can see, customer’s setting is exactly what we suggested.

Configuration Setting

Recommended Value


12 * #CPUs






88 * #CPUs


76 * #CPUs


If you are not familiar with this, please refer to follow KB.

821268  Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications;EN-US;821268

Chapter 10 — Improving Web Services Performance


As I said, this is an 8 CPU system.

0:031> vertarget

Windows Server 2008/Windows Vista SP1 Version 6001 (Service Pack 1) MP (8 procs) Free x64


Then why this setting blocked “ReportServer” accept new requests? Same again, we got the answer via check code in reflector. The incorrect _minExternFreeThreads and _minLocalFreeThreads stop ASP.Net to dequeue a request as the dequeue condition can never be satisfied.

            if (freeThreads >= _minExternFreeThreads) {

                wr = DequeueRequest(false); // enough threads to process even external requests


            else if (freeThreads >= _minLocalFreeThreads) {

                wr = DequeueRequest(true);  // enough threads to process only local requests


            int workerThreads, ioThreads;

            ThreadPool.GetAvailableThreads(out workerThreads, out ioThreads);


            // not enough worker threads to do anything

            if (workerThreads < _minLocalFreeThreads)



            // pick up request from the queue

            HttpWorkerRequest wr = DequeueRequest(workerThreads < _minExternFreeThreads);


My first think is customer uses customized configuration for ProcessModel, and used different HTTPRumtime configuration for “Reports” and “ReportServer”. However, this is not true. By review customer’s configuration file, we found that customer is using auto configure. Here is the “auto” setting for an 8 CPUs system.

MaxWorkerThreads 100, total = 8*100

MinFreeThreads = 88*8 = 704

MinLocalFreeThreads = 76*8 = 608.


Then, why ProcessMode and HTTPRuntime for “Reports” set to 4 CPU values? Look into the dump again, and we found a crazing issue. The CPU counts for “Reports” was 4, and for “ReportServer was 8.

0:031> !do 0x000000019f790380

Name: System.Web.Configuration.ProcessModelSection

MethodTable: 00000642bcf33970

EEClass: 00000642bcf55a50

Size: 120(0x78) bytes

00000642788ca730  4001946      998         System.Int32  1   shared           static cpuCount

                                 >> Domain:Value  0000000001bb2c50:NotInit  0000000001c2f740:8 0000000008221f30:4 <<


0:031> !dumpdomain -stat

    Domain        Num Assemblies        Size Assemblies         Name

000006427fc82130                  0                       0        System Domain

000006427fc82ac0                 18            119,934,976            Shared Domain

0000000001bb2c50                10             72,175,616             DefaultDomain

0000000001c2f740                240           161,005,056            /LM/W3SVC/1/ROOT/ReportServer-1-128885691260563502

0000000008221f30                 73            122,132,992            /LM/W3SVC/1/ROOT/Reports-2-128885692243612024


And current process affinity is 0xf, which means only 4 CPUs were available to this process.

0:000> dt ntdll!_PEB 000007fffffdf000

   +0x0b8 NumberOfProcessors : 8

   +0x138 ActiveProcessAffinityMask : 0xf


This is crazy, right? So, the only reasonable explanation is affinity. A simple test shows that the process affinity of worker process which hosted Reporting Service changed to 4 CPU as long as it is started. However, all other worker processes on this server didn’t experience this issue. We are suspecting “ReportServer” changes the worker process’s affinity itself. Then, we found out the problem by:

1.    Created a test web application which contains simple pages only

2.    Assign Reporting Service’s application pool to the test application.

3.    Start the Reporting Services application pool by access the test application.

4.    Attach a debugger to the worker process, and set breakpoint at Kernel32!SetProcessAffinityMask.

5.    Access “ReportServer”, and then the breakpoint was hit.

Here is the call stack:

0:028> !clrstack

OS Thread Id: 0x61c (28)

Child-SP         RetAddr          Call Site

000000000572e5b0 0000064280713f9d DomainBoundILStubClass.IL_STUB(Void*, UInt64)

000000000572e680 0000064280713a48 Microsoft.ReportingServices.Utilities.ProcessUtilities.SetProcessAffinity(UInt64)

000000000572e6c0 000006428070791a Microsoft.ReportingServices.Diagnostics.ResourceUtilities.ThrottleCpu()

000000000572e730 0000064280706c3b Microsoft.ReportingServices.WebServer.Global.RunOnlyOnceStartReportServer()

000000000572e7a0 00000642807069af Microsoft.ReportingServices.WebServer.Global.StartApp()

000000000572e800 00000642bcb24d10 Microsoft.ReportingServices.WebServer.Global.Application_BeginRequest(System.Object, System.EventArgs)


Here is the code via Reflector. From the code, we can see that “ReportServer” changes the affinity if it is not Dev version, ENT version and EVAL version.

public static void InitializeProcessParameters()


    if (!Sku.IsFeatureEnabled(ConfigManager.Configuration.InstanceID, RestrictedFeatures.NoCpuThrottling))








   case RestrictedFeatures.NoCpuThrottling:

            return SkuUtil.IsEnterpriseDevOrEval(sku);




public static bool IsEnterpriseDevOrEval(SkuType sku)


    if ((sku != SkuType.Developer) && (sku != SkuType.Enterprise))


        return (sku == SkuType.Evaluation);


    return true;



Here comes the cause, customer is using a standard version of SQL. And it support up to 4 CPUs only. Here is the full link:


Scalability and Performance







Number of CPUs




Max OS supported

Includes support for multicore processors


Now, we are able to reproduce this problem easily on my 2 CPU system.

1.    Create two simple applications contains page like helloworld.aspx

2.    Create a test application pool and assign this application pool to these two applications.

3.    Access application 1.

4.    Open task manager and select “set affinity”, un-check one CPU.

5.    Access application 2.

6.    Now, back to IE, refresh application 1, it never load.

7.    Refresh application 2, it works.

8.    Now if we break into the process, we could see follow results. Application 1 set to 2 CPUs setting, and application 2 set to 1 CPU. ProcessMode(max worker threads) set to 1 CPU system.

*This is for application 1*

   RequestQueue 0x060438c0:

   _minExternFreeThreads: 176

   _minLocalFreeThreads: 152


*this is for application 2*

   RequestQueue 0x0612d874:

   _minExternFreeThreads: 88

   _minLocalFreeThreads: 76


0:029> !threadpool

Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 100 MinLimit: 1

Now, come to the last but most important section, what’s the solution? Based on the cause, definitely using a 4CPU machine can resolve the problem. Another solution is customizing HTTPRuntime for “Reporting Service”. Unfortunately, this is not working. The HTTPRuntime settings are reverted to default setting as long as the application started. It just ignores the customized settings in web.config. I don’t have an opportunity to find out why Reports Service overwrites the customized HTTPRuntime settings. But, at last, we resolved the problem by separate them into different application pools.


See you next time.

Best Regards,

Zhao Wei

Comments (1)

  1. Alik Levin says:

    Very informative!

    Thanks for putting this all together.

    I am having similar issue with a customer and I am going to test your solution right away

Skip to main content