Randomly, IE will show follow error when customer accessing their web application. Meantime, ASP.NET event 1310 will recorded as well. According to KB 821268 and MSDN, this is the recommended setting for a two CPU system (Dual core is considered as 2 CPU).
Here is what we see in the browser:
Parser Error Message: The value for 'minFreeThreads' must be less than the thread pool limit of 100 threads.
Line 136: </system.data>
Line 137: <system.web>
Line 138: <httpRuntime minFreeThreads="176" minLocalRequestFreeThreads="152" />
Line 139: <processModel maxWorkerThreads="100" maxIoThreads="100" minWorkerThreads="50" />
Source File: C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Config\machine.config Line: 138
Also, we could see 1310 event in the application event log.
Event Type: Warning
Event Source: ASP.NET 2.0.50727.0
Event Category: Web Event
Event ID: 1310
Time: 下午 02:54:23
Event code: 3008
Event message: A configuration error has occurred.
Exception type: ConfigurationErrorsException
Exception message: The value for 'minFreeThreads' must be less than the thread pool limit of 100 threads. (C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Config\machine.config line 138)
Thread ID: 7
Thread account name: NT AUTHORITY\NETWORK SERVICE
Is impersonating: False
Stack trace: at System.Web.HttpRuntime.FirstRequestInit(HttpContext context)
at System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context)
at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
Here is the MSDN suggested settings and its details description. As you can see, customer’s setting is exactly what we suggested.
12 * #CPUs
88 * #CPUs
76 * #CPUs
Then why this error happens? Based on MSDN, the configuration item in processModel will be automatically multiplied by the number of available CPUs, and MaxWorkerThreads must be equal or greater than minFreeThreads in httpRuntime. For customer’s setting, threads available to the worker process to service request is 100*2-176=24, which are 12 threads per CPU.
Now, come back to the original question, why this error happens? The only factor can affect this is process affinity. Now, we are able to reproduce this problem easily.
1. Create a web application contains hello.aspx
2. Open IE and visit hello.aspx, the page loaded correctly.
3. Open task manager, right click the w3wp.exe and select “set affinity”. Change the worker process’s affinity by un-check one CPU.
4. Open the machine.confog or web.config, make any change and then un-do this change. Save the config file.
5. Open IE and visit hello.aspx again, IE shows the errors and able to find 1310 in event log as well.
Here comes to the new question, who changed my worker process’s affinity? There will be 3 scenarios which may change worker process’s affinity.
1. Other application changed IIS worker process’s affinity by call SetProcessAffinityMask, example Task Manager.
2. User changed the worker process’s affinity by adsutil.vbs. This is done through W3SVC and finally, called SetProcessAffinityMask.
3. The code running in the worker process itself changed the affinity by call SetProcessAffinityMask.
For scenario 1, I don’t know how to find out the culprit application. My Windows colleague send me a checked build of Kernel32.dll, with this build, any process call SetProcessAffinityMask will be crashed at once. I don’t have a chance to test this yet.
I finally find out the problem by attach debugger to worker process and W3SVC service and set breakpoint at SetProcessAffinityMask. Here is the call stack in the worker process’s debugger’s log. A 3rd party DLL used by customer’s web application is calling SetProcessAffinityMask!
# ChildEBP RetAddr
kernel32!SetProcessAffinityMask(void * hProcess = 0xffffffff, unsigned long dwProcessAffinityMask = 1)
WARNING: Stack unwind information not available. Following frames may be wrong.