W3SVC 1310 and who changed my worker process affinity

Randomly, IE will show follow error when customer accessing their web application. Meantime, ASP.NET event 1310 will recorded as well. According to KB 821268 and MSDN, this is the recommended setting for a two CPU system (Dual core is considered as 2 CPU).

Here is what we see in the browser:

Parser Error Message: The value for 'minFreeThreads' must be less than the thread pool limit of 100 threads.

Source Error:

Source File: C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Config\machine.config Line: 138

Also, we could see 1310 event in the application event log.

Event Type: Warning

Event Source: ASP.NET 2.0.50727.0

Event Category: Web Event

Event ID: 1310

Date: 2009/3/27

Time: 下午 02:54:23

User: N/A

Computer: ZW

Description:

Event code: 3008

Event message: A configuration error has occurred.

…..

…..

Exception information:

    Exception type: ConfigurationErrorsException

    Exception message: The value for 'minFreeThreads' must be less than the thread pool limit of 100 threads. (C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Config\machine.config line 138)

Thread information:

    Thread ID: 7

    Thread account name: NT AUTHORITY\NETWORK SERVICE

    Is impersonating: False

    Stack trace: at System.Web.HttpRuntime.FirstRequestInit(HttpContext context)

   at System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context)

   at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)

Here is the MSDN suggested settings and its details description. As you can see, customer’s setting is exactly what we suggested.

Now, come back to the original question, why this error happens? The only factor can affect this is process affinity. Now, we are able to reproduce this problem easily.

1. Create a web application contains hello.aspx

2. Open IE and visit hello.aspx, the page loaded correctly.

3. Open task manager, right click the w3wp.exe and select “set affinity”. Change the worker process’s affinity by un-check one CPU.

4. Open the machine.confog or web.config, make any change and then un-do this change. Save the config file.

5. Open IE and visit hello.aspx again, IE shows the errors and able to find 1310 in event log as well.

Here comes to the new question, who changed my worker process’s affinity? There will be 3 scenarios which may change worker process’s affinity.

1. Other application changed IIS worker process’s affinity by call SetProcessAffinityMask, example Task Manager.

2. User changed the worker process’s affinity by adsutil.vbs. This is done through W3SVC and finally, called SetProcessAffinityMask.

3. The code running in the worker process itself changed the affinity by call SetProcessAffinityMask.

For scenario 1, I don’t know how to find out the culprit application. My Windows colleague send me a checked build of Kernel32.dll, with this build, any process call SetProcessAffinityMask will be crashed at once. I don’t have a chance to test this yet.

I finally find out the problem by attach debugger to worker process and W3SVC service and set breakpoint at SetProcessAffinityMask. Here is the call stack in the worker process’s debugger’s log. A 3^rd party DLL used by customer’s web application is calling SetProcessAffinityMask!

# ChildEBP RetAddr 

kernel32!SetProcessAffinityMask(void * hProcess = 0xffffffff, unsigned long dwProcessAffinityMask = 1)

WARNING: Stack unwind information not available. Following frames may be wrong.

AStrangeComponent!DllCanUnloadNow+0x50a76

AStrangeComponent!DllCanUnloadNow+0x10f9a5

…………..

Regards,

Zhao Wei