Fix for Slow IO Thread Startup

  • Article

The CLR team has recently released a fix for slow startup of asynchronous IO threads that may affect WCF users. After startup, asynchronous IO threads are created at a slow pace to avoid spikes in the number of threads. Having extra idle threads consume system resources needlessly. Having too few available threads creates a performance bottleneck as requests take longer to process and there is a delay in spinning up new threads.

Calling ThreadPool.SetMinThreads is supposed to preserve a minimum number of worker threads (the second parameter represents the number of idle asynchronous IO threads that will be maintained). However, the problem is that the application may still wait while the CLR tries to create more threads.

The published fix doesn't provide a diagnosis for WCF users but here's what I've come up with.

Symptoms to diagnose the problem are if you see all of the following:

- Server throughput scales up slowly during a spike in simultaneous requests

There is idle CPU time
  • Tracing the thread context switches shows significant lag in starting a completion port thread
  • You've called SetMinThreads with a sufficient number of threads to handle all of the requests
  • SetMinThreads returned true (made the requested change)

This issue is probably not the problem if any of the following are true:

- Only the first call on a connection is slow

Server throughput does not eventually scale up to match the request load
  • The CPU use is 100%
  • Thread traces shows the worker threads blocked on anything besides startup
  • The minimum number of asynchronous IO threads is too small to handle the request load
  • SetMinThreads returned false (failed to make the requested change)

You can get the fix from KB article 976898.