Troubleshooting "Connections_refused" errors seen HTTPerr logs

According to KB820729, Connections_Refused means that the "Kernel Non Paged Pool memory has dropped below 20MB and HTTP.SYS has stopped receiving new connections". 

HTTP.SYS is basically telling us "someone is using up a lot of NPP memory, and for protective reasons, I am going to stop accepting requests". We need to figure out what driver is using up all the NPP memory and address it, and the Connections_Refused should naturally go away because NPP memory will not be under pressure.

Here’s a table summarizing the nonpaged pool limits across different version of Windows: 

Tracking Pool Leaks

Poolmon is the tool that helps us in troubleshoot kernel NPP issues. 

C:\windows\system32>poolmon.exe -b 

We see that top most entry in the list is consuming most of the memory. Tag for the driver is “Leak”. After identifying the guilty tag in the left column, in this case ‘Leak’, the next
step is finding the driver that’s using it. Since the tags are stored in the driver image, you can do that by scanning driver images for the tag in question.

The Strings utility from Sysinternals dumps printable strings in the files you specify and since most device driver images are in the %Systemroot%\System32\Drivers directory, you can open a command prompt, change to that directory and execute “strings * | findstr <tag> ”.

C:\windows\system32\drivers> strings * | findstr Leak

After you’ve found a match, you can dump the driver’s version information with the Sysinternals Sigcheck utility.

C:\windows\system32\drivers>sigcheck myfault.sys 

Note: The driver causing the issue here was a tool written to reproduce the leak scenario and we have rightly identified that. In real life scenarios, we will find the actual driver causing the NPP leak.

References:  

Most of the information in this post is derived and compiled from “Pushing the Limits of Windows: Paged and Nonpaged Pool” by Mark Russinovich

https://blogs.technet.com/b/markrussinovich/archive/2009/03/26/3211216.aspx   

and “HOWTO:Diagnose IIS6 failing to accept connections due to Connections_Refused” by David Wang

https://blogs.msdn.com/b/david.wang/archive/2005/09/21/howto-diagnose-iis6-failing-to-accept-connections-due-to-connections-refused.aspx

How to find pool tags that are used by third-party drivers

https://support.microsoft.com/?id=298102

Error logging in HTTP API

https://support.microsoft.com/?id=820729

Downloads:  

“Poolmon” shipped along with Windows Support tools

https://www.microsoft.com/downloads/en/details.aspx?FamilyID=96a35011-fd83-419d-939b-9a772ea2df90&DisplayLang=en

“Strings” utility from sysinternals

https://technet.microsoft.com/en-us/sysinternals/bb897439.aspx

“Sigcheck” utility from sysinternals

https://technet.microsoft.com/en-us/sysinternals/bb897441.aspx