HOWTO: Diagnose IIS6 failing to accept connections due to Connections_Refused


Recently, I have seen a bunch of questions asking: “IIS6 runs fine for X amount of time [where X varies from days to weeks] and then all of a sudden, it stops accepting all connections. If I restart/reboot the server, then it is fine again… until X amount of time later, when I have to repeat the same procedure.”


Sometimes, the user even notices that HTTP Error logs in %SYSTEMROOT%\System32\LogFiles\HTTPERR\*.log mention Connections_Refused for all those requests, but what the heck does that mean?


Ok, ignoring the jokers in the back that are snickering “hmm, this is Microsoft software; what do you expect?” , this issue is actually very diagnosable and should be treatable, as I will shortly show…


Question:


Every four or five days the web site hoste on my machine will stop responding to HTTP requests on port 80.


IIS doesn’t crash so I cannot get an IIS State log, just this:


Excerpt from the HTTPErr.Log:


(… working fine up until this …)
2005-09-02 18:45:12 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:46:07 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:46:27 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:46:42 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:47:02 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:48:22 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:48:57 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:49:02 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:49:42 – – – – – – – – – 7_Connections_Refused –
2005-09-02 18:50:02 – – – – – – – – – 3_Connections_Refused –
2005-09-02 18:50:37 – – – – – – – – – 6_Connections_Refused –
2005-09-02 18:50:42 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:51:02 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:51:47 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:52:02 – – – – – – – – – 3_Connections_Refused –
2005-09-02 18:52:27 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:53:17 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:53:22 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:54:02 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:54:22 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:54:32 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:55:22 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:56:02 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:56:42 – – – – – – – – – 1_Connections_Refused –
2005-09-02 18:58:17 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:58:42 – – – – – – – – – 2_Connections_Refused –
2005-09-02 18:58:57 – – – – – – – – – 1_Connections_Refused –


I have to reboot the machine to get it to work again.


Does anyone have any ideas?


Answer:


Connections_Refused is actually a very diagnosable failure pattern, and I am going to show you how to diagnose it and what is going on.


What does Connections_Refused Mean?


According to KB 820729, Connections_Refused means that the “Kernel Non Paged Pool memory has dropped below 20MB and HTTP.SYS has stopped receiving new connections”. What all this means in English is this:



  • On Windows NT systems, there are many types of memory with different properties. Kernel Non Paged Pool (NPP) memory is one such type, and the important thing to know is that on x86 machines, the size of this memory is fixed and has a maximum size of 128MB. In other words, you may have an x86 machine with 4GB of RAM, but if you only use 128MB and that 128MB all comes from NPP memory, your system is still “out of memory” regardless of how much is actually in use.
    [1/4/2006 – Correction – maximum size on x86 is “around 250MB” and 128MB with /3GB. For Windows Vista, it will be bounded by physical RAM and not arbitrary limit.] 
  • HTTP.SYS, being a kernel-mode driver, uses NPP memory for every single connection that is active, and we made a conscious decision to have it stop accepting connections when NPP memory falls below a threshold, arbitrarily set at 20MB. This is reasonable because lots of bad things can occur if NPP memory is depleted.

    For example: HTTP.SYS is likely the most active codebase in a web server, and if NPP memory gets depleted and HTTP.SYS needs NPP memory to accept a connection, the machine will blue screen. Now, regardless of the driver leaking NPP memory, HTTP.SYS looks like the culprit because it is most active. In any case, all of this is not good, so HTTP.SYS proactively refuses to get near the edge and plays it safe.

Hopefully, it is now clear what Connections_Refused means. HTTP.SYS is basically telling you “someone is using up a lot of NPP memory, and for protective reasons, I am going to stop accepting requests”. We need to figure out what driver is using up all the NPP memory and address it, and the Connections_Refused should naturally go away because NPP memory will not be under pressure.


How to Diagnose this Issue


One of the nice properties of NPP memory is that a unique tag is associated with every piece of this memory, so we can always track down who is using what and how much of NPP memory. The user-mode tool to do this is called POOLMON.EXE (if you have a Kernel Debugger attached, you can use the !poolused  command to get this info within the Kernel Debugger). Here are some additional bits of related information:



I am now going to step through the POOLMON output of a user who had Connections_Refused and comment on what is going on. Here is a sample snippet of running POOLMON.EXE -b :

 Memory: 1048016K Avail:  438396K  PageFlts:   219   InRam Krnl: 3604K P:42372K
Commit: 538908K Limit:11245092K Peak: 609576K Pool N:109056K P:43740K
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc

ISil Nonp 1408366 ( 56) 1327427 ( 58) 80939 36678632 ( -872) 453
I100 Nonp 11048877 ( 217) 10967968 ( 219) 80909 14886928 ( -368) 183
PcNw Nonp 292427 ( 0) 88967 ( 0) 203460 12203296 ( 0) 59
SavE Nonp 1596066 ( 0) 1595659 ( 0) 407 11717016 ( 0) 28788
IAM Nonp 1946164 ( 49) 1904725 ( 50) 41439 8519576 ( -208) 205
tdLL Nonp 74748 ( 0) 69830 ( 4) 4918 2672832 ( -1776) 543
LSwi Nonp 1 ( 0) 0 ( 0) 1 2576384 ( 0) 2576384
R200 Nonp 22 ( 0) 1 ( 0) 21 2297816 ( 0) 109419
MmCm Nonp 302 ( 0) 48 ( 0) 254 2236960 ( 0) 8806
TCPt Nonp 190486 ( 4) 190462 ( 4) 24 1392264 ( 0) 58011
ULHP Nonp 3127 ( 0) 41 ( 0) 3086 1047216 ( 0) 339
PTrk Nonp 999750 ( 0) 996561 ( 0) 3189 956824 ( 0) 300
File Nonp 9628160 ( 65) 9623252 ( 65) 4908 750688 ( 0) 152
rg81 Nonp 6910 ( 0) 5294 ( 0) 1616 603864 ( 0) 373
Pool Nonp 6 ( 0) 3 ( 0) 3 602112 ( 0) 200704
Devi Nonp 680 ( 0) 262 ( 0) 418 558552 ( 0) 1336
Thre Nonp 417221 ( 6) 416457 ( 6) 764 476736 ( 0) 624
LSwr Nonp 128 ( 0) 0 ( 0) 128 416768 ( 0) 3256
Mm Nonp 26 ( 0) 2 ( 0) 24 379880 ( 0) 15828
AfdC Nonp 21989 ( 0) 19872 ( 1) 2117 338720 ( -160) 160
….


Some observations:



  • Notice that this system has 1GB RAM but still only 128MB NPP memory (109MB has been used, hence tripping the 20MB marker of HTTP.SYS).
  • The NPP memory tags of HTTP.SYS all start with “UL” (for the trivia-inclined: as with all software projects, HTTP.SYS has gone through some evolution of its own. It used to be called something else – Universal Listener – hence its tag names all started with “UL”).
  • Notice that HTTP.SYS is not even in the top-10 NPP memory usage on the system, and its largest set of allocations is barely using 1MB total. As it turns out on this system, TOTAL NPP memory usage by HTTP.SYS was under 2MB, so clearly, it is not the aggressive user of NPP memory that is now causing issues.
  • Meanwhile, focus your attention on the top four users of NPP memory, with tags “ISil”, “I100”, “PcNw”, and “SavE”. I looked up drivers for those tags, and I only find that “PcNw” comes from Microsoft and is associated with WDM audio. The others are all 3rd party drivers.
  • Umm, this is a server, so is audio really that important? I checked on my system, and it was at 275KB, so clearly this user is/has done some heavy-duty audio stuff on this server. Thus, the memory may be legitimately used. You know, you might want to reconsider the effects of playing your MP3s on your Web Server’s ability to accept connections…
  • Meanwhile, Isil and I100 have suspiciously similar active allocations and ISIL is the stock ticker for a semiconductor company, so my guess is that I100 is related to a Fast Ethernet card… and since web servers totally rely on the network driver, having a memory-hog for a network driver cannot be a good thing and should be fixed. My guess for SavE is probably anti-virus.
  • Now, I may be totally wrong with those arbitrary tag names, but one thing is for certain – IIS6 and HTTP.SYS is running just fine on this system. There are some other memory-hungry drivers running on the system, and if you get fixes for them such that your system will stay running longer, IIS6 will likely keep running as well

Note: Don’t shoot the messenger (HTTP.SYS)! Always strive to figure out the root of your problem and address that. The rest of the problems should naturally fall away.


//David

Comments (108)

  1. Armando Betancourt says:

    Great article. Also you can suggest the user to use Poolmon -c (2003 DDK) to create a "localtag.txt" (if not present) to find out the driver.

    Cheers,

    Armando

  2. David Wang says:

    Armando – Thanks. POOLMON -c is a good tip to map back to the appropriate driver name, which can hopefully provide some more contextual info…

    //David

  3. Hi Dave,

    I am having a very similiar problem with IIS 5 running .Net 1.1 webservices on Win2k. It is refusing connections in much the same manner as you describe. I am not getting any log information at all. The information I have been able to collect indicates to the client calling the web service that the connection has been refused. The behavior is sporadic, but seems to follow a pattern of worsening over time.

    Does anything you have written thus far about IIS 6 apply? What can I do to troubleshoot this problem on IIS 5 since it doesn’t seem to have an HTTPERR file? Any help at all would be vastly appreciated.

    I’m not an IIS administrator, but a software architect attempting to rule out the software as the culprit.

    Thanks,

    David Campbell

  4. David Wang says:

    David – Unfortunately, since IIS5 and IIS6 are completely different architecturally, what I have said above only applies to IIS6 and has no applicability to IIS5.

    We completely rewrote IIS6 and then worked on behavior-compatibility with IIS5, so you are mostly looking at different issues.

    I would contact Microsoft PSS for paid incident support. Of all the times I have seen the similar sort of issues on IIS5, it was usually something wrong with software on IIS5, including Exchange.

    And if it is a bug in MS software under support, you will not be charged and should expect to either get a fix (if it is known) or get an analysis of what is wrong and possibly add a fix-request into the service pack queue.

    //David

  5. Amy Ashby says:

    Thanks for this entry in your blog! I was able to find the application at fault — Executive Software Undelete (tag is XEFr where X equals 1, or 2, or 3, etc). Once the service was stopped, which was number 1 on the list of tags shown, I was then able to hit pages in the website.

  6. David Wang says:

    Amy – glad to hear that you were able to get to the bottom of things.

    You see, it is very easy to run "bad" software and make it look like something else is at fault. Only by figuring the issue out do you get a good sense of what is good or not.

    There is a long history of IIS being the focal point of such unwarranted blame (not to say that IIS is completely blemish free, but it certainly picks up a lot of excessive slack). I am just doing my little part in helping people figure out what is really going on so that they can make their own decisions.

    //David

  7. We have a similar issue on Windows 2K as David Campbell. 120 website hosting Asp and ASP.NET. After some period of time we get: The server was unable to allocate from the system nonpaged pool because the pool was empty.

  8. We have a similar issue on Windows 2K as David Campbell. 120 website hosting Asp and ASP.NET. After some period of time we get: The server was unable to allocate from the system nonpaged pool because the pool was empty.

  9. David Wang says:

    Ryan – You can use GFLAGS to enable "Pool Tagging" and then use POOLMON to see what is using up your NonPaged Pool memory and go from there. Since IIS5 and IIS6 are different architecturally, I can only start from "what is using up NonPagedPool Memory" and work backwards from there. Unlikely for IIS5 to be directly involved since there is no HTTP.SYS.

    I believe GFLAGS is in the Microsoft Debugging Toolkit ( http://www.microsoft.com/whdc/devtools/debugging/default.mspx ), while POOLMON is in the Windows Resource Kit, both freely available.

    //David

  10. Thanks David. I am doing that. Here is the top of my detail:

    Memory: 3997212K Avail: 2716404K PageFlts: 1248 InRam Krnl: 2480K P:270528K

    Commit:1097260K Limit:5900908K Peak:1294344K Pool N:102460K P:270732

    Tag Type Allocs Frees Diff Bytes Per Alloc

    File Nonp 87044825 ( 768) 86990579 ( 774) 54246 8786880 ( -992) 161

    MmCm Nonp 1559 ( 0) 7 ( 0) 1552 7336512 ( 0) 4727

    MmCa Nonp 991170 ( 1) 940356 ( 5) 50814 6440160 ( -512) 126

    Ntfr Nonp 730966 ( 0) 632909 ( 0) 98057 6276640 ( 0) 64

    tdLL Nonp 361780 ( 0) 351199 ( 27) 10581 6076736 (-11520) 574

    NtFs Nonp 799244 ( 3) 747632 ( 6) 51612 3305600 ( -192) 64

    Ntfn Nonp 440263 ( 0) 388659 ( 2) 51604 3304576 ( -128) 64

    LSwi Nonp 1 ( 0) 0 ( 0) 1 2293760 ( 0) 2293760

    PTrk Nonp 164833 ( 0) 161574 ( 0) 3259 1051520 ( 0) 322

    It does not look like there is a major offender. Just we have a lot going on. I think I need to change the size of NonPagedPoolSize. This link here explains how to do it: http://www.microsoft.com/technet/prodtechnol/windowsserver2003/library/DepKit/fe862e6b-14fb-4ad1-bd76-7e409c37060d.mspx but does not tell me much detail. Any idea what happend to http://support.microsoft.com/?kbid=126402 i have foudn many references to it. But the article seems to be gone.

  11. David Summers says:

    Ryan

    We are running IIS 5 on four Web server at my organization, and are having the same issue with degraded performance over time on two of them.

    We also get an event logged that the NPR pool being empty before the server stops completely and must be rebooted.

    I have been monitoring nonpageable ram in both the PROCESS and MEMORY counters, and have seen that the process counters are stable while the MEMORY (total NPR) increases steadily over time. This tells me that a regular process does not have a memory leak.

    I have been watching poolmon for a few days and have noticed one tag in particular appears to be growing and not releasing memory, and this tag happens to be on your list as well: tdLL.

    I believe this belongs to Symantec (the string is found in three of their sys files). Are you running Symantec A/V? We are running Corporate version 9.0.0.338.

    Anyway, watch tdLL in poolmon over a few days and see if it gets bigger and bigger.

    I’ll post here if I find anything new.

    David Summers

  12. David Summers,

    Yes, tdLL was at the top of my list and it was the only one that I had been unable to identify. It has grown over time from 1,405,440 bytes initially (Saturday)to 10,555,008 bytes currently (Tuesday). Yes we are running Symantic AV Corporate version 9.0.1.1000.

    Ryan M. Hager

  13. OK, one week later. tdLL is 70,586,816.

    Yes, whatever it is, that is my culprit. FYI, we are using a Compaq server. And have diskeeper on top of NAV.

    Ryan.

  14. Mike Ayling says:

    Great article David. One minor thing though, according to Russinovich & Solomon’s "Windows Internals", the maximum nonpaged pool size on a 32-bit system is 256mb, although 128mb if booted with the /3gb switch enabled in the boot.ini.

    Here is an easy way to determine max npp size on a Server 2003 system:

    1. Download and install the debugging tools from:

    http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

    2. Download Process Explorer from:

    http://www.sysinternals.com/Utilities/ProcessExplorer.html

    3. Open Process Explorer and select "Options" then "Configure Symbols…"

    4. Put the path to the "dbghelp.dll" included with the debugging tools you downloaded.

    5. Use the following for the symbols path ("c:symbols" can be whatever…):

    srv*c:symbols*http://msdl.microsoft.com/download/symbols

    6. Select "View" then "System Information". You can now see the "Nonpaged Limit".

    -Mike

  15. David Wang says:

    Mike – Thanks for the info.

    I got that number from the HTTP.SYS folks who swear by those numbers (including the dev that actually implemented the check that results in Connection_Refused), so I’ll report this bit of info to them.

    //David

  16. Just a confirmation on what Mike was saying. I was just diagnosing the connection refused error today and came across your blog (very helpful by the way). I currently have a 206MB Nonpaged kernel limit with 189MB usage causing the problem.

    Thanks for the information, it was very insightful.

    -Rich

  17. Well, After about a month we hit the limit again. Here is what we had this morning:

    ~8:00 am

    Memory: 3997212K Avail: 1619692K PageFlts: 621 InRam Krnl: 2668K P:134468K

    Commit:2298928K Limit:5569132K Peak:2566476K Pool N:252224K P:134656

    Tag Type Allocs Frees Diff Bytes Per Alloc

    tdLL Nonp 5767444 ( 7) 5483265 ( 13) 284179 163683072 ( -320) 575

    File Nonp 1342225323 ( 194) 1342172019 ( 178) 53304 8559040 ( 2816) 160

    AfdC Nonp 31320130 ( 8) 31276564 ( 7) 43566 8364672 ( 192) 192

    And after an IISReset:

    9:20 am

    Memory: 3997212K Avail: 3042456K PageFlts: 1794 InRam Krnl: 2684K P:95548K

    Commit: 742308K Limit:5569132K Peak:2566476K Pool N:215448K P:95740K

    Tag Type Allocs Frees Diff Bytes Per Alloc

    tdLL Nonp 5781423 ( 4) 5581628 ( 0) 199795 115072064 ( 512) 575

    MmCm Nonp 1559 ( 0) 7 ( 0) 1552 7336512 ( 0) 4727

    I guess it is time to give support a call.

    – Ryan.

  18. David Summers says:

    Ryan

    I uninstalled Symantec 9 A/V and installed Symantec 10 last week. My NPR leak has gone away, and the tdLL flag no longer contiuously grows.

  19. Ryan M. Hager says:

    David S,

    Yes, we upgraded today and have a new Memory Tag SavE (My guess Symantic Antivirus). I think this is even better so we can see more clearly if AV is the issue.

    Memory: 3997212K Avail: 3208816K PageFlts: 986 InRam Krnl: 2504K P:64672K

    Commit: 620408K Limit:5569124K Peak: 621072K Pool N:46768K P:64884K

    Tag Type Allocs Frees Diff Bytes Per Alloc

    SavE Nonp 747930 ( 120) 747490 ( 120) 440 13866816 ( 0) 31515

    MmCm Nonp 1558 ( 0) 7 ( 0) 1551 7316032 ( 0) 4716

    File Nonp 858782 ( 320) 842389 ( 331) 16393 2863104 ( -1664) 174

    LSwi Nonp 1 ( 0) 0 ( 0) 1 2293760 ( 0) 2293760

    tdLL Nonp 21511 ( 10) 18486 ( 22) 3025 1727168 ( -1984) 570

    MmCa Nonp 54316 ( 26) 43039 ( 23) 11277 1437152 ( 384) 127

    Ntfr Nonp 22473 ( 6) 158 ( 0) 22315 1429152 ( 384) 64

    NDpp Nonp 54 ( 0) 24 ( 0) 30 981248 ( 0) 32708

    PTrk Nonp 11485 ( 0) 8550 ( 0) 2935 947680 ( 0) 322

    Ryan.

  20. Ben Ryan says:

    Spot on diagnosis… I had this problem recently on win2k3 w/IIS60 and found NPP usage was high. rxWriterSvc.exe was chewing 70MB NPP, which coupled with MSEX2k3 was causing http drops. This service is a component of Symantec Veritas’s Continuous Protection Server, CPS which is like MS Data Protection Manager. Thanks for your blogs – as usual, they are worth their weight in PSS incidents 🙂

  21. David.Wang says:

    Ben – Thanks. Glad to help avoid some PSS incidents. Believe me, MS does not want you to call support, either. 🙂 They cost everyone money.

    This particular issue and resolution is amazingly common, though the symptoms as reported by users can vary dramatically.

    //David

  22. David Wang says:

    I recently sat down and thought a little about the typical user experience when troubleshooting IIS6,…

  23. Mathias Legerer says:

    Hi,

    We have a similar problem with our IIS 6.0 running on W2k3-Server (plus with all patches and SPs installed).

    It seems, that my process with the pooltag "even" takes to much memory (175MB!!).

    from the pooltag.txt:

    Even – <unknown> – Event objects

    from poolmon.exe:

    Tag Type Allocs Frees Diff Bytes Per Alloc

    Even Nonp -208686552 ( 24) -212511357 ( 24) 3824805 183594784 ( 0) 48

    What is the Tag Even ? We have no Virus-Scanner running. But we changed the NW-Card and did update our Exchange-Server to SP2 just before the problems started.

    All other Applications run well.

    Perhaps you can help me.

    Regards,

    Mathias

  24. David.Wang says:

    Mathias – try POOLMON -c

    It should return a file named localtag.txt which associates the tag ID with a driver filename.

    Given a filename, it is now up to your ingenuity to figure out what program/hardware actually installed and uses that file.

    Good Luck,

    //David

  25. Mathias says:

    It says, I need some dlls to create local tag file. Looks like i need the ddk for Server 200, or?

    Costs me shipping. Is there a way to dl exactly these dlls, that i need?

  26. Rad says:

    Well,

        I ‘ve been troubleshooting this problem for a while and I’ve seen this on two servers I’ve setup.

    On both servers I was getting connection refused about every 6 days.

    One Server is Small Business Server 2003 Standard and the other is Small Business Server 2003 Premium.

    On the Standard I stopped the Symantec Veritas’s Continuous Protection Server and they have been up for 29 days with no problems.  I have SAVCE installed as well ver 10.0.2.200

    On the premium server I was running fine until I loaded Symantec Veritas’s Continuous Protection Server.  So I figured no problem I’ll stop the services associated with CPS like I did on the Standard.  Well after 6 days they went down again. So not sure on this one yet.  I have on theory but I will post a follow up after I verify it.  The one tag that I’m not sure of on this server is QAFC (I think it’s Quickbooks Enterprise 2006)  I also have SAVCE installed on this server version 10.0.2.210

    Memory: 4193360K Avail: 1582424K  PageFlts:   517   InRam Krnl: 4368K P:68992K

    Commit:2460984K Limit:10314256K Peak:2493512K       Pool N:51056K P:70308K

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

    MmCm Nonp  576 (0)  14 (0)    562   12955920 (0)  23053

    QaFc Nonp   5588 (0)    0 (0   45588    10211712 (0)    224

    LSwi Nonp   1 (0)    0 (0)     1    2740224 (0) 2740224

    File Nonp   22779779 (216)  22766589 (224)    13190 2011088 (-1216)    152

    VoSm Nonp   811 (0)   783 (0)    28    1679600 (0)  59985

    TCPt Nonp   90493 (6)  90463 (6)  30    1456496 (0)  48549

    RxTi Nonp   18076337 (132)  18070773 (135)  5564   1290848 (-696)    232

    Thre Nonp     818496 (22)   816639 (  26)     1857    1158768 (-2496)    624

    Rad

  27. Rad says:

    I haven’t solved it yet but I’m optimistic.  I found the QAFC is part of Qafilter.sys.   I found this article first http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B840141  but it only applys to Microsoft Windows Storage Server 2003.

    Then I came across this http://seer.support.veritas.com/docs/276469.htm

    I forgot that I had Veritas Storage Exec installed as well.  So hopefully this will fix my problem since  it fixes the memory leak in Qafilter.sys.   I will let y’all know.

    Rad

  28. David.Wang says:

    Mathias – I just performed a SIMPLE search for POOLMON on microsoft.com and came across several links providing download and other support information, such as the following:

    http://technet2.microsoft.com/windowsserver/en/library//0d302498-c947-4655-95af-719ae75acfb51033.mspx

    Interesting information include:

    – POOLMON -c does not work on 64bit editions of Windows Server 2003

    – the required DLLs for operation

    //David

  29. Rad says:

    Mathais – I ran into this same problem, unfortunately it does not provide a location to find them.  I end up installing the following so that I could use poolmon /c instead of the -c (same thing from what I understand).  After installing them all,  I copied the

    msdis130.dll  and pooltag.txt files from the resource kit folder to the support tools folder.  Then it worked.

    Debugging Tools

    http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

    Support Tools for 2K3 SP1

    http://www.petri.co.il/download_windows_2003_sp1_support_tools.htm

    Resource Kit for 2K3

    http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en

    I hope that this helps you,

    Rad

  30. Rad says:

    Mathais – I ran into this problem as well.  Unfortunately all the documentation on poolmon doesn’t tell you where to find the files.

    I had to install the following and then copy the msdis130.dll and pooltag.txt from the resource kit to the support tools folder.  Make sure you install the lastest support tools.    I used the poolmon /c switch instead of the -c although from what I was reading, they both do the same thing.

    Below are the links to what you need to install.

    Resource Kit for W2K3

    http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en

    Support Tools for W2K3 SP1

    http://www.petri.co.il/download_windows_2003_sp1_support_tools.htm

    Debugging Tools for W2K3

    http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

    Hope this helps,

    Rad

  31. Rad says:

    Try this one more time as previous two post didn’t go

    Mathais

    The short of it is that you need to install these three things

    Resource Kit for W2K3

    http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en

    Support Tools for W2K3 SP1

    http://www.petri.co.il/download_windows_2003_sp1_support_tools.htm

    Debugging Tools for W2K3

    http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

    I copied the msdis130.dll and pooltag.txt from the resource kit folder  to the support tools folder and was then able to run poolmon /c (From what I can tell it does the same as the -c, new documentation says to use /c)

    I did a search on Poolmon as well and couldn’t find those files easily.   Unfortunately none of the articles tell you where to get it.

    Hope this helps,

    Rad

  32. David.Wang says:

    Rad – Sorry – it seems like the blog comment filter decided that your comments were spam and unapproved them. Must be the presence of hyperlinks… I’ve just gone and approved them.

    FYI – It’s nothing against you or anyone else and I’m sure you mean well, but I personally do not recommend locating and installing arbitrary binaries from arbitrary locations. I continue to be amazed at the arbitrary binaries that a total stranger can influence others to download and run…

    //David

  33. Rad says:

    Thanks David,  The Petro.co.il links straight to microsoft, I just didn’t take the time to jot the link down but yes I agree with you about the arbitrary locations.

    Here is the actual page that it links to from Microsoft for the Support Tools for W2K3 SP1

    http://download.microsoft.com/download/3/e/4/3e438f5e-24ef-4637-abd1-981341d349c7/WindowsServer2003-KB892777-SupportTools-x86-ENU.exe

  34. Mathias says:

    Rad – thanks for the link and the explanation. Now i managed to build the localtag.txt but guess what: there is no tag that’s called "even" in my localtag.txt.

    i’m getting frustated. I ‘ll try to restart my server again (I think wednesday will it be). I deinstalled a Anti-Virus-Command-Center (provides only delegation of virus-products over the lan, but NO local scanning). Perhaps this program produces the troubles.

    I also contacted a guy, who had the same problems  ("even"-Tag depleting pool) on a W2k-Server in the year 2004. That’s what he wrote to me:

    "I never quite figured out what was specifically causing the problem, but I did get it to stop.

    I figured that the only thing that had changed on the server were updates from windowsupdate.microsoft.com.  I took off all the updates from the time of the problem and it stopped happening.  I think I replaced some printer drivers too.  The server ran great until a few weeks ago when we upgraded to Windows 2003.  I never did any more Windows updates.  

    Apparently there is no easy way to correlate the “even” tag to a specific driver or device.  Or at least people told me there was no good way to find out—I am not a programmer so there may be ways I do not know about."

    I guess, I’ll have to make a night shift…

  35. Rad Piver www.globalstrata.com says:

    Mathias,

      Sorry to hear that it didn’t show up.   I’ve only had this problem on W2K3 servers but as you could read from about mine wasn’t with the even tag.    If it didn’t appear in the localtag.txt then more than likely it is a third party driver.   I remember reading in one of the articles on poolmon to do a search on .sys to find the driver associated with the tag (Just found the link again http://support.microsoft.com/kb/298102/EN-US/ )

    After that do a search in google on the driver.

    Hope this help.

  36. Mathias says:

    Hi everybody.

    Thanks for your help with this Problem. It seems, as if the Bitdefender Management Console did the depleting of the memory pool.

    I restartet the server (after deinstalling it) and everything seems ok.

    The identification of the third-party-driver didn’t help in this case.

    Thank you for your suggestions and help!

    regards,

    Mathias

  37. Courtney Kibbe says:

    I need help with this too.  Is it just the Nonp entries that can cause and are relevent to this error?  Here are my top 10 entries from poolmon text file.

    UlHT Paged         1         0         1 8392704     8392704        

    CM35 Paged      7909      7877        32 4694016     146688        

    MmCm Nonp        659         4       655 4547104       6942        

    Obtb Paged     75778     74647      1131 4271136       3776        

    MmSt Paged   4421779   4419804      1975 4048128       2049        

    LSwi Nonp          1         0         1 2584576     2584576        

    TCPt Nonp    1165659   1165632        27 1405616      52059        

    CMAl Paged      2750      2437       313 1282048       4096        

    CM25 Paged      1184       965       219 1134592       5180        

    VoSm Nonp       5832      5812        20 1122480      56124        

    I hope someone can help me with this or let me know if more information is needed.  Thanks in advance.

  38. Courtney Kibbe says:

    Wait a second, I didn’t include the very first entry thinking it was a header but looking at it again it must be a valid entry.  Here it is:

    File Nonp  103274384 102809238    465146 70707328        152

    Coule this be my culprit?

  39. David.Wang says:

    Mathias – The key is to get you to look for something else running on the system that is depleting non-paged-pool. HTTP.SYS is just the messenger; don’t shoot. 🙂

    It’s not always obvious what the culprit is, especially the low-level-hooking ones that want to obscure their existence…

    //David

  40. David.Wang says:

    Courtney – That’s the main culprit (File). It’s chewing up over 15x more NonPagedPool Memory than the next user. I don’t have it on my system. It did show up on Ryan’s poolmon listing above. Perhaps you can diff your software with his and see what is is similar.

    Or use poolmon -c to try and find the driver file associated with the "File" tag – which gives hints on what product is causing this.

    //David

  41. Courtney says:

    Hopfully Ryan will stop by and give us an update.  I’ll list the software that I have on both my servers:

    Windows 2003 Server Std edition

    Backup Exec 10d (on second server but it backs up first server via File Open Agent)

    Symantec 10.0 corporate edition (however I’m upgrading it to 10.1 as I write)

    I can’t think of anything else that is similar between the servers and put a File tag on both.  If my update of Symantec doesn’t fix the issue I’m going to set up a lab and recreate my servers.  I can’t think of another way to do it.

  42. Ryan M. Hager says:

    Hello all,

    1.  Sorry, I actually went to work for another company and am no longer involved in managign that server.  I did run it for two weeks after the NAV upgrade and I did not notice the "leak" any more.  I left a message for a previous colleague of mine.  When I hear back I will give an update.

    2.  As far as file is concerned, I was under the impression it had to do with having files open (the meta information on the OS for managing files ect.) .    Is there a particular application that is working with lots of files, maybe not closing them correctly?  Although, I am not sure to what degree 70,707,328 (70 Meg) is bad.

    3.  Another thing that affects non paged pool size is the use of the /3gb switch.  My first thoughts way back was that the maximum non-paged pool size would increase.  I was wrong.  If you use the /3GB switch, the maximum non-paged pool size decreases from 256MB to 128MB.  

    4.  Courtney, a listing of installed sofware on you server would probably be helpful.  And, I think we all are assuming the erro message you get after some period of time is:  "The server was unable to allocate from the system nonpaged pool because the pool was empty."

    Ryan M. Hager

  43. Courtney says:

    Thanks for posting back Ryan.

    I am using the /3GB switch on both servers.  A while back I read that it was required for servers using more than 2gb or memory.  I too would think having more than 2gb would INcrease the non-paged pool.  Looks like a design flaw to me.

    I’m going to test your file tag theory that it is related to open files.  We program our own reports using Clarion and maybe there’s a newly created report that has gone heywire.

    To answer number 4, The only error that I noticed was the Connections_Refused error in the HTTP logs.  Restarting the server fixed that issue and then I found this website.

    I’ve upgraded Symantec Corporate edition from 10.0 to 10.1.  I will keep updating this website on my progress.

  44. David.Wang says:

    Courtney – Before you call it a “design flaw”, you need to understand how Memory Management works on Windows.

    By default, a 32bit machine can address 4GB of memory per process. 2GB address space is available for use by User mode applications and the other 2GB is reserved for the Windows Kernel.

    Please distinguish between address space and memory allocation. The 2GB is simply available address space to make memory allocations from but are not necessarily all used. Address space is necessary for the kernel to work, as well as efficiently memory map resources between processes, etc (if the kernel can assure that its code maps to the same memory address in all processes, it can do some optimizations while context-switching between processes and NOT remap itself all the time).

    /3GB tells Windows to give 3GB address space to User mode and only 1GB for Windows kernel in every process. It is the cheesy way to allow more User mode address space before the existence of 64bit OS which gives far larger address space (and Memory Management is different as a result).

    Non Paged Pool memory is special kernel-mode memory and gets its address space from the Windows kernel portion of the memory address. Since /3GB halves the available kernel memory address space, it only makes sense that max NPP size decreases from 256MB to 128MB.

    It doesn’t make sense for NPP to use user-mode memory address, nor does it make sense for NPP to stay at 256MB at the expense of some other kernel mode memory need.

    In other words, /3GB constricts kernel memory address space and hence NPP address space, so seeing it decrease from 256MB to 128MB with /3GB is “by-design”. I see no flaw here.

    //David

  45. vickyd35 says:

    HI Dave, first I would like to say this has site has been a tremendous help in my troubleshooting of IIS6 connection refused problem.  This was the only location that gave me a clue about what my IIS6 connectin issue was.  

    Now I just have to figure out what driver or app is causing the problem.  I’ve reviewed my Poolmon records and it appears that a the DDK drivers are the problem.  How do I determine which one?  

    Here is a sample copy of my poolmon log:

    Memory: 2096668K Avail: 1460508K  PageFlts:   487   InRam Krnl: 3168K P:92976K

    Commit:2418032K Limit:8345212K Peak:2472812K            Pool N:49240K P:94076K

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

    Ddk  Nonp     487036 (  42)    429814 (  36)    57222 21111424 (  2016)    368

    File Nonp    4611529 ( 211)   4564638 ( 208)    46891 7132072 (   456)    152

    LSwi Nonp          1 (   0)         0 (   0)        1 2576384 (     0) 2576384

    TCPc Nonp     156911 (  12)    116183 (   9)    40728 1954944 (   144)     48

    TCPt Nonp     246614 (   2)    246587 (   2)       27 1392560 (     0)  51576

    Irp  Nonp     200406 (   0)    197990 (   1)     2416 1015800 (  -448)    420

    Vickyd

  46. Ryan M. Hager says:

    Hello all,

    Ok, I did get an update from my former colleague .   "tdll" is still an issue for them.   They are going to rebuild on SVR 2003 and see if the problem persists.

    Ryan M. Hager

  47. vickyd35 says:

    I un-installed a CommView and that stopped all leaks.  The driver for this application was the problem.

    Vickyd

  48. Mark says:

    I was wondering if someone could help me figure this one out.  I have a W2k3 SP1 server that’s refusing connections every 3 – 4 weeks.  It appears that Afdb is the tag that is growing.  Here are the poolmon captures:

    Memory: 2096400K Avail:  796916K  PageFlts:   228   InRam Krnl: 3704K P:89264K

    Commit:1095580K Limit:4043356K Peak:1257404K            Pool N:142776K P:91028

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

    AfdB Nonp     265443 (   0)     59297 (   0)   206146 103506360 (     0)    50

    R100 Nonp         21 (   0)         0 (   0)       21 9585384 (     0) 456446

    MmCm Nonp        585 (   0)        32 (   0)      553 3326496 (     0)   6015

    tdLL Nonp      27230 (   0)     21771 (   0)     5459 2998680 (     0)    549

    File Nonp   52425165 (  65)  52408019 (  65)    17146 2692304 (     0)    157

    LSwi Nonp          1 (   0)         0 (   0)        1 2584576 (     0) 2584576

    TCPt Nonp     699513 (   5)    699485 (   5)       28 1456216 (     0)  52007

    PTrk Nonp     655930 (   0)    651945 (   0)     3985 1196840 (     0)    300

    Ntfr Nonp    1062944 (   0)   1045950 (   0)    16994 1088584 (     0)     64

    MmCa Nonp    3450887 (  13)   3442105 (  13)     8782  974976 (     0)    111

    Thre Nonp    2981123 (  39)   2979785 (  39)     1338  834912 (     0)    624

    TCPA Nonp      40121 (   0)     38423 (   0)     1698  624864 (     0)    368

    Pool Nonp          6 (   0)         3 (   0)        3  610304 (     0) 203434

    AfdE Nonp      40618 (   0)     38859 (   0)     1759  492520 (     0)    280

    NtFs Nonp    1819450 (   0)   1808907 (   0)    10543  425464 (     0)     40

    Irp  Nonp     945247 (   2)    944182 (   0)     1065  423592 (   960)    397

    Ntfn Nonp     652134 (   0)    641592 (   0)    10542  422864 (     0)     40

    LSwr Nonp        128 (   0)         0 (   0)      128  416768 (     0)   3256

    CcSc Nonp    1958366 (   0)   1957168 (   2)     1198  373776 (  -624)    312

    Devi Nonp       1688 (   0)      1320 (   0)      368  366368 (     0)    995

    Memory: 2096400K Avail:  657416K  PageFlts:   330   InRam Krnl: 3704K P:81644K

    Commit:1187564K Limit:4043356K Peak:1257404K            Pool N:196036K P:83344

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

    AfdB Nonp     394396 (   0)     86013 (   0)   308383 155097912 (     0)    50

    R100 Nonp         21 (   0)         0 (   0)       21 9585384 (     0) 456446

    MmCm Nonp        585 (   0)        32 (   0)      553 3326496 (     0)   6015

    tdLL Nonp      33103 (   0)     27643 (   0)     5460 2999232 (     0)    549

    LSwi Nonp          1 (   0)         0 (   0)        1 2584576 (     0) 2584576

    File Nonp   68066779 (  90)  68052843 (  91)    13936 2145600 (  -152)    153

    TCPt Nonp     894609 (  11)    894581 (  11)       28 1456216 (     0)  52007

    PTrk Nonp     863427 (   0)    859206 (   0)     4221 1267176 (     0)    300

    TCPA Nonp      52932 (   0)     50457 (   0)     2475  910800 (     0)    368

    Thre Nonp    3799300 (  49)   3797879 (  47)     1421  886704 (  1248)    624

    AfdE Nonp      53618 (   0)     51082 (   0)     2536  710080 (     0)    280

    Pool Nonp          6 (   0)         3 (   0)        3  610304 (     0) 203434

    Ntfr Nonp    1373312 (   0)   1365233 (   0)     8079  518024 (     0)     64

    MmCa Nonp    4489055 (  13)   4485021 (  14)     4034  442624 (   -96)    109

    Irp  Nonp    1020062 (   2)   1018966 (   0)     1096  435368 (   960)    397

    LSwr Nonp        128 (   0)         0 (   0)      128  416768 (     0)   3256

    Devi Nonp       2026 (   0)      1658 (   0)      368  366368 (     0)    995

    Even Nonp    9250970 ( 151)   9243598 ( 151)     7372  360000 (     0)     48

    Vad  Nonp    5896241 (  22)   5890766 (  27)     5475  262800 (  -240)     48

    CcSc Nonp    2518751 (   2)   2517918 (   2)      833  259896 (     0)    312

    MmCi Nonp       6732 (   0)      5605 (   0)     1127  253392 (     0)    224

    TCPc Nonp     349953 (   0)    344732 (   0)     5221  250608 (     0)     48

    I found an article that says that there was a problem with afd.sys that was supposed to be fixed in Win2000 SP3.  Here’s the article:

    http://support.microsoft.com/kb/296265

    Anyone have any ideas what might be causing this?

    Thanks,

    Mark

  49. Ryan M. Hager says:

    Mark,

    What software are you using?  More specifically, assuming AFD=Winsock like the article suggests, what apps are you running on this machine that are communicating over TCP-IP?  I am not sure if it matters, but are you using a NIC that does TCP Offloading?

    Ryan.

  50. Philip Mollica says:

    I followed all the great info here, and found that the NTFC or NTFS driver was taking up about 60MB. After restarting, it’s down to almost nothing. What could cause this?

  51. Mark says:

    Ryan,

    Thanks for the response.  It has SQL Server 2000, IIS, SMTP Service, Norton AntiVirus, Windows Services for Unix and a USQL client app for connecting to a Cobol db.  The NIC does TCP Offloading.  The server is also used to print custom labels for use in the business and I’m starting to think that printing is the problem.  It’s hard to find a time when there isn’t any traffic to the server but it seems that if I watch Poolmon, the Afdb tag grows every time I print a label.  

    Mark

  52. Ryan M. Hager says:

    Mark,

    I am not sure how you are printing, and that could be it.  However, my first thought was the USQL client.  Does your printing application also use the USQL client to connect to the DB?   It may be a combination of Label Printing Application and USQL client.  Additionally, are you using any special print queue software?

    Ryan M. Hager

  53. Mike says:

    Hi,

    We were having this exact problem on a few of our Windows 2003 IIS 6 machines.  A search led me to this blog entry.  While the information provided is extremely helpful, we have hit a wall with finding the actual culprit.

    As a few others have reported here and elsewhere on the web, something that uses the tag "File" is consuming an ever increasing amount of NPP memory.   I have yet to determine which driver this is.  Has anyone else had any luck with this?  Using the findstr command I find 76 drivers.  I can post that list here if anyone thinks it would be helpful.

    Here is some output from poolmon -b -c -p, including the top five users of NPP:

    Memory: 2096580K Avail: 700456K  PageFlts: 316 InRam Krnl: 2192K P:80572K

    Commit:1458792K Limit:4042240K Peak:1552600K  Pool N:200948K P:81972K

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes      Per Alloc  Mapped_Driver

    File Nonp    9946333 ( 193)   9149316 ( 161)   797017 121175032 (  4864)    152 Unknown Driver

    TCPc Nonp    1595850 (  66)    810930 (  34)   784920 37676160 (  1536)     48 [tcpip]

    NDam Nonp     224898 (   2)    223669 (   4)     1229 4192952 (-32768)   3411 [ndis]

    MmCm Nonp        546 (   0)         4 (   0)      542 3891744 (     0)   7180 Unknown Driver

    LSwi Nonp          1 (   0)         0 (   0)        1 2584576 (     0) 2584576 Unknown Driver

    The two third party applications that are running at the time those numbers were taken are SmarterStats 3 and Visnetic Firewall.  Stopping both services shows no noticeable change in the NPP usage.

    Any suggestions or comments would be greatly appreciated.  If any further information is required, please let me know.  Thank you.

  54. Ryan M. Hager says:

    Mike,

    Although I was not working on a 2003 box, your TCPc seems high (37MB) as well.  Of the two apps you listed Visnetic Firewall uses files (Log files) and TCP, so a couple of thoughts based on that:

    1.  How many log files do you have on the machine for Visnetic Firewall?  Is it one big one? Can they/it be shrunk?

    2.  Does stopping Visnetic Firewall really unload it from memory?  I am not sure how firewalls attach to the network stack.  Maybe you can add a hardware firewall in front of the box and disable it and reboot to see if the problem comes back?

    3.  At this point, you may want to open a Ticket with Microsoft Support.   It may be the fastest way to resolve.  

    Ryan M. Hager

  55. Mike says:

    Ryan,

    Thanks for responding.  I agree that TCPc seems fairly high, but I don’t have a lot to compare that to.  While others have posted numbers for TCPc, they may not mean a whole lot without knowing what type and how much traffic occurs on these systems.

    1.  Visnetic Firewall rotates the log files once a month or at 25 megs, whichever comes first.  There are only 28 log files on this particular server covering the past 13 months.  The directory containing the log files is compressed.  I suppose I could disable logging for a period of time to see how that affects the memory usage.

    2.  Good question.  The server is in a remote data center.  Putting a hardware firewall in front for debugging purposes wn’t be an easy task.

    3.  Yeah, we may need to go that route. 🙁

    I’m taking the list of drivers produced by running "findstr /m /l File *.sys" and comparing that with a list of loaded drivers to see if anything stands out.  So far two drivers from the Visnetic Firewall,  AMBRAPP.sys and AMBRIM.sys, show up in both lists.  Does this seem like a reasonable approach to attempt to track down a probable culprit?

    Thanks again for the quick response.

  56. Ryan M. Hager says:

    Mike,

     I was not very successful in directly tracking the issue back to the source of the problem and I spent a considerable amount of time trying to track it.  Unfortunately, it was right before I left that I found David Wang’s blog.  

    I think the information provided by Poolmon can point you in the right direction and help you formulate a theory that you can test.  At this point if it looks like it may be Visnetic Firewall, I would load up a test machine similar to production and running something like Microsoft Web Application Stress Tool against it to simulate the web traffic you receive, and also running a vulnerability scanner like Nessus in a loop to similate the attack packets coming over the net.

    One additional thought, how do you back up this machine?  Possibly a network backup?  

    Ryan M. Hager

  57. Mike says:

    Sorry it took me so long to respond.

    We’ve recently gotten in contact with VisNetic regarding the firewall.  They have acknowleged that the product can cause these issues on Windows 2003 and have offered a beta replacement for the ambrim.sys driver.  I’ll post back with my findings.

    -Mike

  58. Mark says:

    I just wanted to post a followup to my previous posts.  I ended up calling Microsoft and we setup poolmon and perfmon to constantly run and capture data.  After a couple of days I sent them the logs and they found that the leak was in the HP print driver.  We updated the HP print driver to the latest version for the HP4250 printers we have and also changed the setup of the printers to use a Standard TCP/IP port rather than the HP Standard TCP/IP port.  Also, just to be on the safe side they had me run an update for Norton VirusScan.  After a couple of weeks it appears that the leak has been eliminated.  

    Thanks for the help you all have provided!

    Mark

  59. Mike says:

    Another update for our issues.  The replacement ambrim.sys driver provided by VisNetic had no positive effects.  I’ve disabled the HTTP filtering feature of the firewall and that seems to have alleviated the NPP memory usage problems.  Perhaps I should approach 8Signs/C&C Software directly about this and leave VisNetic out of it.

    -Mike

  60. Nick Price says:

    I’m having a similar problem with running out of non-paged memory.  In my case, the tag that is responsible is "Devi" – can anyone suggest a way to work out which specific driver would be responsbile?

    Memory: 2096616K Avail: 1488572K  PageFlts:   403   InRam Krnl: 3928K P:65404K

    Commit: 460988K Limit:4034104K Peak: 498364K            Pool N:78908K P:71948K

    Tag  Type     Allocs            Frees            Diff   Bytes      Per Alloc

    Devi Nonp      14065 (  40)       540 (   0)    13525 47746248 (145280)   3530

    Wdm  Nonp     257238 ( 807)    256463 ( 807)      775 6189872 (     0)   7986

    NVP  Nonp       2010 (   0)         0 (   0)     2010 2888320 (     0)   1436

    ISil Nonp       7185 (   4)      7165 (   4)       20 1638096 (     0)  81904

    MmCm Nonp        706 (   0)        64 (   0)      642 1362952 (     0)   2122

    Stor Nonp       3076 (   0)      1083 (   0)     1993 1326576 (     0)    665

    tdLL Nonp       2190 (   0)       354 (   0)     1836 1211600 (     0)    659

    File Nonp     312933 ( 295)    306434 ( 295)     6499 1032536 (     0)    158

    SNPU Nonp        465 (   0)         1 (   0)      464 1027656 (     0)   2214

    NV   Nonp       2748 (   0)      1876 (   0)      872  827616 (     0)    949

    NRAT Nonp          2 (   0)         0 (   0)        2  516096 (     0) 258048

    Thanks,

    Nick Price

  61. David.Wang says:

    Nick – Since you want to troubleshoot your situation, I suggest carefully reading both this blog entry as well as the associated comments since they both have a lot of information on how to figure out what you want.

    For example, the blog entry mentions:

    http://support.microsoft.com/?id=177415

    Which also mentions:

    http://support.microsoft.com/?id=298102

    //David

  62. Scott says:

    excellent.  just what i needed.

  63. D.Schoblocher says:

    Hi all,

    I have the exact problem on a Windows 2003 IIS 6 machines. This information in this blog is very helpfull. With the help of poolmon I figure out the problem on our server. The Watchguard Admin Gui causes the memory problem.

    Thanks to all

    schobi

  64. Jessie Kom says:

    I seem to be experiencing the same problem with two of my servers.

    Server A has "tdLL" at the top of the list, at 99005465 bytes, and second on the list is "SavE" at 16529752 bytes, at the time of writing.

    Server B, however, has "SavE" at the top of its list with the same bytes as Server A, and with "tdLL" nowhere to be found.

    Both servers are already running Symantec AntiVirus ver 10.0.0.359. The only difference between them is that Server B is running the AntiVirus as a Server instead of a client.

    So I’m not really sure if "tdLL" belongs to Symantec, and it’s not a good idea to uninstall the Antivirus from Server A to find out because I don’t have another authorised antivirus software to protect Server A.

    Any suggestion to my next step?

    JK.

  65. David.Wang says:

    Jessie Kom – people have reported tdLL leaks going away with Symantec AV 10, though unclear what patch/sub-build number. You do not appear to have the latest at 10.1 nor 10.1.

    I suggest contacting Symantec support regarding the issue and getting their latest AV 10 updates. Since you need to run that software, I do not recommend trying to uninstall it from an investigation point-of-view.

    Basically, there is no way for you to deal with the memory leak other than to either:

    1. NOT run the code with the leak

    2. run code without the leak

    And since you cannot do #1, you must do #2 – meaning you must contact and obtain the latest patch, and if it does not work, remain with Symantec support until they prove/show that tdLL is not theirs or is not their issue.

    //David

  66. Jessie Kom says:

    Thanks David.

    I’ve just downloaded 10.0.2 from Symantec, it’s not the latest patch but that’s all I can download, probably due to the support plan my company got.

    However, yeah, I’ll be checking with the Symantec support on this "tdLL" and the patches I can get.

    Will update you guys again…

  67. Rick B. says:

    I have had the same problem this week..

    I used to work for Microsoft Platform Performance and used to track down leaky pool tags all the time.  

    In this case, tdLL and afdc were leaking.  However tdLL was much worse.  I tracked the tag down to two Symantec network drivers that are part of the Symantec Redirector.  The drivers are called Symndis.sys and Symtdi.sys

    I found an update on Symantec’s site for AV corp ed 9.0. here:

    http://www.symantec.com/techsupp/enterprise/products/sav_ce/savce_9.0/files.html

    I’m waiting to see if that actually resolves it.  Otherwise I’ll open collect a LiveKD dump and have Symantec debug their crap.

  68. Rick says:

    "Solution Delivered"

    The patch I mentioned above was for 9.0.5, so I went ahead and removed 9.0 and installed 10.0.  Symndis.sys and Symtdi.sys were actually removed, thus no tdLL tag, thus no leak.

    IMO, if you install 10.0 and still have a tdLL tag, try uninstalling everything and rebooting, rather than upgrading.  Inplace upgrading might leave the driver in place, while uninstalling and upgrading seems to do the trick…

    This analysis was from a log that ran over two days.

    ——————————————————————

    ———————-Pool Log Analysis—————————

    ——————————————————————

    Pool Tag: File

    Type: Pool_Non-Paged

    Leaking?: *YES*

    First/Average/Last/High Values: 1.76 MB / 1.91 MB / 1.99 MB / 1.99 MB

    Mapped Driver: <unknown> – File objects

    ——————————————————————

    Log File: C:Documents and SettingsboyettrvDesktoppoolmon_2.pmls

    Log file size: 8.14MB

    Total Snapshots: 144

    Time to Analyze: 0:15

    Analysis performed by PLAT.EXE Version 2.3.70

    ——————————————————————

  69. Lou says:

    David,

    Great article.  its the most insightful I’ve seen on the topic in my few days of searching.

    My company is loadtesting a new asp.net application running on windows 2003 SP1, IIS 6, 2 gig ram and 2 cpu’s.  We are loadtesting using mercury loadrunner.  When the load gets to about 140 users loadrunner starts receiving 100061 Connection Refused messages.  the strange thing is I don’t see any Connection_Refused messages in the httperr#.log.  The processors are only about 50 % used when this error begins.  

    Any idea’s?  I don’t think it is another process using up memory; instead I think it might be ASP.net or IIS running out of connections or threads but haven’t found any way of verifying this.

    Thanks,

    Lou

  70. David.Wang says:

    Lou – I would totally check on the ASP.Net application and its perf counters:

    – is it queuing?

    – How is memory utilization during this same period?

    – What about lock contention?

    Pure IIS and ASP.Net can handle way more than your load. I believe your ASP.Net application is the bottleneck and cause of bad behavior.

    Can you explain to me why when you see a 10061 error that you consider an ASP.Net application which has probably seen 6-9 months of testing as MORE reliable/performant than IIS6/ASP.Net, which has gone through years of testing, performance benchmarking/tuning, etc.

    Because it is rare for ASP.Net or IIS to "run out of connections or threads", and even if it does happen, it usually indicates a problem with the user’s application code.

    http://blogs.msdn.com/david.wang/archive/2006/03/14/Thoughts_on_Application_Pools_running_out_of_threads.aspx

    //David

  71. JohnE says:

    Thanks for the info in this Blog.  I have got the problem with the File tag.  Trying to track it down. Will let you know what I find.

    Memory: 2096572K Avail:  127812K  PageFlts:    28   InRam Krnl: 2112K P:24936K

    Commit:2116980K Limit:4042232K Peak:2237084K            Pool N:246824K P:46552K

    Tag  Type     Allocs            Frees            Diff   Bytes      Per Alloc

    File Nonp  1432693511 ( 842) 1431507006 ( 847)  1186505 180352184 (  -776)    1

    Thre Nonp  418801034 ( 214) 418795362 ( 218)     5672 3539328 ( -2496)    624

    LSwi Nonp          1 (   0)         0 (   0)        1 2584576 (     0) 2584576

    MmCm Nonp         42 (   0)        16 (   0)       26 2015776 (     0)  77529

    TCPt Nonp   21858688 (   7)  21858659 (   7)       29 1456480 (     0)  50223

  72. JohnE says:

    I have had three live servers with the problem.  All with the File tag at around 200MB.  I had to reboot those servers to get the users back on, but I have now found a dev server that is not being used and is currently at 194MB and I am getting the Connection Refused error.

    I am having problems trying to track down the File tag.  As with some of the other entries here, the word File appears in a number of drivers.  I have also tried hFile as recommended in one of the Microsoft articles and I come up with en IBM TSM driver.  However, I have another PC that has the File tag, but when I search for hFile, it does not return anything….so I am not sure I trust the IBM find, although it is file related…..still searching.

    Rgds

  73. JohnE says:

    I have found the problem.

    It was an IBM WMI service with a process called PegasusProviderAdapter.exe. Killed this off (as it did not matter what services I stopped…it didn’t) and the File tag went back to 400K.  

    It was also highlighted by using the Task Manager and showing the Handles column.  This was over 1 million.  Also the FileMon tool from SysInternals showed a lot of access from the exe.

    So, if you problem is the File tag, check Task manager for Handles.

    Thanks all above for your pointers….it got me there.

  74. Chris Jones says:

    Really helpful, but what do we do when:

    poolmon /c

    gives us the following output

    Poolmon: No localtag.txt in current directory

    Poolmon: Unable to load required dlls, cannot create local tag file

    I gather that this means we can’t generate a localtag.txt file and therefore can’t workout which driver is the problem – any ideas?

  75. RobertK says:

    Hello

    Finally i found the right place. Great postings.

    My problem is with clustered active/passive exchange 2003 running on windows2003 ent. Both nodes have 4GB of RAM and Boot.ini includes /3GB and USERVA=3030 switches. When i check my NPP Pool is actually shows around 250Megs instead of the ususal 128Megs

    I too keep getting connections refused after few days

    It manifest itself within my HTTP cluster resource which goes off line and then it simply dies. The only way to solve this problem is to fail over to the other node. This system was build about 2 years ago and only after applying service pack 2 for exchange 2003 i;m seeing this error.

    My top 6 main culprits from poolmon

    Memory: 3669384K Avail: 1157048K  PageFlts: 89870   InRam Krnl: 2812K P:145184K

    Commit:1974712K Limit:7718392K Peak:2047588K            Pool N:96468K P:146244K

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

    MmCm Nonp       5861 (   0)       202 (   0)     5659 25068320 (     0)   4429

    File Nonp   24187515 (4013)  24127756 (4050)    59759 9096488 ( -5672)    152

    TCPT Nonp      64297 (  13)     44875 (  20)    19422 7924608 ( -2856)    408

    AfdE Nonp     801580 ( 332)    781989 ( 339)    19591 5485480 ( -1960)    280

    Irp  Nonp  137215759 (15216) 137204307 (15253)    11452 4891440 ( -9056)    427

    Mdl  Nonp     287772 (   4)    273441 (   4)    14331 4850408 (     0)    338

  76. Ryan M. Hager says:

    RobertK,

    According to what David Wang says (which I believe because it matched my experience) Using the /3gb switch minimizes your non-paged pool memory to 128MB.  

    1.  Where are you getting the status that you are actually using 250Megs.   I have found some performance stats are not as reliable on this measure.  THe only thing that seems to be correct is poolmon.  If you look at your poolmon – Pool N:96468K.  Please let me know why you think it is 250MB?

    2.  Have you tried not using the /3GB switch?

    Ryan M. Hager

  77. Jessie Kom says:

    Hi guys,

    I’m so sorry I actually forgot to update you!

    My problem was confirmed resolved after I upgraded the SAV server and clients on the servers to version 10.0.2.2021.

    The "tdLL" tag on one server running SAV client is now gone for good, and a leaking "NtFC" on another server running SAV server is also gone.

    It’s been a few couple of months now, and my monitoring still shows me stable NPP usage.

    Cheers!

  78. Mehul Choksi says:

    Hi,

    We have examined this document. we are receiving same error which is specified in this document and we have also install this poolmoon.exe but, we could not find how to resolve this issue in this document so, please add some resolution document on this error.

    Thanks,

    Mehul Choksi.

  79. paritoshjani says:

    The error of connection refused, leaves IIS unresponsive….

    below is the result of poolmon.exe, if someone can advice on it.

    Memory: 4062672K Avail: 2774072K  PageFlts: 37671   InRam Krnl: 2500K P:233456K

    Commit:1211472K Limit:6000160K Peak:1216708K            Pool N:169772K P:235880

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

    File Nonp    5116787 (3619)   4559778 (3183)   557009 85373928 ( 66272)    153

    TCPc Nonp     959614 ( 572)    482062 ( 286)   477552 22922496 ( 13728)     48

    MmCa Nonp     450958 ( 475)    374788 ( 401)    76170 8523088 (  7984)    111

    Ntfr Nonp     126776 ( 266)     10695 (   0)   116081 7430152 ( 17024)     64

    NDam Nonp      57933 (  41)     56351 (  42)     1582 7365312 (-16384)   4655

    CcSc Nonp     285330 ( 272)    270347 ( 239)    14983 4794560 ( 10560)    320

    MmCm Nonp        663 (   0)         4 (   0)      659 4456992 (     0)   6763

    Ntfn Nonp     111696 ( 156)     27099 (  48)    84597 3385360 (  4320)     40

    NtFs Nonp     381250 ( 615)    296715 ( 570)    84535 3382648 (  1800)     40

    LSwi Nonp          1 (   0)         0 (   0)        1 2584576 (     0) 2584576

    Irp  Nonp     698615 (   0)    694779 (   1)     3836 1668008 (  -160)    434

    TCPt Nonp    1868090 (1366)   1867946 (1366)      144 1471912 (     0)  10221

    TPLA Nonp        256 (   0)         0 (   0)      256 1048576 (     0)   4096

    Thre Nonp      55309 (  46)     54022 (  23)     1287  803088 ( 14352)    624

    Wmit Nonp         13 (   0)         0 (   0)       13  667648 (     0)  51357

    Pool Nonp          6 (   0)         3 (   0)        3  610304 (     0) 203434

    FSfm Nonp      87889 ( 111)     74653 (   9)    13236  529440 (  4080)     40

    ReTa Nonp      13744 (  53)       887 (   0)    12857  516360 (  2120)     40

    Io   Nonp    4471379 (3014)   4470604 (3017)      775  495528 (  8032)    639

    ReEv Nonp      22072 (  13)      2210 (   0)    19862  476688 (   312)     24

    Ddk  Nonp       3118 (   0)       460 (   0)     2658  425280 (     0)    160

    LSwr Nonp        128 (   0)         0 (   0)      128  416768 (     0)   3256

    Even Nonp     517899 ( 298)    510041 ( 173)     7858  380368 (  6016)     48

    UlCO Nonp       2624 (   0)      2000 (   1)      624  379392 (  -608)    608

    Stat Nonp     505305 ( 305)    503049 ( 298)     2256  379008 (  1176)    168

    Devi Nonp        446 (   0)       116 (   0)      330  356280 (     0)   1079

    MmCi Nonp       1582 (   0)       140 (   0)     1442  313888 (     0)    217

    UlCJ Nonp       1579 (   0)         0 (   0)     1579  290536 (     0)    184

    Mm   Nonp         90 (   0)        72 (   0)       18  284536 (     0)  15807

    Vad  Nonp     908965 ( 701)    903490 ( 483)     5475  262800 ( 10464)     48

    TCPp Nonp         64 (   0)         0 (   0)       64  262144 (     0)   4096

    AfdC Nonp     467791 ( 283)    466222 ( 283)     1569  251040 (     0)    160

    NDpp Nonp         74 (   0)         1 (   0)       73  250816 (     0)   3435

    Mdl  Nonp       5472 (   0)      3565 (   0)     1907  248136 (     0)    130

    TCPB Nonp     194439 (  93)    194300 (  93)      139  214696 (     0)   1544

  80. Corey says:

    I get IIS crashing about every 2-3 hours.

    When i run poolmon it lists the tag mmcm using almost 60MB or non paged ram.  Does anyone know what this is and how to fix it.

  81. ravindrams says:

    Dear David,

    We are having the similar issue in windows server 2003 (SP1) in our plant automation system.

    Windows Server 2003 system system hosts the following

    1. IIS 6.0 webserver

    2. TougoServer service (COM server) written by us which uses bacnet API`s (uses WINSOCK APIs) to get the data from the devices on the plant (currently simulated with Area server running in Linux system)

    3. VB.Net webservice which runs in the context of IIS worker process, which clients can connect to webserver to get the various data.

    Problem: Kernel non-paged memory is getting increased 1-3 MB per day. After it reaches the limit, bluescreens.

    We have running pool monitor every one hour.

    Below is one snapshot of pool monitor.

    Any advice on how to procede on this.

    I can provide any other data you need.

    thanks in advane,

    Ravi

    Tag Type Allocs Frees Diff Bytes Per Alloc Mapped_Driver

    MmSt Paged 12212334 12129732 82602 88373496 1069 [nt!mm – Mm section object prototype ptes]

    File Nonp 137869051 137708041 161010 26918944 167 [<unknown> – File objects]

    CcSc Nonp 22773687 22697317 76370 23827440 312 [nt!cc – Cache Manager Shared Cache Map]

    Ntfl Paged 608677 532946 75731 12116960 160 [ntfs.sys –     LCB]

    Ntfr Nonp 213098 50895 162203 10381960 64 [ntfs.sys –     ERESOURCE]

    CcBc Nonp 16072587 15996615 75972 10332192 136 [nt!cc – Cache Manager Bcb from pool]

    MmCa Nonp 49033417 48952076 81341 9101856 111 [nt!mm        – Mm control areas for mapped files]

    Sect Paged 61708966 61632141 76825 6762856 88 [<unknown>    – Section objects]

    Thre Nonp 27246452 27238149 8303 5181072 624 [nt!ps        – Thread objects]

    Gcac Paged 106173 105960 213 4534536 21288 [<unknown>    – Gdi glyph cache]

    MmSm Paged 10144050 10063228 80822 4526032 56 [nt!mm       – segments used to map data files]

    UlHT Paged 1 0 1 4198400 4198400 [http.sys – Hash Table]

    ReEv Nonp 167564 13143 154421 3706104 24 [<unknown>    – Resource Event]

    Ntfn Nonp 883212 801305 81907 3277464 40 [ntfs.sys     – SCB_NONPAGED]

    NtFs Nonp 10391825 10309919 81906 3276528 40 [ntfs.sys     – StrucSup.c]

    MmCm Nonp 200 177 23 3155008 137174 [nt!mm        – Calls made to MmAllocateContiguousMemory]

    LSwi Nonp 1 0 1 2576384 2576384 [<unknown> –     initial work context]

    Ttfd Paged 369475 368522 953 2576328 2703 [<unknown>    – TrueType Font driver]

    Gh05 Paged 1974843 1974660 183 2492360 13619 [<unknown>    – Gdi Objects]

    ReSe Nonp 86457 9345 77112 2467584 32 [<unknown>    – Resource Semaphore]

    Gh25 Paged 333349 333005 344 1642488 4774 [<unknown>    – Gdi Objects]

    FSim Paged 709677 697024 12653 1619584 128 [nt!fsrtl     – File System Run Time Mcb Initial Mapping Lookas

    NtfF Paged 57731 56032 1699 1590264 936 [ntfs.sys     – FCB_INDEX]

    TCPt Nonp 71858 71832 26 1392544 53559 [TCP          – TCP/IP network protocol]

    TSdd Paged 82864 82830 34 1127424 33159 [<unknown>    – RDPDD – Hydra Display Driver]

  82. Hiram says:

    Vickyd,

    Were you able to find what driver was associated to the Ddk tag?

    Can someone please help find what could cause the Ddk driver to take the most memory?

    Thank you,

    Hiram

    Here is a sample copy of my poolmon log:

    Tag Type Allocs Frees Diff Bytes Per Alloc

    Ddk Nonp 4247411 2134379   21 13032 1 86928776 88

    MmCm Nonp 31471 30363 1108 1 3652784 12322

    Thre Nonp 29212 27806 1406 877344 624

    WLBS Nonp 36050 36038 12 766528 63877

    R100 Nonp 28 2 26 9 658968 371498

    File Nonp 1183112 1179027 4085 622344 152

    Pool Nonp 6 3 3 610304 203434

  83. yaplej says:

    I am almost having a problem with a Ddk driver, but cannot figure out what one.  The methods listed in KB298102 to find what drivers are using the Ddk tag does not seem to work.

  84. Michael says:

    I am having this issue but it is the MmCm continually growing until connection refused shows up in the logs.  Here is the poolmon read from a Server 2003 SP2.

    Memory: 8386228K Avail: 7580940K  PageFlts:   659   InRam Krnl: 3472K P:60848K

    Commit: 580648K Limit:10281340K Peak: 593208K            Pool N:240764K P:61784

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

    NTID Nonp    3068013 (  86)   2807570 (  76)   260443 205441816 (   960)    788

    MmCm Nonp     241685 (   1)    241524 (   1)      161 9434384 (     0)  58598

    UlHT Paged         1 (   0)         0 (   0)        1 4198400 (     0) 4198400

    NDpp Nonp        987 (   0)         0 (   0)      987 3930304 (     0)   3982

    Wdm  Nonp          2 (   0)         0 (   0)        2 3420160 (     0) 1710080

    LSwi Nonp          1 (   0)         0 (   0)        1 2576384 (     0) 2576384

    MmSt Paged    182187 (   7)    180907 (   8)     1280 2213072 (  -520)   1728

    TCPt Nonp     399304 (  11)    399274 (  11)       30 1458080 (     0)  48602

    TPLA Nonp        256 (   0)         0 (   0)      256 1048576 (     0)   4096

    Mdl  Nonp     184477 (  38)    178670 (  36)     5807  857824 (   256)    147

    brcm Nonp     104343 (   0)    104319 (   0)       24  851968 (     0)  35498

    Gh15 Paged      2809 (  16)      2709 (  16)      100  808528 (     0)   8085

    File Nonp    7593995 ( 156)   7589981 ( 158)     4014  612384 (  -304)    152

    Thre Nonp     160179 (   3)    159259 (  11)      920  574080 ( -4992)    624

    CM16 Paged       501 (   0)       372 (   0)      129  557056 (     0)   4318

    I found tech article 944011 and have update the nic drivers and team drivers and it is still climbing.  Any suggestions would be helpful.  THis is a IIS6 server with asp and WebSphere Edgeserver components running.

    Michael

  85. Overview It’s not unnatural to assume an IIS process hang when web client browsers begin reporting either

  86. Overview It’s not unnatural to assume an IIS process hang when web client browsers begin reporting either

  87. David.Wang says:

    It may be "natural", but that is not necessarily correct.

    People frequently confuse what seems natural to them as "their way of thinking is correct", so they assume they are correct… and we all know how far that flies…

    //David

  88. David.Wang says:

    Corey – your problem does not sound like this issue. It sounds like you are running unstable web applications and should diagnose it using my other blog entries on how to diagnose Application Pool Crashes.

    //David

  89. David.Wang says:

    paritoshjani – for FILE pooltag, it indicates that there are lots of open file handles in user mode, causing kernel mode depletion of NPP.

    You should check with Task Manager to see what process(s) have the most "Handles" and stop it — that will most likely be the culprit which is leaking the file handles in user mode and thus the FILE pooltag in kernel mode.

    //David

  90. Khai says:

    Hi,

    I’m having the same issue. I’m running an Exchange front-end/back-end config and my back-end IIS site goes down every once in a while because of this problem (connection refused).

    Here’s a screenshot of poolmon when the problem exists:

    http://img167.imageshack.us/img167/9890/poolmonbeforerebootto4.gif

    The two largest tags are File (50MB) and MmCm (34Mb). These are relatively small compared to others who have posted, nonetheless, the problem exists. I generated then looked through the localtag.txt file but could not find either of these tags.

    My server is a DL385G2 so I downloaded the latest PSP pack (contains all drivers for the server) and installed it. We’ll see if the problem comes back next week.

  91. Khai says:

    I restarted the system and File takes up only 1.8MB. After about 2 hours, it grew to 2.2MB.

    Running "findstr /m /l File *.sys" under c:windowssystem32drivers returns a long list of drivers. At this point, I’m stuck. I can track down each and every one of those drivers and try to remove the ones I don’t need but that’s asking for trouble since I can’t be sure which one is safe to remove.

    Is there a better solution? Somebody posted about an IBM-related driver causing the leak on his system but I don’t have that on my system.

    So far, JohnE seems to have the answer for those with "File" tag problems. Viewing the number of handles using Task Manager when the problem occurs should point you to the culprit. Unfortunately for me, I’ve already restarted my server so I’ll have to wait until the next occurrence.

  92. DIALOGSoftware says:

    Hi,

    Since the launch of our new W2k3 Webserver, we encounter this connexion_refused problem each week.

    I’ve followed the whole tutorial and find some tags that could cause the NPP occupation :

    . file

    . even

    After a findstr in the drivers folders, it seems that the tag "even" is linked to the AMBRAPP.SYS used by my firewall (8 signs).

    I’ve patched it in last release and verified that the driver has been correctly patched : it has been done.

    The connexion_refused just came back.

    I’ve patched my NIC drivers and all that could be patched, deleted all softs that were useless, stopped all services that were useless.

    The connexion_refused just came back.

    I don’t know what to do more.

    Here is an extract of my poolmon when the connexion_refused came today :

    Memory: 4192636K Avail: 1756012K  PageFlts:  1305   InRam Krnl: 2556K P:52452K

    Commit:2771316K Limit:6117192K Peak:2817516K            Pool N:109652K P:53576K

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

    File Nonp   12396850 ( 179)  11996683 ( 176)   400167 60826792 (   392)    152

    TCPc Nonp    1329917 (  12)    932866 (   7)   397051 19058448 (   240)     48

    NDam Nonp      55692 (   0)     54550 (   0)     1142 3207936 (     0)   2809

    NAI0 Nonp    8524200 ( 162)   8523805 ( 158)      395 2969816 (  1120)   7518

    LSwi Nonp          1 (   0)         0 (   0)        1 2576384 (     0) 2576384

    MmCm Nonp         72 (   0)        33 (   0)       39 1733016 (     0)  44436

    TCPt Nonp    2803864 (  51)   2803817 (  51)       47 1459936 (     0)  31062

    Io   Nonp   25209203 ( 433)  25208578 ( 438)      625 1308264 (-40960)   2093

    TPLA Nonp        256 (   0)         0 (   0)      256 1048576 (     0)   4096

    Thre Nonp     211269 (   0)    209810 (   5)     1459  910416 ( -3120)    624

    Even Nonp   31492660 ( 184)  31476918 ( 195)    15742  856176 (  -528)     54

    Irp  Nonp     847400 (  10)    845278 (  18)     2122  785320 ( -2144)    370

    VadS Nonp   20987784 ( 103)  20964716 ( 108)    23068  738176 (  -160)     32

    Ddk  Nonp       4002 (   0)       352 (   0)     3650  583944 (     0)    159

    LSwr Nonp        128 (   0)         0 (   0)      128  416768 (     0)   3256

    Mdl  Nonp    2533985 ( 100)   2530766 ( 101)     3219  413552 (  -128)    128

    Pool Nonp          5 (   0)         2 (   0)        3  348160 (     0) 116053

    AfdC Nonp     499106 (   4)    496991 (   2)     2115  338400 (   320)    160

    Mm   Nonp        440 (   0)       425 (   0)       15  306040 (     0)  20402

    Vad  Nonp    6256579 (  24)   6250357 (  24)     6222  298656 (     0)     48

    NDpp Nonp        162 (   0)        73 (   0)       89  292256 (     0)   3283

    Sema Nonp    6016269 (  24)   6011717 (  24)     4552  255376 (     0)     56

    Devi Nonp       6014 (   0)      5681 (   0)      333  246168 (     0)    739

    TCPp Nonp         59 (   0)         0 (   0)       59  241664 (     0)   4096

    Stat Nonp     485622 (   5)    484248 (  10)     1374  230832 (  -840)    168

    Dump Nonp          6 (   0)         0 (   0)        6  229776 (     0)  38296

    TCPC Nonp      67410 (   0)     64981 (   0)     2429  206568 (     0)     85

    None Nonp         32 (   0)         0 (   0)       32  200448 (     0)   6264

    Hal  Nonp    5870290 ( 258)   5870279 ( 258)       11  197616 (   -72)  17965

    Ntf0 Nonp          3 (   0)         0 (   0)        3  196608 (     0)  65536

    AfdE Nonp     928161 (   9)    927472 (   7)      689  192920 (   560)    280

    MmCa Nonp    3702374 (  34)   3700523 (  34)     1851  185600 (     0)    100

    Ntfr Nonp      25528 (   0)     22754 (   0)     2774  178504 (     0)     64

    brcm Nonp         24 (   0)        12 (   0)       12  164544 (     0)  13712

    usbp Nonp        195 (   0)       145 (   0)       50  158096 (     0)   3161

    RxHb Nonp        201 (   0)         0 (   0)      201  155976 (     0)    776

    MmCi Nonp       3198 (   0)      2545 (   0)      653  150432 (     0)    230

    RceT Nonp          1 (   0)         0 (   0)        1  131072 (     0) 131072

    CcSc Nonp    1238135 (  25)   1237777 (  25)      358  114560 (     0)    320

    UlCO Nonp       1536 (   0)      1355 (   0)      181  110048 (     0)    608

    UlLS Nonp       6073 (   2)      6059 (   0)       14  107144 ( 16384)   7653

    Vadl Nonp     223433 (   1)    221783 (   6)     1650  105600 (  -320)     64

    SeTd Nonp   18304474 ( 137)  18302931 ( 134)     1543   98752 (   192)     64

    TCPA Nonp     802441 (   8)    802246 (   6)      195   71760 (   736)    368

    Ntfi Nonp      85855 (   1)     85607 (   9)      248   67456 ( -2176)    272

    AmlH Nonp          1 (   0)         0 (   0)        1   65536 (     0)  65536

    NtFs Nonp     491804 (  31)    490239 (  37)     1565   65096 (  -240)     41

    TCPT Nonp      64573 (   0)     64431 (   0)      142   64816 (     0)    456

    Ntfn Nonp     312607 (  25)    311027 (  31)     1580   63792 (  -240)     40

    Bfpl Nonp         40 (   0)         0 (   0)       40   62400 (     0)   1560

    Info Nonp     114430 (   3)    114429 (   3)        1   61440 (     0)  61440

    Ica  Nonp      14278 (  38)     14208 (  37)       70   60672 (    56)    866

    ReTa Nonp      16290 (   2)     15062 (   0)     1228   53696 (    80)     43

    IpSI Nonp       1027 (   0)         0 (   0)     1027   53272 (     0)     51

    Fltr Nonp       1100 (   0)         0 (   0)     1100   52800 (     0)     48

    FSfm Nonp     250792 (   8)    249563 (  34)     1229   49160 ( -1040)     40

    UlFU Nonp      56802 (   0)     56788 (   0)       14   46200 (     0)   3300

    AfdB Nonp     160933 (   0)    160873 (   0)       60   44464 (     0)    741

    Port Nonp    3335726 (  50)   3334424 (  47)     1302   41664 (    96)     32

    CcVa Nonp          1 (   0)         0 (   0)        1   40960 (     0)  40960

    Muta Nonp      36305 (   0)     35770 (   0)      535   39328 (     0)     73

    CMpa Nonp      13280 (   2)     12686 (   1)      594   38016 (    64)     64

    Proc Nonp        350 (   0)       294 (   0)       56   37184 (     0)    664

    Dnod Nonp        291 (   0)       168 (   0)      123   36408 (     0)    296

    NtFL Nonp     204599 (   5)    204591 (   5)        8   34768 (     0)   4346

    PooL Nonp          4 (   0)         0 (   0)        4   32768 (     0)   8192

    AmlC Nonp         72 (   0)        68 (   0)        4   32768 (     0)   8192

    UlOT Nonp          8 (   0)         0 (   0)        8   32768 (     0)   4096

    VoSb Nonp       8601 (   0)      8599 (   0)        2   32768 (     0)  16384

    Lfsr Nonp          2 (   0)         0 (   0)        2   32768 (     0)  16384

    WmiG Nonp       1192 (   0)      1054 (   0)      138   30912 (     0)    224

    RxNr Nonp       2537 (   0)      2530 (   0)        7   26888 (     0)   3841

    LScn Nonp         72 (   0)        35 (   0)       37   26344 (     0)    712

    Driv Nonp        112 (   0)        11 (   0)      101   25392 (     0)    251

    CPnp Nonp      10896 (   0)     10799 (   0)       97   24832 (     0)    256

    Ioin Nonp         16 (   0)         5 (   0)       11   22880 (     0)   2080

    ReEv Nonp      16677 (   1)     15786 (   2)      891   21384 (   -24)     24

    Qphf Nonp        236 (   0)       232 (   0)        4   21336 (     0)   5334

    Evnt Nonp         20 (   0)         0 (   0)       20   21120 (     0)   1056

    SMBi Nonp        105 (   0)        91 (   0)       14   20752 (     0)   1482

    TCPB Nonp      14187 (   0)     14173 (   0)       14   20696 (     0)   1478

    AcpD Nonp        187 (   0)       128 (   0)       59   20120 (     0)    341

    ReSe Nonp      17447 (   1)     16828 (   0)      619   19808 (    32)     32

    UlIR Nonp      48743 (   1)     48730 (   1)       13   19560 (     0)   1504

    CM44 Nonp      13284 (   2)     12687 (   1)      597   19080 (    32)     31

    Time Nonp       1482 (   0)      1379 (   0)      103   19000 (     0)    184

    CcVl Nonp         90 (   0)        64 (   0)       26   18336 (     0)    705

    UlHR Nonp       3774 (   0)      3760 (   0)       14   17808 (     0)   1272

    RxNf Nonp      35235 (   0)     35140 (   0)       95   16808 (     0)    176

    MmPd Nonp          1 (   0)         0 (   0)        1   16384 (     0)  16384

    TChi Nonp          4 (   0)         0 (   0)        4   16384 (     0)   4096

    TChu Nonp          4 (   0)         0 (   0)        4   16384 (     0)   4096

    TChe Nonp          4 (   0)         0 (   0)        4   16384 (     0)   4096

    TCht Nonp       2929 (   0)      2925 (   0)        4   16384 (     0)   4096

    MmSP Nonp          2 (   0)         0 (   0)        2   16384 (     0)   8192

    TChc Nonp          4 (   0)         0 (   0)        4   16384 (     0)   4096

    CcBc Nonp    3245000 (  74)   3244881 (  28)      119   16184 (  6256)    136

    Nb07 Nonp         73 (   0)        22 (   0)       51   15504 (     0)    304

    Wait Nonp    3942538 (  65)   3942484 (  65)       54   15368 (   -48)    284

    Gsem Nonp       1137 (   0)       947 (   0)      190   15136 (     0)     79

    SmMm Nonp       6305 (   0)      6284 (   0)       21   14776 (     0)    703

    Nbtl Nonp      13271 (   0)     13169 (   0)      102   14688 (     0)    144

    ObjT Nonp         31 (   0)         0 (   0)       31   14384 (     0)    464

    NDw1 Nonp         16 (   0)         6 (   0)       10   14328 (     0)   1432

    NBSf Nonp         69 (   0)        52 (   0)       17   13176 (     0)    775

    PciB Nonp         62 (   0)        46 (   0)       16   12776 (     0)    798

    PSC3 Nonp        294 (   0)       258 (   0)       36   11848 (     0)    329

    MmLd Nonp        136 (   0)        33 (   0)      103   11760 (     0)    114

    IoCo Nonp       1166 (   0)      1003 (   0)      163   11736 (     0)     72

    NpFc Nonp       4293 (   0)      4159 (   2)      134   10720 (  -160)     80

    WmiR Nonp        576 (   0)       413 (   0)      163   10432 (     0)     64

    IdeP Nonp      19785 (   0)     19750 (   0)       35   10112 (     0)    288

    RxIr Nonp     199440 (   0)    199429 (   0)       11    9944 (     0)    904

    NaTa Nonp         37 (   0)         0 (   0)       37    9768 (     0)    264

    SdHt Nonp         37 (   0)         0 (   0)       37    9768 (     0)    264

    UlRP Nonp      67865 (   0)     67849 (   0)       16    9352 (     0)    584

    RxMs Nonp       1200 (   0)      1198 (   0)        2    9288 (     0)   4644

      rx Nonp          3 (   0)         0 (   0)        3    9024 (     0)   3008

    CcPc Nonp     480765 (   4)    480677 (   4)       88    8448 (     0)     96

    ScFt Nonp          8 (   0)         1 (   0)        7    8432 (     0)   1204

    FMfr Nonp          1 (   0)         0 (   0)        1    8192 (     0)   8192

    Usfd Nonp          1 (   0)         0 (   0)        1    8192 (     0)   8192

    AfdL Nonp     798692 (  34)    798489 (  29)      203    8120 (   200)     40

    AfdF Nonp      11799 (   0)     11787 (   0)       12    7968 (     0)    664

    NbL2 Nonp       4033 (   2)      3990 (   1)       43    7912 (   184)    184

    NDpf Nonp         41 (   0)        15 (   0)       26    7904 (     0)    304

    SePa Nonp    1275879 (   0)   1275824 (   0)       55    7680 (     0)    139

    NDdb Nonp     287989 (   0)    287972 (   2)       17    7472 ( -1088)    439

    NCRE Nonp        115 (   0)         0 (   0)      115    7360 (     0)     64

    UlCK Nonp      57635 (   0)     57624 (   0)       11    7288 (     0)    662

    NbL0 Nonp       3327 (   3)      3288 (   3)       39    7176 (     0)    184

    LStb Nonp         91 (   0)        51 (   0)       40    6848 (     0)    171

    NbtD Nonp       4033 (   2)      3994 (   1)       39    6552 (   168)    168

    TCPr Nonp     290238 (   5)    290159 (   6)       79    6136 (   -72)     77

    FcbL Nonp       6796 (   0)      6702 (   0)       94    6016 (     0)     64

    ScPC Nonp          3 (   0)         0 (   0)        3    5928 (     0)   1976

    IpTI Nonp          3 (   0)         0 (   0)        3    5400 (     0)   1800

    Ksec Nonp          6 (   0)         0 (   0)        6    5352 (     0)    892

    MmWe Nonp         31 (   0)         0 (   0)       31    5208 (     0)    168

    CcVp Nonp     149014 (   4)    148989 (   4)       25    5064 (     0)    202

    Ntft Nonp         45 (   0)        31 (   0)       14    4992 (     0)    356

    MuoC Nonp          3 (   0)         0 (   0)        3    4944 (     0)   1648

    UlBS Nonp        684 (   0)       671 (   0)       13    4872 (     0)    374

    MmSd Nonp       1994 (   0)      1908 (   0)       86    4816 (     0)     56

    LSwq Nonp          2 (   0)         0 (   0)        2    4704 (     0)   2352

    TSRM Nonp         17 (   0)         3 (   0)       14    4704 (     0)    336

    TCIt Nonp          6 (   0)         3 (   0)        3    4552 (     0)   1517

    NpFn Nonp        229 (   0)       150 (   0)       79    4544 (     0)     57

    Wmip Nonp        247 (   0)        59 (   0)      188    4512 (     0)     24

      Ip Nonp          4 (   0)         0 (   0)        4    4416 (     0)   1104

    TD   Nonp        305 (   0)       282 (   0)       23    4376 (     0)    190

    NDmo Nonp         25 (   0)         9 (   0)       16    4360 (     0)    272

    TCiA Nonp    1349672 (  16)   1349623 (  14)       49    4312 (   176)     88

    FMpl Nonp          2 (   0)         0 (   0)        2    4232 (     0)   2116

    NBSl Nonp         16 (   0)        14 (   0)        2    4224 (     0)   2112

    VoSt Nonp       2777 (   0)      2742 (   0)       35    4200 (     0)    120

    PlBd Nonp          2 (   0)         0 (   0)        2    4192 (     0)   2096

    NDPt Nonp          2 (   0)         0 (   0)        2    4176 (     0)   2088

    NDPw Nonp          2 (   0)         0 (   0)        2    4176 (     0)   2088

    MmHt Nonp         13 (   0)        11 (   0)        2    4112 (     0)   2056

    Wmit Nonp          1 (   0)         0 (   0)        1    4096 (     0)   4096

    FMvo Nonp       4386 (   0)      4382 (   0)        4    4056 (     0)   1014

    L2T4 Nonp          1 (   0)         0 (   0)        1    4016 (     0)   4016

    PTPc Nonp          1 (   0)         0 (   0)        1    4008 (     0)   4008

    NbL1 Nonp       1860 (   0)      1839 (   0)       21    3864 (     0)    184

    UlCJ Nonp         40 (   0)        20 (   0)       20    3680 (     0)    184

    AfdP Nonp    2943719 ( 116)   2943699 ( 116)       20    3656 (     0)    182

    AcpR Nonp          9 (   0)         6 (   0)        3    3536 (     0)   1178

    NDan Nonp         45 (   0)        20 (   0)       25    3504 (     0)    140

    TOBJ Nonp         48 (   0)        40 (   0)        8    3440 (     0)    430

    Adap Nonp         13 (   0)         3 (   0)       10    3416 (     0)    341

    Mdp  Nonp          8 (   0)         0 (   0)        8    3072 (     0)    384

    HidC Nonp         35 (   0)        10 (   0)       25    2824 (     0)    112

    NpFr Nonp    1993329 (  17)   1993260 (  17)       69    2760 (     0)     40

    FSrm Nonp       6115 (   2)      6094 (   2)       21    2688 (     0)    128

    MntA Nonp         11 (   0)         5 (   0)        6    2688 (     0)    448

    RxVn Nonp       2547 (   1)      2536 (   0)       11    2560 (   224)    232

    MmDm Nonp         32 (   0)         0 (   0)       32    2560 (     0)     80

    MmIn Nonp       2759 (   0)      2743 (   0)       16    2560 (     0)    160

    KbdC Nonp          3 (   0)         0 (   0)        3    2544 (     0)    848

    Desk Nonp         47 (   0)        34 (   0)       13    2496 (     0)    192

    FLex Nonp      10144 (   0)     10106 (   0)       38    2432 (     0)     64

    is there something i missed ?

  93. David.Wang says:

    DIALOGsoftware – a large number for FILE usually indicates that some user-mode process has a large number of handles. Find it and fix it.

    Likewise for TCPc – large number of sockets being "leaked" by something running on your system.

    You will have these problems until you fix the software that is leaking the resources.

    //David

  94. DIALOGSoftware says:

    Thanks for this help.

    I’ll try to find this problems and will inform you if i’ve found the soft that is causing me these troubles.

  95. Ray says:

    David,

    Thanks for taking the time to write this article.  It is very helpful.  I come seeking advice on whether my logic in dealing with a problem similar to this is correct.

    We run a relatively high-traffic web-server using IIS6 on Server 2003 Web Edition in a dual-CPU / 1GB RAM configuration.  One site on this machine hosts thousands of small images.  We have been experiencing the Connections_Refused problem in the web-site in IIS that serves these images.  Using poolmon brings the Muta driver all the way to the top in terms of Bytes allocated in NPP memory space:

    Memory: 1068492K Avail:  524016K  PageFlts: 58558   InRam Krnl: 1796K P:55752K

    Commit: 620868K Limit:2568348K Peak:2179800K            Pool N:97584K P:91684K

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

    Muta Nonp  747242711 (1135) 745941065 (   1)  1301646 83307168 ( 72576)     64

    Vmem Paged       262 (   0)       121 (   0)      141 47570944 (     0) 337382

    Obtb Paged   2093054 (   2)   2090310 (   0)     2744 10894104 (  8192)   3970

    MmSt Paged   1378399 (   4)   1375142 (   3)     3257 5706488 (  2056)   1752

    UlHT Paged         1 (   0)         0 (   0)        1 4198400 (     0) 4198400

    LSwi Nonp          1 (   0)         0 (   0)        1 2576384 (     0) 2576384

    Gh15 Paged   3946268 (  53)   3945941 (  53)      327 1453184 (     0)   4443

    Running a search to identify the driver name reveals ntfs.sys:

    C:WINDOWSsystem32drivers>findstr /m /l  Muta  *.sys

    ntfs.sys

    My theory: This NTFS driver becomes exhausted by the thousands of requests for small (25-100kb) images it receives daily.  The site contains 70k+ images.  

    I also checked fragmentation on the volume and it reports 23% file fragmentation — we have a scheduled task to defrag this disk but I’m sure it is forced to skip over many locked files each time it runs.

    Does this sound reasonable?  I’m afraid the next step will be to load-balance the image-store across multiple IIS servers.

    Thanks in advance for any insight into this issue you may have,

    Ray

  96. PaulJ says:

    Thanks to everyone for their input and observations on these issues.  I have been fighting stability issues with Websense 6.3.2 (a web filtering system) in a Windows 2003 Server environment for months. We would end up with a system that could not make ANY kind of network connections due to the exhausted Nonpaged pool.

    After getting a pointer toward ‘tdLL’ as a memory leak source from Websense Support, I found this article and the collective experiences and ideas.

    It appears that SAV9ce was a significant part of our issues, and after upgrading to Symantec Endpoint Protection 11 (11.0.3001.2224 specifically) our servers appear to be much better behaved.

    Huge thanks to David and all of the thread contributors!

  97. vish says:

    Tag  Type     Allocs         Frees    Diff   Bytes    Per Alloc

    File Nonp  920718825 920584429    134396 20523872        152        

    Even Nonp    9561711   9440980    120731 5798032         48        

    TCPc Nonp    2604286   2494258    110028 5281344         48        

    Ntfr Nonp     843921    809887     34034 2179144         64        

    Mdl  Nonp    4991145   4958701     32444 4154432        128        

    NtFs Nonp  477786384 477765570     20814  835056         40        

    Ntfn Nonp     876015    855203     20812  834552         40        

    MmCa Nonp    1315376   1298180     17196 1916240        111

    how to find where is mem leaks from first line

    File Nonp  920718825 920584429    134396 20523872        152        

    which driver is causing issue ?

    Vish

  98. David.Wang says:

    Vish – I suggest reading the collective knowledge from earlier in this thread for information on how to diagnose the "File" tag.

    //David

  99. C Sanders says:

    David – You’re my hero!

    IIS on one of our Exchange servers was refusing connections.  Turned out to be Exchange Public Folder replication NPP memory leak.  Your How-To worked like a charm in diagnosing our root cause.

    Thanks!

  100. yang lei says:

    http://blogs.technet.com/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx

    according to the url above,

    the size of non-paged pool depends on the amount of physical memory present in the system.

  101. Leigh Treacy says:

    David Wang you saved my life!!

    We had webmail going down every 6 days and I couldn’t for the life of me work out what was going on, there was nothing in the event viewer, just connections_refused in the httperr.log.

    Poolmon showed the "File" tag as the culprit and the further suggestion by david to turn on "handles" in task manager under the process tab, showed a process called DEVManRA.exe with over 16000.  I killed the process and viola, no more memory leak!

    The process is associated with HP Power Manager, UPS management software, I’ve removed it from the exchange server and things are back to normal.

    Thanks again David!!

    Leigh

  102. vishal patel says:

    We are facing same problem from last 6 months, i have one doubt that after we have installed addon of url-rewrite this thing is happening, Let me know if anybody else has used that and if that could be cause of problem.

  103. Eric says:

    Thanks David!  My web server was using 107mb of 127mb due to SavE consuming 42mb.  As soon as i uinstalled symantec and restarted IIS the problem was cleared.  It has the 3gb switch and has for over a year now, so I’m not sure what finally triggered it after it being fine for so long.

  104. john says:

    Dave,

    i got MPIO in my poolmon output. MPIO is MS iSCSI initiator, boucing the service didn’t help resolve the problem. still not able to see the web page..please advise

    MPIO Nonp    6088888 (  12)        31 (   0)  6088857 170516336 (   336)     28 [mpio]

    MmCm Nonp       2209 (   0)      2045 (   0)      164 9061648 (     0)  55253 Unknown Driver

    AfdB Nonp    5460432 (  43)   5441960 (  43)    18472 8881344 (     0)    480 [afd]

    tmte Nonp       1949 (   0)         0 (   0)     1949 7983104 (     0)   4096 [tmtdi]

    TCPB Nonp     459974 (   0)    412860 (   0)    47114 4966880 (     0)    105 Unknown Driver

    File Nonp   93043247 (2752)  93012719 (2752)    30528 4643120 (     0)    152 Unknown Driver

    AfdC Nonp     222392 (   0)    195975 (   0)    26417 4226720 (     0)    160 [afd]

    Wdm  Nonp          2 (   0)         0 (   0)        2 3420160 (     0) 1710080 [ati2mtag][hid

    TCPC Nonp      73028 (   0)     46348 (   0)    26680 2246032 (     0)     84 [tcpip]

    TCPt Nonp    3357329 (   0)   3357294 (   0)       35 1458688 (     0)  41676 [tcpip]

    Mm   Nonp        139 (   0)       114 (   0)       25 1369536 (     0)  54781 Unknown Driver

    TCPc Nonp     526016 (   1)    499134 (   1)    26882 1290336 (     0)     48 [tcpip]

    LSwi Nonp          2 (   0)         1 (   0)        1 1290240 (     0) 1290240 Unknown Driver

  105. Junior Lazuardi says:

    Now I have this problem.

    In my case, I found that server restart is not that necessary. Recycling (manually) the application pool, and then restarting the WWW service should work.

  106. Martin Neosys says:

    Excellent post Dr. Wang

    I had that problem since we took over the support for this W2003 server. I did set a maintenance reboot every week and solved the problem. But now it is IIS6.0 that behaves and refuses connections.

    So my homework brought me to these NPP results :

    Memory: 3667824K Avail:  732136K  PageFlts:   793   InRam Krnl: 2980K P:48752K

    Commit:4620184K Limit:7194488K Peak:4885544K            Pool N:124028K P:50728K

    System pool information

    Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

    File Nonp   31021512 ( 169)  30681043 ( 160)   340469 51754600 (  1368)    152

    Even Nonp   20775711 ( 162)  20349506 ( 155)   426205 20464880 (   336)     48

    MmCm Nonp     276399 (   0)    275249 (   0)     1150 16413864 (     0)  14272

    R100 Nonp         41 (   0)         2 (   0)       39 9683040 (     0) 248283

    Obtb Paged      5021 (   0)      3294 (   0)     1727 6716368 (     0)   3889

    R100 Paged        47 (   0)         2 (   0)       45 5461800 (     0) 121373

    I found that both File, Even and R100 were linked to ATI related *.sys files.

    (when running "findstr /m /l File *.sys" under c:windowssystem32drivers )

    Then checked the driver used by this ATI Video card driver, it was dated from 2006, and upgraded this ATI ES1000 driver and….

  107. Brian says:

    Further research turned up the answer, to the question in my previous comment, here on Mark Russinovich's blog:

    blogs.technet.com/…/3283844.aspx

    FYI,

    (] B [)