Security Token Handle Leak in ASP.NET Application


 


We got several similar cases in the past two months about security token leak in customer’s ASP.NET application. All these cases are related to Impersonate. Usually, user will noticed the whole server performance is down, and may found event 2020 in their system event log.


Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2020
Date: 12/24/2008
Time: 12:13:31 AM
User: N/A
Computer: MYW2K3
Description:
The server was unable to allocate from the system paged pool because the pool was empty.


If we captured a kernel dump and look at the paged pool, most of them are tokens. And process w3wp.exe occupies tons of such tokens.


0: kd> !token ea698028
_TOKEN ea698028
TS Session ID: 0
User: S-1-5-21-606747145-527237240-1605580848-740255


 From task manager, you could see handle count for w3wp.exe and lsass.exe is keep growing. Usually, it should be normal if your application has 2,000~3,000 handles, but it may abnormal if it reached 10,000.



How to Troubleshooting:


We have a great tool (IIS Diagnostics 1.1) to troubleshooting unmanaged handle leaking; unfortunately, it doesn’t work well with managed world. You may get follow report with Debugdiag for ASP.NET application. You can see the function information is incorrect and can’t help us find out the function which is leaking handles.


Module details for Unknown module - Base address : 0x00000000










Module Name


  Unknown module - Base address : 0x00000000


Handle Count


  2165 handle(s)


 


Top 3 functions by handle count















0x0e491718



722 handle(s)


0x01f8aa82


 


722 handle(s)


0x0e491662


 


721 handle(s)


 


Handle type statistics by handle count








Security Token handles

 

2,165 handle(s)

 Function details















Function


  0x0e491718


Allocation type


  Security Token handle(s)


Handle Count


  722 handle(s)


Leak Probability


  88%


 
















Function


  0x01f8aa82


Allocation type


  Security Token handle(s)


Handle Count


  722 handle(s)


Leak Probability


  88%


 
















Function


  0x0e491662


Allocation type


  Security Token handle(s)


Handle Count


  721 handle(s)


Leak Probability


  88%


 


In fact, the address recorded by Debugdiag is not the function call, but a CLRSTUB for native code. So, unless the leaking functions is executing when collecting dump, the leaking address (es) reported by DebugDiag can’t help.


 


Here is an example for the assembly of CLRSTUB


0:000> u 0x0e491662


CLRStub[StubLinkStub]@e491662:


0e491662 c6430801        mov     byte ptr [ebx+8],1


CLRStub[StubLinkStub]@e491666:


0e491666 833de0d23a7a00  cmp     dword ptr [mscorwks!g_TrapReturningThreads (7a3ad2e0)],0


CLRStub[StubLinkStub]@e49166d:


0e49166d 7527            jne     CLRStub[StubLinkStub]@e491696 (0e491696)


CLRStub[StubLinkStub]@e49166f:


0e49166f 50              push    eax


CLRStub[StubLinkStub]@e491670:


0e491670 52              push    edx


CLRStub[StubLinkStub]@e491671:


0e491671 56              push    esi


CLRStub[StubLinkStub]@e491672:


0e491672 e861f49e6b      call    mscorwks!CleanupWrapper (79e80ad8)


CLRStub[StubLinkStub]@e491677:


0e491677 5a              pop     edx


 


There are two “stupid” ways may help you find out the impersonate code if you don’t have the source code.


1.      Search related functions in the dump file.


To implement Impersonate, we must call Windows APIs like  LogonUser, DuplicateToken. So, we can search the APIs in memory and found which assemblies referenced these APIs. Then we can review these codes using reflector to make tokens are closed properly.


0:018> s-a 0 L?7fffffff "LogonUserA"


0e752a90  4c 6f 67 6f 6e 55 73 65-72 41 00 44 75 70 6c 69  LogonUserA.Dupli


0e762a90  4c 6f 67 6f 6e 55 73 65-72 41 00 44 75 70 6c 69  LogonUserA.Dupli


77f747e4  4c 6f 67 6f 6e 55 73 65-72 41 00 4c 6f 67 6f 6e  LogonUserA.Logon


0:018> !address 0e752a90 


0e750000 : 0e752000 - 00002000


                    Type     01000000 MEM_IMAGE


                    Protect  00000020 PAGE_EXECUTE_READ


                    State    00001000 MEM_COMMIT


                    Usage    RegionUsageImage


FullPath C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\securityhandleleak\91744b5d\ab584e94\App_Web_oeootcha.dll


 


If search command returned a lot of results, this script command can help you.


0:000> .foreach (place { s-[1]a 0 L?7fffffff "LogonUser" }) {!address place}


    03530000 : 03532000 - 0000e000


                    Type     01000000 MEM_IMAGE


                    Protect  00000020 PAGE_EXECUTE_READ


                    State    00001000 MEM_COMMIT


                    Usage    RegionUsageImage


                    FullPath C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\erpapv\be46890c\b4f7eeae\assembly\dl3\83aa2552\f8a67fd5_af8fc901\Assembly1.DLL


    03590000 : 03592000 - 0000e000


                    Type     01000000 MEM_IMAGE


                    Protect  00000020 PAGE_EXECUTE_READ


                    State    00001000 MEM_COMMIT


                    Usage    RegionUsageIsVAD


    035b0000 : 035b2000 - 00011000


                    Type     01000000 MEM_IMAGE


                    Protect  00000020 PAGE_EXECUTE_READ


                    State    00001000 MEM_COMMIT


                    Usage    RegionUsageImage


                    FullPath C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\erpapv\be46890c\b4f7eeae\assembly\dl3\73e6c81e\fcbd4e76_6f96c901\Approve.DLL


    03700000 : 03702000 - 00032000


                    Type     01000000 MEM_IMAGE


                    Protect  00000020 PAGE_EXECUTE_READ


                    State    00001000 MEM_COMMIT


                    Usage    RegionUsageImage


                    FullPath C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\erpapv\be46890c\b4f7eeae\assembly\dl3\19c4bca1\9a54ebdc_5a97c901\Assembly2.DLL


……………..


 


2.      Using Debuggers to print a stack log each time LogonUser API get called.


To do this:


-          Save below content to a file like TokenLeak.cfg, modify the OutputDir if you prefer another foldee


-          Open a command window and go to the debug tools installation folder, run this command:


-          Cscript.exe –p <PID of the w3wp.exe has token leak problem> -c <FullPathTo TokenLeak.cfg>


-          Monitor the handle count in task manager and check the log file if you see handle count has been increased since the debugger start.


-          You may found the stack like this in log files, review such code.


# ChildEBP RetAddr  Args to Child             


00 01f1ef08 0e49126a 01f1f128 01f1f028 01f1ef28 ADVAPI32!LogonUserA (FPO: [Non-Fpo]) (CONV: stdcall)


01 01f1f244 0e790605 00000001 00000000 060e3178 CLRStub[StubLinkStub]@e49126a


02 01f1f2a4 0e790505 00000000 00000000 00000000 App_Web_oeootcha!_Default.impersonateValidUser(System.String, System.String, System.String)+0x7d*** WARNING: Unable to verify checksum for C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\securityhandleleak\91744b5d\ab584e94\App_Web_oeootcha.dll


 (Managed)


03 00000000 66f12980 00000000 00000000 00000000 App_Web_oeootcha!_Default.Page_Load(System.Object, System.EventArgs)+0x3d (Managed)


04 01f1f504 6628efd2 00000000 00000000 00000000


…….


Configuration File:


<ADPlus>


      <Settings>


                      <RunMode> CRASH </RunMode>


                      <Sympath> http://msdl.microsoft.com/download/symbols </Sympath>


                      <Option> Quiet </Option>


                      <OutputDir> d:\HandleLeak </OutputDir>


      </Settings>


      <Breakpoints>


                      <NewBP>


                                      <Address> advapi32!LogonUserA </Address>


                                      <Type> BP </Type>


                                      <Actions> stack  </Actions>


                                      <CustomActions>  </CustomActions>


                                      <ReturnAction> G </ReturnAction>


                      </NewBP> 


 


                      <NewBP>


                                      <Address> advapi32!LogonUserW </Address>


                                       <Type> BP </Type>


                                       <Actions> stack  </Actions>


                                        <CustomActions>  </CustomActions>


                                        <ReturnAction> G </ReturnAction>


                        </NewBP> 


      </Breakpoints>


</ADPlus>


 


Summary:


Handle is a very expensive and limited resource. Leaking handles can affect the whole server performance not only IIS itself, and maybe unexpected shutdown. Before using any API related to handle, read the MSDN document carefully. MSDN will tell us whether we need to release the handle or not.


 


Regards,


 


Wei Zhao

Comments (1)

  1. AtulGupta says:

    Very useful. Thanks Wei. BTW, is there any best practice that one should follow to avoid such leaks? and also for the sample you used above, what was in the code that caused these leaks?

Skip to main content