Security Token Handle Leak in ASP.NET Application
We got several similar cases in the past two months about security token leak in customer’s ASP.NET application. All these cases are related to Impersonate. Usually, user will noticed the whole server performance is down, and may found event 2020 in their system event log.
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2020
Date: 12/24/2008
Time: 12:13:31 AM
User: N/A
Computer: MYW2K3
Description:
The server was unable to allocate from the system paged pool because the pool was empty.
If we captured a kernel dump and look at the paged pool, most of them are tokens. And process w3wp.exe occupies tons of such tokens.
0: kd> !token ea698028
_TOKEN ea698028
TS Session ID: 0
User: S-1-5-21-606747145-527237240-1605580848-740255
From task manager, you could see handle count for w3wp.exe and lsass.exe is keep growing. Usually, it should be normal if your application has 2,000~3,000 handles, but it may abnormal if it reached 10,000.
How to Troubleshooting:
We have a great tool (IIS Diagnostics 1.1) to troubleshooting unmanaged handle leaking; unfortunately, it doesn’t work well with managed world. You may get follow report with Debugdiag for ASP.NET application. You can see the function information is incorrect and can’t help us find out the function which is leaking handles.
Module details for Unknown module - Base address : 0x00000000
|
|
|
|
|
|
Top 3 functions by handle count
|
|
| |
|
|
| |
|
|
|
Handle type statistics by handle count
* *
Function details
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In fact, the address recorded by Debugdiag is not the function call, but a CLRSTUB for native code. So, unless the leaking functions is executing when collecting dump, the leaking address (es) reported by DebugDiag can’t help.
Here is an example for the assembly of CLRSTUB
0:000> u 0x0e491662
CLRStub[StubLinkStub]@e491662:
0e491662 c6430801 mov byte ptr [ebx+8],1
CLRStub[StubLinkStub]@e491666:
0e491666 833de0d23a7a00 cmp dword ptr [mscorwks!g_TrapReturningThreads (7a3ad2e0)],0
CLRStub[StubLinkStub]@e49166d:
0e49166d 7527 jne CLRStub[StubLinkStub]@e491696 (0e491696)
CLRStub[StubLinkStub]@e49166f:
0e49166f 50 push eax
CLRStub[StubLinkStub]@e491670:
0e491670 52 push edx
CLRStub[StubLinkStub]@e491671:
0e491671 56 push esi
CLRStub[StubLinkStub]@e491672:
0e491672 e861f49e6b call mscorwks!CleanupWrapper (79e80ad8)
CLRStub[StubLinkStub]@e491677:
0e491677 5a pop edx
There are two “stupid” ways may help you find out the impersonate code if you don’t have the source code.
1. Search related functions in the dump file.
To implement Impersonate, we must call Windows APIs like LogonUser, DuplicateToken. So, we can search the APIs in memory and found which assemblies referenced these APIs. Then we can review these codes using reflector to make tokens are closed properly.
0:018> s-a 0 L?7fffffff "LogonUserA"
0e752a90 4c 6f 67 6f 6e 55 73 65-72 41 00 44 75 70 6c 69 LogonUserA.Dupli
0e762a90 4c 6f 67 6f 6e 55 73 65-72 41 00 44 75 70 6c 69 LogonUserA.Dupli
77f747e4 4c 6f 67 6f 6e 55 73 65-72 41 00 4c 6f 67 6f 6e LogonUserA.Logon
0:018> !address 0e752a90
0e750000 : 0e752000 - 00002000
Type 01000000 MEM_IMAGE
Protect 00000020 PAGE_EXECUTE_READ
State 00001000 MEM_COMMIT
Usage RegionUsageImage
FullPath C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\securityhandleleak\91744b5d\ab584e94\App_Web_oeootcha.dll
If search command returned a lot of results, this script command can help you.
0:000> .foreach (place { s-[1]a 0 L?7fffffff "LogonUser" }) {!address place}
03530000 : 03532000 - 0000e000
Type 01000000 MEM_IMAGE
Protect 00000020 PAGE_EXECUTE_READ
State 00001000 MEM_COMMIT
Usage RegionUsageImage
FullPath C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\erpapv\be46890c\b4f7eeae\assembly\dl3\83aa2552\f8a67fd5_af8fc901\Assembly1.DLL
03590000 : 03592000 - 0000e000
Type 01000000 MEM_IMAGE
Protect 00000020 PAGE_EXECUTE_READ
State 00001000 MEM_COMMIT
Usage RegionUsageIsVAD
035b0000 : 035b2000 - 00011000
Type 01000000 MEM_IMAGE
Protect 00000020 PAGE_EXECUTE_READ
State 00001000 MEM_COMMIT
Usage RegionUsageImage
FullPath C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\erpapv\be46890c\b4f7eeae\assembly\dl3\73e6c81e\fcbd4e76_6f96c901\Approve.DLL
03700000 : 03702000 - 00032000
Type 01000000 MEM_IMAGE
Protect 00000020 PAGE_EXECUTE_READ
State 00001000 MEM_COMMIT
Usage RegionUsageImage
FullPath C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\erpapv\be46890c\b4f7eeae\assembly\dl3\19c4bca1\9a54ebdc_5a97c901\Assembly2.DLL
……………..
2. Using Debuggers to print a stack log each time LogonUser API get called.
To do this:
- Save below content to a file like TokenLeak.cfg, modify the OutputDir if you prefer another foldee
- Open a command window and go to the debug tools installation folder, run this command:
- Cscript.exe –p <PID of the w3wp.exe has token leak problem> -c <FullPathTo TokenLeak.cfg>
- Monitor the handle count in task manager and check the log file if you see handle count has been increased since the debugger start.
- You may found the stack like this in log files, review such code.
# ChildEBP RetAddr Args to Child
00 01f1ef08 0e49126a 01f1f128 01f1f028 01f1ef28 ADVAPI32!LogonUserA (FPO: [Non-Fpo]) (CONV: stdcall)
01 01f1f244 0e790605 00000001 00000000 060e3178 CLRStub[StubLinkStub]@e49126a
02 01f1f2a4 0e790505 00000000 00000000 00000000 App_Web_oeootcha!_Default.impersonateValidUser(System.String, System.String, System.String)+0x7d*** WARNING: Unable to verify checksum for C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\securityhandleleak\91744b5d\ab584e94\App_Web_oeootcha.dll
(Managed)
03 00000000 66f12980 00000000 00000000 00000000 App_Web_oeootcha!_Default.Page_Load(System.Object, System.EventArgs)+0x3d (Managed)
04 01f1f504 6628efd2 00000000 00000000 00000000
…….
Configuration File:
<ADPlus>
<Settings>
<RunMode> CRASH </RunMode>
<Sympath> https://msdl.microsoft.com/download/symbols </Sympath>
<Option> Quiet </Option>
<OutputDir> d:\HandleLeak </OutputDir>
</Settings>
<Breakpoints>
<NewBP>
<Address> advapi32!LogonUserA </Address>
<Type> BP </Type>
<Actions> stack </Actions>
<CustomActions> </CustomActions>
<ReturnAction> G </ReturnAction>
</NewBP>
<NewBP>
<Address> advapi32!LogonUserW </Address>
<Type> BP </Type>
<Actions> stack </Actions>
<CustomActions> </CustomActions>
<ReturnAction> G </ReturnAction>
</NewBP>
</Breakpoints>
</ADPlus>
Summary:
Handle is a very expensive and limited resource. Leaking handles can affect the whole server performance not only IIS itself, and maybe unexpected shutdown. Before using any API related to handle, read the MSDN document carefully. MSDN will tell us whether we need to release the handle or not.
Regards,
Wei Zhao