Some Multi-threaded terminal server applications crashes in Win 2008 R2


In Win2008 R2 some Multi-threaded terminal server application may crashes with access violation in the test  eax, eax cpu instruction with following symptoms. This issue is very intermittent.


1.         


2.       You may find following 2 threads,



  • Executing a test instruction and causing AV.

  • Trying to change the protection of exactly same code memory page which the first thread is causing access violation ( Not waiting ).

  • If DEP for the process is turned off application is working fine.

Following are the analysis details,


The stored exception information can be accessed via .ecxr.


(428.b50): Access violation – code c0000005 (first/second chance not available)


eax=00000000 ebx=7740f85c ecx=74e92dd9 edx=00000000 esi=00000000 edi=00000001


eip=741c17cd esp=0018a4dc ebp=0018a508 iopl=0         nv up ei pl nz na pe nc


cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206


mswsock!SockWaitForSingleObject+0x3a:


741c17cd 85c0            test    eax,eax


 


 


0:000> k100


ChildEBP RetAddr 


0018a508 741c678c mswsock!SockWaitForSingleObject+0x3a


0018a5f4 75f04a20 mswsock!WSPSelect+0x3a6


*** ERROR: Symbol file could not be found.  Defaulted to export symbols for DvZediRimac002.dll –


0018a674 4b6f447d ws2_32!select+0x494


WARNING: Stack unwind information not available. Following frames may be wrong.


0018e598 4b6e5a80 ThirdParty!ThirdParty +0xac


0018e6ac 774bfd6e ThirdParty!ThirdParty +0x2fc


0018e698 00000000 ntdll!RtlpValidateHeap+0x20


 


0:000> !analyze -v


*******************************************************************************


*                                                                             *


*                        Exception Analysis                                   *


*                                                                             *


*******************************************************************************


 


Debugger CompCtrlDb Connection::Open failed 80004005


Debugger CompCtrlDb Connection::Open failed 80004005


Debugger CompCtrlDb Connection::Open failed 80004005


Debugger Dbgportaldb Connection::Open failed 80040e4d


Database Dbgportaldb not connected


 


FAULTING_IP:


mswsock!SockWaitForSingleObject+3a


741c17cd 85c0            test    eax,eax


 


EXCEPTION_RECORD:  ffffffff — (.exr ffffffffffffffff)


ExceptionAddress: 741c17cd (mswsock!SockWaitForSingleObject+0x0000003a)


   ExceptionCode: c0000005 (Access violation)


  ExceptionFlags: 00000000


NumberParameters: 2


   Parameter[0]: 00000008


   Parameter[1]: 741c17cd


Attempt to execute non-executable address 741c17cd


 


PROCESS_NAME:  KAREWE.exe


 


ERROR_CODE: (NTSTATUS) 0xc0000005 – The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.


 


WRITE_ADDRESS:  741c17cd


 


BUGCHECK_STR:  ACCESS_VIOLATION


 


LAST_CONTROL_TRANSFER:  from 741c678c to 741c17cd


 


STACK_TEXT: 


0018a508 741c678c 0000025c 00000264 00000001 mswsock!SockWaitForSingleObject+0x3a


0018a5f4 75f04a20 00000000 0018a6a0 00000000 mswsock!WSPSelect+0x3a6


0018a674 4b6f447d 00000000 0018a6a0 00000000 ws2_32!select+0x494


WARNING: Stack unwind information not available. Following frames may be wrong.


0018e598 4b6e5a80 0018e6fc 00000200 00000001 ThirdParty!ThirdParty +0xac


0018e6ac 774bfd6e 00000000 00000044 00310150 ThirdParty!ThirdParty +0x2fc


0018e698 00000000 02c30ac8 23040027 0000000d ntdll!RtlpValidateHeap+0x20


 


 


STACK_COMMAND:  ~0s; .ecxr ; kb


 


PRIMARY_PROBLEM_CLASS:  SOFTWARE_NX_FAULT_FALSE_POSITIVE


 


DEFAULT_BUCKET_ID:  SOFTWARE_NX_FAULT_FALSE_POSITIVE


 


FAULTING_THREAD:  00000b50


 


FOLLOWUP_IP:


mswsock!SockWaitForSingleObject+3a


741c17cd 85c0            test    eax,eax


 


SYMBOL_STACK_INDEX:  0


 


FOLLOWUP_NAME:  wsstress


 


MODULE_NAME:  mswsock


 


DEBUG_FLR_IMAGE_TIMESTAMP:  4a5bda77


 


SYMBOL_NAME:  mswsock!SockWaitForSingleObject+3a


 


IMAGE_NAME:  mswsock.dll


 


FAILURE_BUCKET_ID:  ACCESS_VIOLATION_mswsock!SockWaitForSingleObject+3a


 


BUCKET_ID:  ACCESS_VIOLATION_mswsock!SockWaitForSingleObject+3a


 


Followup: wsstress


———


 


 


 


·         The application is failing at


741c17cd 85c0            test    eax,eax


 


due to access violation.


·         I checked if any dll is tampered with


0:000> !for_each_module !chkimg @#ModuleName


 


found that the dll under question(mswsock) is not tampered.



  • Above instruction (test) don’t access any memory location other than where EIP points.

  • So for the above instruction to generate an AV, the EIP should point to an address in a non executable page of memory.

  • Again from !address command we can see that this a code region and it is from a mapped dll which should ideally an executable page.

0:000> !address eip


 TEB 7efdd000 in range 7efdb000 7efde000


 TEB 7efda000 in range 7efd8000 7efdb000


 ProcessParametrs 003213f0 in range 00320000 0038f000


 Environment 00320810 in range 00320000 0038f000


    741c0000 : 741c1000 – 00001000


                    Type     01000000 MEM_IMAGE


                    Protect  00000004 PAGE_READWRITE


                    State    00001000 MEM_COMMIT


                    Usage    RegionUsageImage


                    FullPath C:\Windows\System32\mswsock.dll


 


On further debugging I found the following as well, which should be true in at least some cases of execution.


 


Only two threads in the application,


Ø  Executing the test instruction and causing  AV.


Ø  Trying to change the protection of exactly same code memory page which the first thread is causing access violation.


0:000> ~


.  0  Id: 428.b50 Suspend: 1 Teb: 7efdd000 Unfrozen


   1  Id: 428.900 Suspend: 1 Teb: 7efda000 Unfrozen


0:000> ~1s


eax=00000000 ebx=80000000 ecx=00000000 edx=00000000 esi=741c1098 edi=741c0000


eip=7740ffea esp=030afacc ebp=030afb08 iopl=0         nv up ei pl nz na po nc


cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202


ntdll!NtProtectVirtualMemory+0x12:


7740ffea 83c404          add     esp,4


 


0:001> k100


ChildEBP RetAddr  


030afacc 7416aa2c ntdll!NtProtectVirtualMemory+0x12


030afb08 7416ac22 TSAPPCMP!TsRedirectRegisteredImage+0xcf


030afb58 7416af58 TSAPPCMP!TsWalkProcessDlls+0xf5


030afb8c 4b740f0c TSAPPCMP!TLoadLibraryA+0x3a


WARNING: Stack unwind information not available. Following frames may be wrong.


030afbe8 4b73f737 ThirdParty!ThirdParty +0x38a9


030afc84 774236fa ThirdParty!ThirdParty +0x20d4


030afd78 4b6b83c3 ntdll!RtlpFreeHeap+0xb7a


030afde0 774236fa ThirdParty!ThirdParty +0x9b


030afddc 50000063 ntdll!RtlpFreeHeap+0xb7a


030afec4 774236fa 0x50000063


030affd4 77429d45 ntdll!RtlpFreeHeap+0xb7a


030affec 00000000 ntdll!_RtlUserThreadStart+0x1b


 


 


Looking at that stack based on following link and trying to sleuth the parameters, ( NtProtectVirtualMemory has FPO and is written in assembly so normal methods don’t work )


 


http://undocumented.ntinternals.net/UserMode/Undocumented%20Functions/Memory%20Management/Virtual%20Memory/NtProtectVirtualMemory.html


 


NTSYSAPI


NTSTATUS


NTAPI


NtProtectVirtualMemory(


 


  IN HANDLE               ProcessHandle,


  IN OUT PVOID            *BaseAddress,


  IN OUT PULONG           NumberOfBytesToProtect,


  IN ULONG                NewAccessProtection,


  OUT PULONG              OldAccessProtection );


 


Picking up the function arguments from stack


 


0:001> dds esp


030afacc  7740ffea ntdll!NtProtectVirtualMemory+0x12


030afad0  7416aa2c TSAPPCMP!TsRedirectRegisteredImage+0xcf


030afad4  ffffffff


030afad8  030afafc


030afadc  030afb00


030afae0  00000004


030afae4  030afaf8


030afae8  774f020c ntdll!PebLdr+0xc


030afaec  0038af48


030afaf0  0038be08


030afaf4  00000004


030afaf8  00000020


030afafc  741c1098 mswsock!_imp__NtOpenKey


030afb00  00000004


030afb04  741c0000 mswsock!_imp__OutputDebugStringA <PERF> (mswsock+0x0)


030afb08  030afb58


030afb0c  7416ac22 TSAPPCMP!TsWalkProcessDlls+0xf5


030afb10  741e4818 mswsock!DNSAPI_NULL_THUNK_DATA_DLB+0x8


030afb14  00000000


030afb18  00000001


030afb1c  3c144767


030afb20  00000000


030afb24  4b7a203c DvZediRimac002!CWatchedProxyObj::`vftable’+0x30280


030afb28  00000000


030afb2c  001a0018


030afb30  74162684 TSAPPCMP!`string’


030afb34  030afb54


030afb38  741c00d8 mswsock!_imp__OutputDebugStringA <PERF> (mswsock+0xd8)


030afb3c  0038be08


030afb40  030afb1c


030afb44  00000000


030afb48  030afb7c


 


 


Unassembleing the call to NtProtectVirtualMemory.


 


0:001> ub 7416aa2c L10


TSAPPCMP!TsRedirectRegisteredImage+0xaa [d:\w7rtm\termsrv\tsappcmp\register.c @ 1368]:


7416aa07 59              pop     ecx


7416aa08 59              pop     ecx


7416aa09 e9c2000000      jmp     TSAPPCMP!TsRedirectRegisteredImage+0x173 (7416aad0)


7416aa0e 6a04            push    4


7416aa10 58              pop     eax


7416aa11 8d4df0          lea     ecx,[ebp-10h]


7416aa14 51              push    ecx


7416aa15 50              push    eax


7416aa16 8945f8          mov     dword ptr [ebp-8],eax


7416aa19 8d45f8          lea     eax,[ebp-8]


7416aa1c 50              push    eax


7416aa1d 8d45f4          lea     eax,[ebp-0Ch]


7416aa20 50              push    eax


7416aa21 6aff            push    0FFFFFFFFh


7416aa23 8975f4          mov     dword ptr [ebp-0Ch],esi


7416aa26 ff151c131674    call    dword ptr [TSAPPCMP!_imp__NtProtectVirtualMemory (7416131c)]


 


What address is trying to change the protection.


 


0:001> dc 030afafc


030afafc  741c1098 00000004 741c0000 030afb58  …t…….tX…


030afb0c  7416ac22 741e4818 00000000 00000001  “..t.H.t……..


030afb1c  3c144767 00000000 4b7a203c 00000000  gG.<….< zK….


030afb2c  001a0018 74162684 030afb54 741c00d8  …..&.tT……t


030afb3c  0038be08 030afb1c 00000000 030afb7c  ..8………|…


030afb4c  7416e80a 4b084bdf 00000000 030afb8c  …t.K.K……..


030afb5c  7416af58 3c1447b3 00000000 4b7a203c  X..t.G.<….< zK


030afb6c  00000000 611a0000 030afb60 7499fa2e  …….a`……t


 


 


Location being changed by NtProtectVirtualMemory.


 


0:001> !address 741c1098


TEB 7efdd000 in range 7efdb000 7efde000


TEB 7efda000 in range 7efd8000 7efdb000


ProcessParametrs 003213f0 in range 00320000 0038f000


Environment 00320810 in range 00320000 0038f000


    741c0000 : 741c1000 – 00001000


                    Type     01000000 MEM_IMAGE


                    Protect  00000004 PAGE_READWRITE


                    State    00001000 MEM_COMMIT


                    Usage    RegionUsageImage


                    FullPath C:\Windows\System32\mswsock.dll


 


Failing EIP in the other thread.


 


0:001> !address 741c17cd


TEB 7efdd000 in range 7efdb000 7efde000


TEB 7efda000 in range 7efd8000 7efdb000


ProcessParametrs 003213f0 in range 00320000 0038f000


Environment 00320810 in range 00320000 0038f000


    741c0000 : 741c1000 – 00001000


                    Type     01000000 MEM_IMAGE


                    Protect  00000004 PAGE_READWRITE


                    State    00001000 MEM_COMMIT


                    Usage    RegionUsageImage


                    FullPath C:\Windows\System32\mswsock.dll


 


 


Conclusion


 


This is an issue in TSAPPCMP.dll


 


Until the fix is release if you happen to run in to this issue following is the solution.


Set the following registry value:


[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server]


to


“IAT”=dword:00000001


 


 


 


Comments (0)