LPC part 2 Kernel Debugger Extensions
Hello my name is Roy, I’m an EE on the Microsoft global escalation services / CPR team. This blog is a follow on to my first LPC blog. We will be discussing debugger extensions that allow you to look at LPC related issues.
Disclaimer: The purpose of this blog is to illustrate debugging techniques with LPC. Please do not rely on this information as documentation for the APIs when writing your own code. As always, this is subject to change in the future.
LPC Kernel Debugger Extensions
Command |
Description |
!lpc |
Display the list and description of all the !lpc commands |
!lpc message [MessageId] |
Display the message with a given ID and all related information by attempting to match the given Message ID to the EHTREAD->LpcReceivedMessageId and to the ETHREAD->LpcReplyMessageId of all threads in the system. If the MessageId parameter is omitted then it attempts to display all the outstanding messages in the system by searching for the tag ‘LpcM’ in the pools. |
!lpc port [PortAddress] |
Displays port information. If a server connection port address is specified then only information about that port is displayed. If either a client or server communication port is specified it prints information about the specified communication port, the peer communication port and the server connection port. If the PortAddress parameter is omitted then it attempts to walk the list of all objects of type “Port” and “WaitablePort” and display them. Note that for this feature to work the GFlags option “+otl” i.e. “Maintain a list of objects for each type” must be enabled. |
!lpc scan PortAddress |
Displays port information. It attempts to walk the list of all objects of type “Port” and “WaitablePort” and display the one matching the specified port address. Note that for this feature to work the GFlags option “+otl” i.e. “Maintain a list of objects for each type” must be enabled. |
!lpc thread [ThreadAddr] |
If ThreadAddr is specified it walks the list of threads in the ETHREAD-> LpcReplyChain to locate the list head i.e. a “Port” or “WaitablePort” object on which the thread is waiting for a reply. If the ThreadAddr parameter is omitted then it attempts to find all LPC server threads by looking for threads with a non-NULL EHTREAD->LpcReceivedMessageId and all client threads by looking for threads with a non-NULL ETHREAD->LpcReplyMessageId and displays them. |
!lpc PoolSearch |
Toggles a setting that controls whether the “lpc message” command will search for LPC message tag (‘LpcM’) in the kernel pools or not. |
LPC Kernel Debugger Extension Usage
LPC Connection Port information from call stack
On the call stack of any client or server threads blocked on LPC data transfer or LPC connection there will be a frame containing either one of the functions NtRequestWaitReplyPort() or NtReplyWaitReceivePortEx() . The first parameter to either one of these functions is the handle to the port they are blocked on.
kd> !thread 810de2a8
THREAD 810de2a8 Cid 01dc.01f4 Teb: 7ffde000 Win32Thread: 00000000 WAIT: (WrLpcReceive) UserMode Non-Alertable
81131188 Semaphore Limit 0x7fffffff
810de398 NotificationTimer
Not impersonating
DeviceMap e196c460
Owning Process 810ddda0 Image: rpclpcs.exe
Wait Start TickCount 64555 Ticks: 402 (0:00:00:04.025)
Context Switch Count 2
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x77e76bf0
Start Address 0x7c810856
Stack Init f8e28000 Current f8e27c4c Base f8e28000 Limit f8e25000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr Args to Child
f8e27c64 804dc6a6 810de318 810de2a8 804dc6f2 nt!KiSwapContext+0x2e (FPO: [Uses EBP] [0,0,4])
f8e27c70 804dc6f2 e1084108 8055a540 e1084108 nt!KiSwapThread+0x46 (FPO: [0,0,0])
f8e27c98 8056a50a 00000001 00000010 00000001 nt!KeWaitForSingleObject+0x1c2 (FPO: [Non-Fpo])
f8e27d48 804df06b 000007c4 002bff70 00000000 nt!NtReplyWaitReceivePortEx+0x3dc (FPO: [Non-Fpo])
f8e27d48 7c90eb94 000007c4 002bff70 00000000 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ f8e27d64)
002bff80 00000000 00000000 00000000 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
kd> !handle 7c4 3 810ddda0
processor number 0, process 810ddda0
PROCESS 810ddda0 SessionId: 0 Cid: 01dc Peb: 7ffd9000 ParentCid: 01b4
DirBase: 058cd000 ObjectTable: e1a13278 HandleCount: 18.
Image: rpclpcs.exe
Handle table at e107d000 with 18 Entries in use
07c4: Object: e1084108 GrantedAccess: 001f0001 Entry: e107df88
Object: e1084108 Type: (812b5c80) Port
ObjectHeader: e10840f0 (old version)
HandleCount: 1 PointerCount: 4
Directory Object: e14c72c8 Name: rpclpc
kd> !lpc port e1084108
Server connection port e1084108 Name: rpclpc
Handles: 1 References: 4
Server process : 810ddda0 (rpclpcs.exe)
Queue semaphore : 81131188
Semaphore state 0 (0x0)
The message queue is empty
The LpcDataInfoChainHead queue is empty
LPC Messages that are waiting to be picked up by the server thread
When a message is queued to the server connection port and the server thread has not yet not been signaled to pull out the message the ‘!lpc port <Port>” displays the following output that includes the message.
kd> !lpc port e10d8388
Client communication port 0xe10d8388
Handles: 1 References: 2
The LpcDataInfoChainHead queue is empty
Connected port: 0xe106dc00 Server connection port: 0xe15512e0
Server communication port 0xe106dc00
Handles: 1 References: 1
The LpcDataInfoChainHead queue is empty
Server connection port e15512e0 Name: rpclpc
Handles: 1 References: 9
Server process : ffbbebe8 (rpclpcs.exe)
Queue semaphore : 81250848
Semaphore state 0 (0x0)
Messages in queue:
0000 e10513f8 - Busy Id=000048d8 From: 05f4.0108 Context=80020000 [e15512f0 . e15512f0]
Length=0054003c Type=00000001 (LPC_REQUEST)
Data: 00000001 00000241 00000000 00000000 f877bc04 804ec10e
The message queue contains 1 messages
The LpcDataInfoChainHead queue is empty
LPC Messages being processed by the server
When an LPC message has been queued by the client, dequeued by the sever and being worked upon by the server both the client and the server thread have the message ID associated with them
kd> !lpc message 16fa
Searching message 16fa in threads ...
Server thread 810de2a8 is working on message 16fa
Client thread 810dc930 waiting a reply from 16fa
Searching thread 810dc930 in port rundown queues ...
Server connection port e1084108 Name: rpclpc
Handles: 1 References: 3
Server process : 810ddda0 (rpclpcs.exe)
Queue semaphore : 81131188
Semaphore state 0 (0x0)
The message queue is empty
The LpcDataInfoChainHead queue is empty
Done.
kd> !thread 810de2a8
THREAD 810de2a8 Cid 01dc.01f4 Teb: 7ffde000 Win32Thread: 00000000 RUNNING on processor 0
Not impersonating
DeviceMap e196c460
Owning Process 810ddda0 Image: rpclpcs.exe
Wait Start TickCount 65732 Ticks: 10 (0:00:00:00.100)
Context Switch Count 4
UserTime 00:00:00.000
KernelTime 00:00:00.010
Win32 Start Address 0x000016fa
LPC Server thread working on message Id 16fa
Start Address kernel32!BaseThreadStartThunk (0x7c810856)
Stack Init f8e28000 Current f8e27728 Base f8e28000 Limit f8e25000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr Args to Child
f8e27d64 7c90eb94 badb0d00 002bfe24 00000000 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ f8e27d64)
002bfe18 7c90e399 77e76703 000007c4 002bff70 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
002bfe1c 77e76703 000007c4 002bff70 00000000 ntdll!NtReplyWaitReceivePortEx+0xc (FPO: [5,0,0])
002bff80 77e76c22 002bffa8 77e76a3b 00084540 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0xf4 (FPO: [Non-Fpo])
002bff88 77e76a3b 00084540 7c90ee18 0006fb10 RPCRT4!RecvLotsaCallsWrapper+0xd (FPO: [Non-Fpo])
002bffa8 77e76c0a 00083f20 002bffec 7c80b50b RPCRT4!BaseCachedThreadRoutine+0x79 (FPO: [Non-Fpo])
002bffb4 7c80b50b 00084718 7c90ee18 0006fb10 RPCRT4!ThreadStartRoutine+0x1a (FPO: [Non-Fpo])
002bffec 00000000 77e76bf0 00084718 00000000 kernel32!BaseThreadStart+0x37 (FPO: [Non-Fpo])
kd> !thread 810dc930
THREAD 810dc930 Cid 01e4.01ec Teb: 7ffde000 Win32Thread: 00000000 READY
Waiting for reply to LPC MessageId 000016fa:
Pending LPC Reply Message:
e16b8538: [e16b8538,e16b8538]
Not impersonating
DeviceMap e196c460
Owning Process 810e9860 Image: rpclpcc.exe
Wait Start TickCount 65731 Ticks: 11 (0:00:00:00.110)
Context Switch Count 26
UserTime 00:00:00.000
KernelTime 00:00:00.020
Win32 Start Address rpclpcs!pre_c_init (0x01001a91)
Start Address kernel32!BaseProcessStartThunk (0x7c810867)
Stack Init fbe0e000 Current fbe0dc28 Base fbe0e000 Limit fbe0b000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0 DecrementCount 16
ChildEBP RetAddr Args to Child
fbe0dc40 804ea3a4 00000000 810dc930 e107f388 nt!KiUnlockDispatcherDatabase+0x77 (FPO: [Uses EBP] [0,0,4])
fbe0dc58 80586255 81131188 00000001 00000001 nt!KeReleaseSemaphore+0x70 (FPO: [Non-Fpo])
fbe0dd10 80598c58 000847b0 0006f8c0 0006f8c8 nt!NtSecureConnectPort+0x635 (FPO: [Non-Fpo])
fbe0dd3c 804df06b 000847b0 0006f8c0 0006f8c8 nt!NtConnectPort+0x24 (FPO: [Non-Fpo])
fbe0dd3c 7c90eb94 000847b0 0006f8c0 0006f8c8 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ fbe0dd64)
0006f8d0 7c93040b ffffffff 00000000 7c859e48 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
0006f9b0 7c8107fd 00000000 00084718 7c90e642 ntdll!DbgPrint+0x1b (FPO: [Non-Fpo])
0006fe68 77ea11a8 00000022 00084540 00000000 kernel32!CreateRemoteThread+0x284 (FPO: [Non-Fpo])
000847a8 00000000 00000001 00000000 00000000 RPCRT4!LRPC_ADDRESS::CompleteListen+0x84 (FPO: [Non-Fpo])
Finding all threads in the system involved in LPC communication
To get a consolidated list of all client threads in the system waiting on LPC replies and a list of server threads working on received LPC requests use the “!lpc threads” command without any parameters
0: kd> !lpc thread
Searching message 0 in threads ...
Client thread 89a4cdb0 waiting a reply from 15f6
Server thread 898c6a50 is working on message 63f09
Server thread 89671020 is working on message 63f75
Server thread 89904a48 is working on message 63f10
Server thread 88d6bdb0 is working on message 3d6b1
Server thread 89973db0 is working on message 63f32
Server thread 896d32b8 is working on message 3454
Server thread 8995d020 is working on message 63f76
Server thread 8960f020 is working on message 63f77
Server thread 898cd350 is working on message 61bd9
Server thread 8900edb0 is working on message 3832f
Server thread 8900fbf0 is working on message 33c2e
Server thread 88d539a8 is working on message 3343f
Client thread 89be4020 waiting a reply from 6077e
Client thread 89012990 waiting a reply from 39239
Server thread 89012990 is working on message 39232
Client thread 89531020 waiting a reply from 3923d
Server thread 89531020 is working on message 39236
Client thread 88d13b40 waiting a reply from 3c96e
Quick Tips
8 No thread in the system is doing a NtListenPort( ) does that mean no LPC connection port is open for connections?
!stacks 2 NtListenPort does not return any thread because NtListenPort( ) calls NtRequestWaitReplyPort( ) to wait for an incoming LPC_CONNECTION_REQUEST message. Instead of calling NtListenPort( ) for listening for incoming connections most LPC connection threads directly call NtRequestWaitReplyPort( ).
8 !lpc message does not show any message?
!poolfind LpcM 1 should do the trick.
0: kd> !lpc message
Scanning large pool allocation table for Tag: LpcM (8974e000 : 8976e000)
Searching Paged pool (e1000000 : f7000000) for Tag: LpcM
0: kd> !poolfind LpcM 1
Scanning large pool allocation table for Tag: LpcM (8974e000 : 8976e000)
Searching Paged pool (e1000000 : f7000000) for Tag: LpcM
e1061000 size: 168 previous size: 0 (Allocated) LpcM
e1090758 size: 168 previous size: 8 (Allocated) LpcM
e10a7a40 size: 168 previous size: 8 (Allocated) LpcM
e1188468 size: 8 previous size: 20 (Free) LpcM
e1261000 size: 168 previous size: 0 (Allocated) LpcM
e1297428 size: 168 previous size: 8 (Allocated) LpcM
Once you have the pool blocks listed with the “LpcM” tag dump the ones that are “Allocated”. On a 32-bit machine add 8bytes to the pool address listed on the left most column. On a 64-bit machine add 16bytes i.e. 0x10. Dump the message as below to retrieve the message id.
0: kd> dt nt!_LPCP_MESSAGE e1061000+8
+0x000 Entry : _LIST_ENTRY [ 0xe295fa80 - 0xe29279e8 ]
+0x000 FreeEntry : _SINGLE_LIST_ENTRY
+0x004 Reserved0 : 0xe29279e8
+0x008 SenderPort : 0xe2327020
+0x00c RepliedToThread : (null)
+0x010 PortContext : 0x8021000d
+0x018 Request : _PORT_MESSAGE
0: kd> dt nt!_LPCP_MESSAGE e1061000+8 Request.
+0x018 Request :
+0x000 u1 : __unnamed
+0x004 u2 : __unnamed
+0x008 ClientId : _CLIENT_ID
+0x008 DoNotUseThisField : 9.8460604703041575e-311
+0x010 MessageId : 0x4617f
+0x014 ClientViewSize : 0
+0x014 CallbackId : 0
0: kd> !lpc message 0x4617f
Searching message 4617f in threads ...
Server thread 88f91c08 is working on message 4617f
Client thread 88f0f5f8 waiting a reply from 4617f
Searching thread 88f0f5f8 in port rundown queues ...
Server communication port 0xe2300a38
Handles: 1 References: 1
The LpcDataInfoChainHead queue is empty
Connected port: 0xe2327020 Server connection port: 0xe13734b0
Client communication port 0xe2327020
Handles: 1 References: 9
The LpcDataInfoChainHead queue is empty
Server connection port e13734b0 Name: epmapper
Handles: 1 References: 93
Server process : 896e1450 (svchost1.exe)
Queue semaphore : 8995ac10
Semaphore state 0 (0x0)
The message queue is empty
8 The number of server threads shown by !lpc thread processing requests is much more than the number of client threads waiting for reply.
!lpc thread interprets stale data in ETHREAD structure as valid LPC information. An indication of the stale data is the ETHREAD.LpcExitThreadCalled will be set. This field is only set when the thread is exiting.
0: kd> !lpc thread
Searching message 0 in threads ...
Client thread 89a4cdb0 waiting a reply from 15f6
Server thread 898c6a50 is working on message 63f09
Server thread 89671020 is working on message 63f75
Server thread 89904a48 is working on message 63f10
Server thread 88d6bdb0 is working on message 3d6b1
Server thread 89973db0 is working on message 63f32
Server thread 896d32b8 is working on message 3454
Server thread 8995d020 is working on message 63f76
Server thread 8960f020 is working on message 63f77
Server thread 898cd350 is working on message 61bd9
Server thread 8900edb0 is working on message 3832f
Server thread 8900fbf0 is working on message 33c2e
Server thread 88d539a8 is working on message 3343f
0: kd> !lpc message 3454
Searching message 3454 in threads ...
Server thread 896d32b8 is working on message 3454
Done.
0: kd> dt nt!_ETHREAD 896d32b8 Lpc.
+0x1c0 LpcReplyChain : [ 0x896d3478 - 0x896d3478 ]
+0x000 Flink : 0x896d3478 _LIST_ENTRY [ 0x896d3478 - 0x896d3478 ]
+0x004 Blink : 0x896d3478 _LIST_ENTRY [ 0x896d3478 - 0x896d3478 ]
+0x1ec LpcReplySemaphore :
+0x000 Header : _DISPATCHER_HEADER
+0x010 Limit : 1
+0x200 LpcReplyMessage :
+0x200 LpcWaitingOnPort :
+0x220 LpcReceivedMessageId : 0x3454
+0x234 LpcReplyMessageId : 0
+0x248 LpcReceivedMsgIdValid : 0x1 ''
+0x248 LpcExitThreadCalled : 0x1 ''
8 Finding all client threads in the system waiting for LPC reply.
!lpc thread