The other day I had to troubleshoot an issue with a process that a specific Windows service was trying to launch without success. We could see with a debugger (i.e. WinDbg) that the new process was actually being created but it exited before we even got to its main function.
This may be a typical Desktop Heap issue. I won't explain Desktop Heap here. We already have a great overview on this topic: Desktop Heap Overview. They also explain how to troubleshoot this kind of issues with DHeapMon tool. You should read this article before you continue reading this post.
If everything is explained in previous link, why do I write this post? Well, they don't explain in detail how to configure and use DHeapMon, and it took me a while to figure this out. I hope this saves you some time.
We can get latest version of DHeapMon.exe here: Desktop Heap Monitor Version 8.1.
You will also need to download the following set of tools: Debugging Tools for Windows. We'll need Symchk.exe tool included in those tools.
Once you install all tools in target machine you can use cmd.exe to run the following commands:
1) Set Microsoft symbols server:
set _NT_SYMBOL_PATH = srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Note: We'll need win32k.sys correct symbols to be able to setup DHeapMon.
2) Load symbols for win32k.sys:
symchk c:\windows\system32\win32k.sys /v
Note: The symbols we need will be copied to c:\symbols.
3) Install DHeapMon driver:
dheapinst -y c:\symbols
4) Start DHeapMon driver:
5) Get DHeapMon output:
5.1) For current user session:
5.2) For session 0 (Windows services):
at 13:12 c:\path_to_dheapmon\dheapmon.exe -f c:\result.txt
Note: To access session 0 information we need to run DHeapMon under a high priviledged account in the Windows services world. A trick to do that is to launch the tool as an scheduled task with an AT command. This way DHeapMon will be running (by default) as System in the same session as the other services. We print the output to a .txt file because the tool will be running in an invisible desktop.
6) Stop DHeapMon driver:
7) Uninstall DHeapMon driver:
DHeapMon shows Desktop Heap usage for different Window Station\Desktop combinations. If you want to understand these combinations better, check this article: INFO: Services, Desktops, and Window Stations.
If you want to know more info on typical Desktop Heap issues and changes to SharedSection and SessionViewSize values (commented in the overview article above), check these articles:
I hope this helps.
Alex (Alejandro Campos Magencio)