Today’s journey debugging into DCOM process creation code

I love the feeling of euphoria after you finish debugging a difficult problem. My difficult problem of the day was figuring out why one of the test suites that I own was failing. The test works by running a macro in Visual Studio. The problem was that my macro was failing to read an environment variable. Here is how I debugged it:

Step #1: Determine what process macros run in. To figure this out, I put a message box in my macro, and saw what process the message box came from. It turns out that the answer to this is vsmsvr.exe.

Step #2: Check the process environment of that process. To do this, I attached a debugger, did a Debug->Break. Then I went to the immediate window, and evaluated ‘@env=0’. This uses a new Whidbey feature of dumping the process environment.

Step #3: Determine how the process is being created. For this I ran ‘tlist.exe -t’ this prints out a tree of the current processes. I found vsmsvr.exe, and went up one level. It turns out that its parent process is winlogon.exe.

Step #4: Look at the process creation code in winlogon. I attached a debugger to winlogon. I have never had a chance to actually debug winlogon before, so this was something of a thrill. I then set a breakpoint on kernel32!CreateProcessW. I ran my scenario again, but my breakpoint wasn’t hit. So I deleted my breakpoint, and set breakpoints on the process creation functions from ntdll instead —  {,,ntdll}_NtCreateProcessEx@36 and {,,ntdll}_ZwCreateProcess@32. Then I reran my scenario again. It turns out that this code works by calling CreateProcessAsUser:







Step #5: Look at the arguments to CreateProcessAsUser. I was debugging on the machine that worked, so I noticed that the environment was being passed along. I looked into the code for wlnotify.dll!ProcessExecRequest, and noticed that it was waiting on a named pipe and calling process created based on a request from the pipe. Therefore, I wanted to know who was sending the request.

Step #6: Find client of winlogon request. I started by using procexp.exe from I knew the handle of the pipe, so using that I found its name in procexp.exe. But that turned out to be unhelpful. Next I looked at some of the other local variables with the locals window (you need full symbols for this one). I noticed that one was a handle to a process. I looked up that handle in procexp.exe. It was the svchost.exe process that was hosting the RPC service.

Step #7: Debug into svchost.exe. I then attached a debugger to svchost.exe. I found the bellow thread that was clearly making the request.














Step #8: Try and figure out where the environment was coming from. I looked at the bottom frame (SCMActivatorCreateInstance). It turns out that this frame took the process environment as a member of a structure.

Step #9: Jump to a conclusion that turned out to be correct. At this point I made a guess. I knew that NT didn’t like it if the process environment got too big. I also knew that the team that runs the lab used big configuration scripts that puts a lot of stuff in the process environment. So I tried removing some of the crap, and it turns out that I was correct.

Lessons learned:

  • Don’t fill your process environment with too much stuff

  • DCOM appids that are set to run as ‘launching user’ do get the process environment of their parent process

Comments (5)

  1. Pavel Lebedinsky says:

    Setting breakpoints on system functions in VC is a pain. Why do I need to know how many parameters a function takes to specify something like {,,ntdll}_NtCreateProcessEx@36 ?

    It would have been nice if I could just go to the Command window and do something like

    x ntdll!*createprocess*

    and then

    bp ntdll!NtCreateProcess

  2. Gregg Miskelly says:

    In Whidbey, you can do exsactly what you typed.

  3. Gregg Miskelly says:

    Actually, it would be:

    x ntdll!.*CreateProcess.*

    This is a regular expression instead of a wild card. Not quite sure why we picked regular expressions instead of wild cards…

  4. Andy Pennell says:

    We picked regular expressions as it was easy (just pull in the right VS lib). Wildcards would require some actual code to be written. I think.