Useful service tricks – Debugging service startup


For the better part of the past 15 years, I've been working on one or another services for the Windows platform (not always services for windows, but always services ON windows).

Over that time, I've developed a bag of tricks for working with services, I mentioned one of them here.  Here's another.

One of the most annoying things to have to debug is a problem that occurs during service startup.  The problem is that you can't attach a debugger to the service until it's started, but if the service is failing during startup, that's hard.

It's possible to put a Sleep(10000) to cause your service startup to delay for 10 seconds (which gives you time to attach the debugger during start), that usually works, but sometimes service startup failures only happen on boot (for autostart services).

First off, before you start, you need to have a kernel debugger attached to your computer, and you need the debugging tools for windows (this gets you the command line debuggers).  I'm going to assume the debuggers are installed into "C:\Debuggers", obviously you need to adjust this for your local machine.

One thing to keep in mind: As far as I know, you need have the kernel debugger hooked up to debug service startup issues (you might be able to use ntsd.exe hooked up for remote debugging but I'm not sure if that will work). 

This of course begs the next question: "The kernel debugger?  Why on earth do I need a kernel debugger when I'm debugging user mode code?".  You're completely right.  But in this case, you're not actually using the kernel debugger.  Instead, you're running using a user mode debugger (ntsd.exe in my examples) that's running over the serial port using facilities that are enabled by the kernel debugger.  It's not quite the same thing.

There are multiple reasons for using a debugger that's redirected to a kernel debugger.  First off, if your service is an autostart service, it's highly likely that it starts long before the a user logs on.  So an interactive debugger won't really be able to debug the application.  Secondly, services by default can't interact with the desktop (heck, they often run in a different TS session from the user (this is especially true in Vista, but it's also true on XP with Fast User Switching), so they CAN'T interact with the desktop).  That means that when the debugger attempts to interact with the user, it can't because it flat-out can't because the desktop is sitting in a different TS session.

There are a couple of variants of this trick, all of which should work.

Lets start with the simplest:

If your service runs with a specific binary name, you can use the Image File Execution Options registry key (documented here) to launch your executable under the debugger.  The article linked shows how to launch using Visual Studio, for a service, you want to use the kernel debugger, so instead of using "devenv /debugexe" for the value, use "C:\Debuggers\NTSD.EXE -D", that will redirect the output to the kernel debugger.

 

Now for a somewhat more complicated version - You can ask the service controller to launch the debugger for you.  This is useful if your service is a shared service, or if it lives in an executable that's used for other purposes (if you use a specific -service command line switch to launch your exe as a service, for example).

This one's almost easier than the first.

From the command line, simply type:

sc config <your service short name> binpath= "c:\debuggers\ntsd.exe -d <path to your service executable> <your service executable options>

 

Now restart your service and it should pick up the change.

 

I suspect it's possible to use the ntsd.exe as a host process for remote debugging, I've never done that (I prefer assembly language debugging when I'm using the kernel debugger), so I don't feel comfortable describing how to set it up 🙁

Edit: Answered Purplet's question in the comments (answered it in the post because it was something important that I left out of the article).

Edit2: Thanks Ryan.  s/audiosrv/<your service>/

 

Comments (22)

  1. Anonymous says:

    I’ve written a service one time only (and it could run also as a console application so most of the debug was done in console mode) but I remember I just debugged it using devenv.

    Why is the kernel debugger needed for a standard service (which runs in user mode) ?

  2. Anonymous says:

    Did I miss something?  I put __asm int 3 in the c code and the message box pops up saying to I want to debug it.

  3. Anonymous says:

    I may have missed something, but I put __asm int 3 in my code when I the ability to have a debugger come up and windows puts up that nice dialog asking if I want to attach a debugger to the process.

    Am I missing a big point here?

  4. Anonymous says:

    I may have missed something, but I put "__asm int 3;" in my code where I want to debug and windows puts up that nice dialog asking if I want to attach a debugger to the process.

    Am I missing a big point here?

  5. Anonymous says:

    >sc config audiosrv binpath= "c:debuggersntsd.exe -d <path to your service executable> <your service executable options>

    Presumably, change "audiosrv" to the name of your service as well.

  6. Rob,

     The __asm int 3 (or calling DebugBreak() which works cross platform) relies on the fact that the AE debugger will catch and launch.  First off, it doesn’t work at system startup (there’s no user to catch the debugger).  Second, I’ll be honest and say that I don’t trust the AEDebug stuff, it’s not reliable enough for my tastes (especially on Vista where session 0 isn’t the first logon session).

  7. Anonymous says:

    Another thing that should be mentioned is that if the service does not respond to the SCM’s start request within 30 seconds, the SCM will terminate the service process.  If you’re poking around in the debugger (or waiting to attach the debugger) that 30 seconds can fly by and suddenly your service is gone.

    The timeout can be increased using the registry value:

       HKLMSYSTEMCurrentControlSetControlServicesPipeTimeout

    Set it to the number of milliseconds (default is 30000 = 30 seconds).

  8. Anonymous says:

    I would use a dbgsrv process server in this case.  This is generally going to work as TCP-based networking support doesn’t really rely on user mode components – perhaps an exception might be if you are doing something like debugging the DHCP client service, in which case you would want to assign a static IP address.

    A big advantage to doing it like this is that you can safely try to load symbols on a different computer that is connected to the dbgsrv.  You can still *try* to load symbols in the ntsd controlled via kd, but this is highly dangerous and prone to deadlocking, as there are a couple of services that are touched when doing symbol loading and especially now that most services run in a shared process, you run the risk of having one of them frozen in the debugger that is trying to load symbols.

  9. Anonymous says:

    Is "TS session" the new name for WindowStation?

  10. Anonymous says:

    > Secondly, services by default can’t interact with the

    > desktop (heck, they often run in a different TS session from

    > the user (this is especially true in Vista, but it’s also true on

    > XP with Fast User Switching), so they CAN’T interact with

    > the desktop).  

    That was my understanding too, and it posed a big problem, so I posted in one of the Microsoft public newsgroups (probably the kernel group) and Microsoft gave an answer I wasn’t expecting.  Microsoft said that a service can still open Window Station 0 and still get the desktop of whichever user is logged in at the console.

    Of course we don’t want a service to pump messages from the user’s console, but that’s not an issue here.  The service calls the SendInput() API under conditions where the user has asked the service to do so.  We want SendInput’s target to be in the desktop of that user, not some unsuspecting other user.  I sure hope Microsoft’s answer is going to remain correct.

    Wednesday, March 01, 2006 5:10 PM by mikeb

    > Another thing that should be mentioned is that if the service

    > does not respond to the SCM’s start request within 30

    > seconds, the SCM will terminate the service process.

    Quiz:  Under what condition will the SCM terminate the service process a lot quicker than 30 seconds, and record a message in the event log saying that the service didn’t respond for 30 seconds even when you’ve been watching and 30 seconds haven’t passed yet?

    This happened to me enough times that I’ve learned to recognize it and remember the answer  `_`;;

  11. Anonymous says:

    Why tunnel through KD? You can do it through serial from usermode:

       ntsd -server com:port=COMPort,baud=BaudRate

    Or better yet, TCP/IP:

       ntsd -server tcp:port=1234

    I guess this won’t work when you’re debugging something that prevents networking/COM from working, but how often does that happen?

  12. Jonathan, I did say it might work, I’ve never been able to make it work though, so I didn’t include it.

    chiph: No, you can have multiple windowstations per TS session.

  13. Anonymous says:

    Controlling ntsd through kd is something that has been around for a long time – longer than the other methods of remote debugger control (i.e. process servers).  Since the start, this has been the way you always debugged things like winlogon or (in particular) CSRSS.

    CSRSS debugging is where this feature originated from, I suspect, as when you are doing CSRSS debugging the debugger is unable to call any functions that are ultimately sent to the CSRSS LPC port for further processing (which includes many functions in kernel32 (including almost all console I/O functions) or user32/gdi32 (especially in NT 3.x – almost all GUI output in that case).  Because of these limitations, your allowed I/O functions are restricted to a very minimal subset – the kernel debugger connection being one of the main ways to interact with the user without going through CSRSS.

    In more modern Windows versions, these restrictions are not quite as severe, although there are still a number of functions you can’t call while CSRSS is frozen in the debugger or you would deadlock.

  14. >> If your service runs with a specific binary name, you can use the Image File Execution Options registry key (documented here) to launch your executable under the debugger.

    If your service runs in the shared service host (svchost), then this approach will not help. There are usually 4 or 5 instances of svchost running on a machine. You’ll actually need to move your service into a host with a different name (i.e. not svchost.exe) and then set the Image File Execution options.

  15. mithuns, absolutely.  That’s why I called out the sc config option, especially if it’s coupled with the other linked trick.

    Btw, svchost is internal-to-windows ONLY, so non Microsoft people, please ignore 🙂

  16. Larry,

    Ours is a critical(?) service and so we have ACL’ed both the service and its "CurrentControlSet" in the registry. Only LocalSystem can tweak the service. This is to prevent admins from tampering with it. Even unknowingly.

    Sadly this means "sc config" will return an access denied error even if I am logged on as an admin. Whenever we have to move the service out of its host for testing, we apply a simple workaround/hack.

    Mail me offline to know more……………..:-)

  17. Anonymous says:

    The simple way around that is the age-old "at (current time plus one second) /interactive cmd.exe" which gives you a LocalSystem command prompt if you are logged on session zero.

    Sadly, this trick will be going away with Vista – it’s served me well, though.

  18. Anonymous says:

    Process life support guidance.

  19. Anonymous says:

    Just a few days ago, we had a discussion about ways to attach debuggers to processes whenever a process…

  20. Anonymous says:

    Just a few days ago, we had a discussion about ways to attach debuggers to processes whenever a process

Skip to main content