How to Setup a Debug Crash Cart to Prevent Your Server from Flat Lining

This is Ron Stock from the Global Escalation Services team and I recently had the task of live debugging a customer’s remote server.  In debug circles we use what is known as a crash cart to live debug production servers. The phrase conjures up visions of a wheeled cabinet containing an emergency defibrillator, a heart monitor and latex gloves. Luckily for our purposes, the term merely denotes a machine setup with the Debugging Tools for Windows. This life saving machine is attached to the ailing production server for debugging and no medical degree is required.

 

The ailing production server is referred to as the Target Computer and the Debugging Tools for Windows are installed on the Host computer. The machines are attached with either a null-modem cable, 1394 cable, a special USB cable, or an ethernet cable (network debugging was added in Windows 8). Below I outline serial debugging because this is the most common technique.  In future articles I plan to discuss configuring the other methods.

 

image001

Serial Connection Setup

A null-modem cable is a serial cable used to send data between two serial ports and it can be cheaply purchased at most electronics stores. Be aware these are different from standard serial cables because the transmit and receive lines are cross linked.

 

Plug the null-modem cable into a serial port on each of the computers.  The serial port on the target computer must be built into the system, add on components such as PCI cards will not work for serial debugging on the target computer.

Target Computer setup

 

1. To enable debugging enter the following command from an elevated command prompt.

bcdedit /debug on

 

2. In most systems the default debug settings are sufficient. The current settings can be verified with the below command.

bcdedit /dbgsettings

 

3. Use the below command if you need to change the debug settings, where x is the number of the COM port connected to the null modem cable on the target machine and rate is the baud rate desired for debugging. The rate is usually 115200.

bcdedit /dbgsettings serial debugport:x baudrate:115200

 

5. Reboot the target computer.

 

Host Computer setup

1. First install the Windows Debugging Tools on the host computer. Navigate to the Windows Software Development Kit (SDK) currently located at this link https://msdn.microsoft.com/en-US/windows/hardware/hh852363 and choose the download option.

      a. If you are not able to install the SDK on the host computer, the debugging tools for windows can be installed to a different system and the debugger directory can be copied to the host computer.

 

2. Click Next until you see the Select the features you want to install screen.

 

3. Select only the option named Debugging Tools for Windows and click the Install button. I typically install the tools to a directory named C:\debugger

 

image002

 

4. After the Windows Debugging Tools are installed I set my symbol path by setting the environment variable _NT_SYMBOL_PATH. I recommend setting it to the public symbol server SRV*c:\localsymbols*https://msdl.microsoft.com/download/symbols. If you prefer, you can specify any path in place of ‘c:\localsymbols’.

 

5. Open the debugger by running windbg.exe from the c:\debugger folder.

 

6. On the File menu, choose Kernel Debug.

 

7. In the Kernel Debugging dialog box, open the COM tab.

 

8. In the Baud rate box, enter the same rate you selected for the Target Machine in the steps above. This is usually 115200.

 

9. In the Port box, enter COMx where x is the COM port connected to the null modem cable on this computer. In my example I plugged my null modem cable to com port 1 so I typed com1 in the field.

      a. It is not necessary to use the same port number on both the target and the host.  For example, it is possible to use com1 on the target and com2 on the host.

 

image003

 

10. Click OK and you’ll receive a message indicating the Host computer is waiting to connect.

image004

 

11. From the Debug menu select Break will cause the debugger to break into the target machine and give you the opportunity to debug your ailing production server. Good Luck!!