Why does my COM port disappear when I enable the kernel debugger?

A lot of folks are told to connect a kernel debugger (over a serial calbe) to their systems if
it is constantly blue screening or if there are suspected issues in the kernel
or a loaded driver. Most of these folks do not have the skills to debug the
issue themselves, they are just setting up their machine so that once the problem
shows up, somebody else can debug it. One thing that happens is that the
person whose machine is being debugged looks in device manager and sees one of
two things...

1. The COM that they are using to debug the machine is yellow !'ed out and
   non functional.
2. The COM port does not appear in device manager at all.

...and then summarily freak out. Reactions range from wondering if the kernel
debugger is working at all (how could it? the port it using appears to be non functional)
to anger that the COM port that they wanted to use is now gone.

So what exactly is happening? First and foremost, in both situations the
kernel has assumed ownership of the COM port's resources and is using those resources
to directly manipulate the port. The kernel is not opening the port by name and
using the serial driver (serial.sys) to control the hardware, it is direct
manipulation. This has an important side affect...there must be real UART hardware
for the kernel to access; a USB to serial adapter will not work on the target side
of a kernel debug session because there are no real UART registers to touch! When debugging
over a serial cable is enabled, the kernel export KdComPortInUse
contains the starting address of the UART resources being used by the kernel debugger.

So why are there different visual results to enabling a kernel debugger? Well,
the results depend on if it is an ACPI machine (and chances are, any machine
built in the past 5 years is ACPI enabled by default). If ACPI is enabled, ACPI
will look at the address in KdComPortInUse and see if a device
in the ACPI table starts with that address. If a match is made, ACPI does not
even enumerate the COM at all. This is why you don't see a !'ed out device. Looking at
ACPI.sys's imports, we see that it does reference the export:

 
C:\WINDOWS\system32\drivers>link /dump /imports acpi.sys | findstr KdComPortInUse
                   38 KdComPortInUse

The !'ed out device occurs when ACPI is not enabled on the machine. In this
case, the BIOS reported COM is enumerated, the serial driver is attached to the stack,
and an IRP_MN_START_DEVICE is sent to the stack.
In the process of starting the device, serial looks at the assigned resources to
the device and if they match KdComPortInUse,
it will fail the IRP_MN_START_DEVICE, resulting
in a failed start which shows up as a !'ed device in device manager. Since the
serial driver is a DDK/WDK sample, we can see what it does:

 
NTSTATUS
SerialInitController(
    IN PDEVICE_OBJECT PDevObj,
    IN PCONFIG_DATA PConfigData
    )
{
   [...]

#ifdef _WIN64
   BOOLEAN DebugPortInUse = FALSE;

   //
   // First check what type of AddressSpace this port is in. Then check
   // if the debugger is using this port. If it is set DebugPortInUse to TRUE.
   //
   if(PConfigData->AddressSpace == CM_RESOURCE_PORT_MEMORY) {
        PHYSICAL_ADDRESS  KdComPhysical;
        KdComPhysical = MmGetPhysicalAddress(*KdComPortInUse);
        if(KdComPhysical.LowPart == PConfigData->Controller.LowPart) DebugPortInUse = TRUE;
   } else {
        if ((*KdComPortInUse) == (ULongToPtr(PConfigData->Controller.LowPart))) DebugPortInUse = TRUE;
   }

   if (DebugPortInUse) {
      [...]
      return STATUS_INSUFFICIENT_RESOURCES;
   }
#else
   //
   // This compare is done using **untranslated** values since that is what
   // the kernel shoves in regardless of the architecture.
   //
   if ((*KdComPortInUse) == (ULongToPtr(PConfigData->Controller.LowPart))) {
      [...]
      return STATUS_INSUFFICIENT_RESOURCES;
   }
}