Christophe Nasarre: Reviewing Windows Internals 6E and excerpt #1

Hello! Christophe Nasarre here. I’m the Technical Reviewer of the upcoming sixth edition of Windows Internals (which is being released in two parts, the first part in March 2012). and I’ve read all previous editions of the book, going back to when it was titled Inside Windows NT . Microsoft Press asked me to comment on the experience of reviewing the book and, in this and future posts, to share book excerpts containing information I found particularly enlightening.

Opening such a book is like base jumping: you start with a high, level, panoramic view and you juuuummmp deep down, very close to all the little details that help you really understand how everything works. It’s just incredible how much can be found in this book:

The high-level view of all the features included in all editions of Windows 7 and Windows Server 2008 R2 is given to you, in addition to the tools you can use to modify or check their status. All the mandatory notions around system mechanism, process, threads, memory, security, and much more are explained in detail.

Win32 developers are able to find out how their APIs are related to internal data structures and kernel functions. It’s like understanding that you used to walk on an iceberg without any notion of what was underneath your feet.

Kernel-mode developers get a wide view of all the APIs but, more important, all the rules that apply within each part of the system.

How about reviewing this book? Well, first, I’ve learned a lot, but I also learned more about what I thought I already knew! And I’ve used my Win32 view of the Windows API and behaviors to help fill the gaps with all the kernel sides of the story.

Another side of providing the technical review is frustration because it is difficult either to find a valuable technical comment to add for pages or, sometimes, to understand portions of the book’s explanation. (“Am I missing something?” appeared fairly often in my comments in the manuscript.) However, when a tiny detail did not fit in the picture for my mere mortal mind, after a round of questions and answers with the authors, we were able to create an expanded explanation that ended up in the book for all readers. This wipes out all frustration!

So, moving on to my first excerpt. Chapter 1, “Concepts and Tools,” discusses how to use the tools provided by Microsoft to be able to see the details of major kernel/user mode structures. Don’t be afraid! There is no need to be a living god of Cdb or WinDbg. Thanks to LiveKd and the explanations provided by the book, you’ll navigate easily among these structures. But more important, you’ll be able to use these tools and techniques to continue the journey on your own.

Kernel Debugging

Kernel debugging means examining internal kernel data structures and/or stepping through functions
in the kernel. It is a useful way to investigate Windows internals because you can display internal
system information not available through any other tools and get a clearer idea of code flows within
the kernel.

Before describing the various ways you can debug the kernel, let’s examine a set of files that you’ll
need in order to perform any type of kernel debugging.

Symbols for Kernel Debugging

Symbol files contain the names of functions and variables and the layout and format of data
structures. They are generated by the linker and used by debuggers to reference and display these
names during a debug session. This information is not usually stored in the binary image because it is
not needed to execute the code. This means that binaries are smaller and faster. However, this means
that when debugging, you must make sure that the debugger can access the symbol files that are
associated with the images you are referencing during a debugging session.

To use any of the kernel debugging tools to examine internal Windows kernel data structures (such
as the process list, thread blocks, loaded driver list, memory usage information, and so on), you must
have the correct symbol files for at least the kernel image, Ntoskrnl.exe. (The section “Architecture
Overview” in Chapter 2 explains more about this file.) Symbol table files must match the version of
the image they were taken from. For example, if you install a Windows Service Pack or hot fix that
updates the kernel, you must obtain the matching, updated symbol files.

While it is possible to download and install symbols for various versions of Windows, updated
symbols for hot fixes are not always available. The easiest solution to obtain the correct version of
symbols for debugging is to use the Microsoft on-demand symbol server by using a special syntax for
the symbol path that you specify in the debugger. For example, the following symbol path causes the
debugging tools to load required symbols from the Internet symbol server and keep a local copy in
the c:\symbols folder:

srv*c:\symbols*https://msdl.microsoft.com/download/symbols

For detailed instructions on how to use the symbol server, see the debugging tools help file or the
Web page https://msdn.microsoft.com/en-us/windows/hardware/gg462988.aspx.

Debugging Tools for Windows

The Debugging Tools for Windows package contains advanced debugging tools used in this book
to explore Windows internals. The latest version is included as part of the Windows Software
Development Kit (SDK). These tools can be used to debug user-mode processes as well as the kernel.
(See the following sidebar.)

Note The Debugging Tools for Windows are updated frequently and released
independently of Windows operating system versions, so check often for new versions.

Sidebar: User-Mode Debugging

The debugging tools can also be used to attach to a user-mode process and examine and/or
change process memory. There are two options when attaching to a process:

• Invasive Unless specified otherwise, when you attach to a running process, the
  DebugActiveProcess Windows function is used to establish a connection between the
  debugger and the debugee. This permits examining and/or changing process memory,
  setting breakpoints, and performing other debugging functions. Windows allows you to
  stop debugging without killing the target process, as long as the debugger is detached,
  not killed.
• Noninvasive With this option, the debugger simply opens the process with the
  OpenProcess function. It does not attach to the process as a debugger. This allows you to
  examine and/or change memory in the target process, but you cannot set breakpoints.

You can also open user-mode process dump files with the debugging tools. User-mode
dump files are explained in Chapter 3 in the section on exception dispatching.

There are two debuggers that can be used for kernel debugging: a command-line version (Kd.exe)
and a graphical user interface (GUI) version (Windbg.exe). Both provide the same set of commands,
so which one you choose is a matter of personal preference. You can perform three types of kernel
debugging with these tools:

• Open a crash dump file created as a result of a Windows system crash. (See Chapter 7, “Crash
  Dump Analysis,” in Part 2 for more information on kernel crash dumps.)
• Connect to a live, running system and examine the system state (or set breakpoints if you’re
  debugging device driver code). This operation requires two computers—a target and a host.
  The target is the system being debugged, and the host is the system running the debugger.
  The target system can be connected to the host via a null modem cable, an IEEE 1394 cable,
  or a USB 2.0 debugging cable. The target system must be booted in debugging mode (either
  by pressing F8 during the boot process and selecting Debugging Mode or by configuring
  the system to boot in debugging mode using Bcdedit or Msconfig.exe). You can also connect
  through a named pipe, which is useful when debugging through a virtual machine product
  such as Hyper-V, Virtual PC, or VMWare, by exposing the guest operating system’s serial port
  as a named pipe device.
• Windows systems also allow you to connect to the local system and examine the system state.
  This is called local kernel debugging. To initiate local kernel debugging with WinDbg, open the
  File menu, choose Kernel Debug, click on the Local tab, and then click OK. The target system
  must be booted in debugging mode. An example output screen is shown in Figure 1-6. Some
  kernel debugger commands do not work when used in local kernel debugging mode (such as
  creating a memory dump with the .dump command—however, this can be done with LiveKd,
  described later in this section).

Once connected in kernel debugging mode, you can use one of the many debugger extension
commands (commands that begin with “!”) to display the contents of internal data structures such
as threads, processes, I/O request packets, and memory management information. Throughout this
book, the relevant kernel debugger commands and output are included as they apply to each topic
being discussed. An excellent companion reference is the Debugger.chm help file, contained in the
WinDbg installation folder, which documents all the kernel debugger functionality and extensions. In
addition, the dt (display type) command can format over 1000 kernel structures because the kernel
symbol files for Windows contain type information that the debugger can use to format structures.

EXPERIMENT: Displaying Type Information for Kernel Structures

To display the list of kernel structures whose type information is included in the kernel symbols,
type dt nt!_* in the kernel debugger. A sample partial output is shown here:

lkd> dt nt!_*
nt!_LIST_ENTRY
nt!_LIST_ENTRY
nt!_IMAGE_NT_HEADERS
nt!_IMAGE_FILE_HEADER
nt!_IMAGE_OPTIONAL_HEADER
nt!_IMAGE_NT_HEADERS
nt!_LARGE_INTEGER

You can also use the dt command to search for specific structures by using its wildcard
lookup capability. For example, if you were looking for the structure name for an interrupt
object, type dt nt!_*interrupt* :

lkd> dt nt!_*interrupt*
nt!_KINTERRUPT
nt!_KINTERRUPT_MODE
nt!_KINTERRUPT_POLARITY
nt!_UNEXPECTED_INTERRUPT

Then you can use dt to format a specific structure as shown next:

lkd> dt nt!_kinterrupt
nt!_KINTERRUPT
+0x000 Type : Int2B
+0x002 Size : Int2B
+0x008 InterruptListEntry : _LIST_ENTRY
+0x018 ServiceRoutine : Ptr64 unsigned char
+0x020 MessageServiceRoutine : Ptr64 unsigned char
+0x028 MessageIndex : Uint4B
+0x030 ServiceContext : Ptr64 Void
+0x038 SpinLock : Uint8B
+0x040 TickCount : Uint4B
+0x048 ActualLock : Ptr64 Uint8B
+0x050 DispatchAddress : Ptr64 void
+0x058 Vector : Uint4B
+0x05c Irql : UChar
+0x05d SynchronizeIrql : UChar
+0x05e FloatingSave : UChar
+0x05f Connected : UChar
+0x060 Number : Uint4B
+0x064 ShareVector : UChar
+0x065 Pad : [3] Char
+0x068 Mode : _KINTERRUPT_MODE
+0x06c Polarity : _KINTERRUPT_POLARITY
+0x070 ServiceCount : Uint4B
+0x074 DispatchCount : Uint4B
+0x078 Rsvd1 : Uint8B
+0x080 TrapFrame : Ptr64 _KTRAP_FRAME
+0x088 Reserved : Ptr64 Void
+0x090 DispatchCode : [4] Uint4B

Note that dt does not show substructures (structures within structures) by default. To recurse
through substructures, use the –r switch. For example, using this switch to display the kernel
interrupt object shows the format of the _LIST_ENTRY structure stored at the InterruptListEntry
field:

lkd> dt nt!_kinterrupt -r
nt!_KINTERRUPT
+0x000 Type : Int2B
+0x002 Size : Int2B
+0x008 InterruptListEntry : _LIST_ENTRY
+0x000 Flink : Ptr64 _LIST_ENTRY
+0x000 Flink : Ptr64 _LIST_ENTRY
+0x008 Blink : Ptr64 _LIST_ENTRY
+0x008 Blink : Ptr64 _LIST_ENTRY
+0x000 Flink : Ptr64 _LIST_ENTRY
+0x008 Blink : Ptr64 _LIST_ENTRY

The Debugging Tools for Windows help file also explains how to set up and use the kernel
debuggers. Additional details on using the kernel debuggers that are aimed primarily at device driver
writers can be found in the Windows Driver Kit documentation.

LiveKd Tool

LiveKd is a free tool from Sysinternals that allows you to use the standard Microsoft kernel debuggers
just described to examine the running system without booting the system in debugging mode. This
approach might be useful when kernel-level troubleshooting is required on a machine that wasn’t
booted in debugging mode—certain issues might be hard to reproduce reliably, so a reboot with the
debug option enabled might not readily exhibit the error.

You run LiveKd just as you would WinDbg or Kd. LiveKd passes any command-line options you
specify to the debugger you select. By default, LiveKd runs the command-line kernel debugger (Kd).
To have it run WinDbg, specify the –w switch. To see the help files for LiveKd switches, specify the –?
switch.

LiveKd presents a simulated crash dump file to the debugger, so you can perform any operations
in LiveKd that are supported on a crash dump. Because LiveKd is relying on physical memory to back
the simulated dump, the kernel debugger might run into situations in which data structures are in the
middle of being changed by the system and are inconsistent. Each time the debugger is launched,
it starts with a fresh view of the system state. If you want to refresh the snapshot, quit the debugger
(with the q command), and LiveKd will ask you whether you want to start it again. If the debugger
enters a loop in printing output, press Ctrl+C to interrupt the output and quit. If it hangs, press
Ctrl+Break, which will terminate the debugger process. LiveKd will then ask you whether you want to
run the debugger again.