WinDBG Tutorial - Part 2

  • Article

Items covered

- Stack overflow

- Stack trace (k)

- Frame inspection (.frame n)

Code used

 

For this exercise, we will work with the following test program (Win32 C++ console project):

#include "stdafx.h"

int pow(int,int);

int _tmain(int argc, _TCHAR* argv[])

{

      int a = 2;

      int p = 9;

      int c = 0;

     

      printf("a=");

      scanf("%d",&a);

      printf("p=");

      scanf("%d",&p);

      c = pow(a,p);

      printf("C=%d\n",c);

      return 0;

}

int pow(int a, int p){

      if(a > 2*p)

            return a-p;

      return pow(a*a, p + 1);

}

What the program does is: read two variables, a and p and afterwards call a rather unusual method named “pow”, which seems to be a finite recursion, ending when the value of the variable a is more than twice the value of variable p. The recursion seems correct, as variable a is squared every iteration whereas p is only incremented. Let’s see what the trouble actually is.

Debugging

 

Compile and run the program from console:

D:\home\…\Emptyapp\Debug>Emptyapp.exe

Set the symbol path in WinDBG to the Debug folder of the C++ program (where EmptyApp.exe and PDB files are located).

Open WinDBG and attach to the process named “EmptyApp”

For the moment, we will choose Go, as no exception as occurred yet:

0:001> g

(The message in WinDBG will be “Debugee is running”)

Let’s now input two “special” values: a = 256, b = 65556

When you press ENTER, winDBG will break because it receives a First-hand exception:

0:001> g

(1b64.1d68): Stack overflow - code c00000fd (first chance)

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

Why did the stack overflow happen?

Let’s see the stack trace:

0:000> k

ChildEBP RetAddr

00083004 00391474 Emptyapp!pow+0x9 [d:\home\allinoner\emptyapp\emptyapp\emptyapp.cpp @ 23]

000830e0 00391474 Emptyapp!pow+0x44 [d:\home\allinoner\emptyapp\emptyapp\emptyapp.cpp @ 26]

000831bc 00391474 Emptyapp!pow+0x44 [d:\home\allinoner\emptyapp\emptyapp\emptyapp.cpp @ 26]

00083298 00391474 Emptyapp!pow+0x44 [d:\home\allinoner\emptyapp\emptyapp\emptyapp.cpp @ 26]

00083374 00391474 Emptyapp!pow+0x44 [d:\home\allinoner\emptyapp\emptyapp\emptyapp.cpp @ 26]

Obviously, the method “pow” is called indefinitely and does not end until the stack overflows.

Let’s see the local variables:

0:000> .frame 0

00 00083004 00391474 Emptyapp!pow+0x9 [d:\home\allinoner\emptyapp\emptyapp\emptyapp.cpp @ 23]

0:000> dv

              a = 0

              p = 70259

Hmm, value of a is 0…

0:000> ?? sizeof(a)

unsigned int 4

Let’s go back to previous frames (the actual number of total frames might vary):

0:000> .frame 125E

125e 0017f8cc 00391474 Emptyapp!pow+0x44 [d:\home\allinoner\emptyapp\emptyapp\emptyapp.cpp @ 26]

0:000> dv

              a = 65536

              p = 65557

This is OK. What is the next frame, then?

0:000> .frame 125D

125d 0017f7f0 00391474 Emptyapp!pow+0x44 [d:\home\allinoner\emptyapp\emptyapp\emptyapp.cpp @ 26]

0:000> dv

              a = 0

              p = 65558

Remember that the code at frame 125E calls “pow” at 125D with the argument: a = 65536.

What would a*a be evaluated to at frame 125E?

0:000> .frame 125E

125e 0017f8cc 00391474 Emptyapp!pow+0x44 [d:\home\allinoner\emptyapp\emptyapp\emptyapp.cpp @ 26]

0:000> dv

              a = 65536

              p = 65557

0:000> ?? a*a

int 0

Conclusion: when multiplying a, the result value will be 0, because the multiplication of integers so high will result in integer overflow.

After a = 0, the method cannot end, as p steadily increases and a remains 0. This results in the found stack overflow.