Debug Fundamentals Exercise 1: Reverse engineer a function

 

Hello ntdebuggers! We’ve seen a lot of interest in our Puzzlers, and we’ve also seen requests and interest in topics covering debugging fundamentals. So we’ve decided to combine the two topics and post a series of “Fundamentals Exercises”. These exercises will be designed more as learning experiences rather than simply puzzlers. We hope you find them interesting and educational!

 

Feel free to post your responses here, but we won’t put them on the site until after we post the “official” responses, so as to avoid spoilers.

 

 

Examine the following code, registers, and stack values to determine the following:

 

1. When the function “DoTheWork” returns, what is the return value from that function?

2. Bonus: what is the mathematical operation that “DoTheWork” performs?

 

Hints:

1. The bracket notation [] in the assembly means to treat the value in brackets as a memory address, and access the value at that address.

2. 32-bit integer return values are stored in eax

 

// Code

0:000> uf eip

demo2!DoTheWork:

0040101c 55 push ebp

0040101d 8bec mov ebp,esp

0040101f 8b4d08 mov ecx,dword ptr [ebp+8]

00401022 8bc1 mov eax,ecx

00401024 49 dec ecx

00401025 0fafc1 imul eax,ecx

00401028 83f902 cmp ecx,2

0040102b 7ff7 jg demo2!DoTheWork+0x8 (00401024)

0040102d 5d pop ebp

0040102e c3 ret

// Current register state

0:000> r

eax=00000007 ebx=7ffd9000 ecx=ffffffff edx=00000007 esi=00001771 edi=00000000

eip=0040101c esp=0012fe9c ebp=0012feac iopl=0 nv up ei pl nz na po nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202

demo2!DoTheWork:

0040101c 55 push ebp

// Current stack values for this thread

0:000> dps esp

0012fe9c 00406717 demo2!main+0x27

0012fea0 00000007

0012fea4 82059a87

0012fea8 00000007

0012feac 0012ff88

0012feb0 004012b2 demo2!mainCRTStartup+0x170

0012feb4 00000002

0012feb8 00980e48

0012febc 00980e80

0012fec0 00000094

0012fec4 00000006

0012fec8 00000000

0012fecc 00001771

0012fed0 00000002

0012fed4 76726553

0012fed8 20656369

0012fedc 6b636150

0012fee0 00003120

0012fee4 00000000

0012fee8 00000000

0012feec 00000000

0012fef0 00000000

0012fef4 00000000

0012fef8 00000000

0012fefc 00000000

0012ff00 00000000

0012ff04 00000000

0012ff08 00000000

0012ff0c 00000000

0012ff10 00000000

0012ff14 00000000

0012ff18 00000000

 


[Update: our answer. Posted 11/19/2008]

Wow - what a great response from our readers on this exercise! It is great to see the various approaches to reverse engineering this code. As for the answer, the numerical result (stored in eax) is 5040, and the corresponding mathematical operation is a factorial. So 7! is the result calculated, given that 7 was passed to the function. Congratulations to all of you that got it right!

Many of you posted some C code to give an idea of what the original source of DoTheWork() might have looked like. The original source was actually written in assembly! However, it was written to be called from C, and it uses ebp in the same way that compiled C code might. This function wasn’t written with optimal performance in mind, but rather for learning about reverse engineering.