NTDebugging Puzzler 0x00000006: Invalid Handle - can you handle it?

Hi NTDebuggers, this week’s puzzler just so happens to match its number: 0x000000006 = ERROR_INVALID_HANDLE. That said, let me give you a scenario and the challenge will be to provide the best action plan to isolate the problem. This should include an explanation of what types of code problems cause invalid handles.

 

Scenario 1 : You have a customer or client that is getting invalid handle errors. This is causing unhandled exceptions and the application is crashing. What action plan do you give the customer? In scenario 1, you don’t have access to the remote machine. Your customer will execute any action plan you give them, gather the data, and provide you with the results.

 

Scenario 2: You have full access to the process and you can debug it live on your own personal machine; you even have source access. What do you do?

 

Good luck and happy debugging.

 


 

[Update: our answer. Posted 5/27/2008]

 

Hi NTDebuggers, if you’ve been debugging for a while I’m sure you’ve run into an Invalid Handle Error 6 (aka ERROR_INVALID_HANDLE or “The handle is invalid”). If you see it in your debugger output it indicates you have tried to pass an invalid handle to a call that is either expecting a different handle type or a the value for the handle its self is not valid. To understand this error in more detail you have to understand what handles are and where they live in the operating system. Each process has a handle table in the kernel. Think of this as an array or table of objects in kernel mode. The handle values are actually multiples of four. Review https://blogs.msdn.com/oldnewthing/archive/2005/01/21/358109.aspx to see for what the bottom 3 bits are used for. For this illustration we will be numbering them 1,2,3 etc.

When you open a file you will get back a simple pointer-sized value that in the handle table points to a file object. In this case let’s say you opened a file and you received a handle value of 2. If you envision the handle table you might see something like this.

------------- USER ADDRESS SPACE FOR YOUR PROCESS -------------

Your handle for your open file is 2

------------- KERNEL ADDRESS SPACE FOR YOUR PROCESS -------------

Handle table

Handle [1]=Pointer to ->Reg

Handle [2]=Pointer to ->File <<<<<< Your handle is 2 It points to a file object

Handle [3]=Pointer to ->File

Handle [4]=Pointer to ->Reg

Handle [5]=Pointer to ->Process

Handle [6]=Pointer to ->Thread

Handle [7]=Pointer to ->Thread

Handle [8]=Pointer to ->File

Handle [9]=Pointer to ->File

In the debugger, you can see the handle table is referenced from the EPROCESS object for each process. In the !process output it is called the ObjectTable.

0: kd> !process 85df8da0

PROCESS 85df8da0 SessionId: 0 Cid: 0114 Peb: 7ffda000 ParentCid: 0268

    DirBase: 0acc07e0 ObjectTable: e4b6a4c8 HandleCount: 49

Before we get into troubleshooting let’s look at a couple of scenarios for bad handles.

In the first bad handle scenario, let’s say your application closes handle 2 using CloseHandle(), however your application keeps the value 2 in a variable somewhere. In this scenario we will assume no other code has come along and opened a new object that is occupying element 2 of the handle table. If you go to make a file I/O call using handle 2 the object manager will catch this and your app will get an invalid handle exception. This is somewhat harmless but your app my crash if you don’t catch the exception and your I/O will obviously not complete.

In the second scenario we’ll say that your file I/O code closes the handle, and holds on to that handle value. Meanwhile another part of your application opens a registry key or some other object that is not a file object. Now your file I/O code goes to use handle 2 again. The problem is it’s not a handle to a file now. When the registry code opened the registry key element 2 was available so we now have a registry object in that location. If you go to do a file I/O on handle 2 you will get an invalid handle message.

Now in our final scenario, we close a file handle in our file I/O code and keep the handle around. In this case though we have different code that is also doing file I/O, It opens a file and gets handle value 2. Now you have big trouble because if the first file I/O code now writes to handle 2, and it should not be because it closed it! This handle now belongs to a different file. This means you have code basically doing a write into an unknown region for that code’s context and file format. This will result in file corruption and no invalid handle error.

So this really comes down to best practice. If you’re writing code and close a handle NULL IT OUT! That way you guarantee it will not be reused accidently at some other point.

CloseHandle(MyHandle);

MyHandle = NULL;

On to debugging: So how do you track this down? For the remote scenario you could give your customer the following action plan:

1. Start gflags and under the image file tab, enter your application name in the image edit control.

2. Check Create User mode stack trace database.

3. Start the process under windbg. “WinDBG processname.exe”

4. In windbg Run !htrace -enable

5. Do a sxe ch. This will cause windbg to break on an invalid handle exception.

6. Run sx to confirm “ch - Invalid handle – break” should be set.

7. Enter g for go.

8. Let the process run until you get an invalid handle message. The debugger should break in.

9. Now all you have to do is run !htrace

Htrace will dump out all the call stacks for each handle that was opened or closed.

You need to take the invalid handle value and search backward to see where it was last opened, and where it was closed before that. It’s likely that the close before the last open is your culprit code path. Make sure that suspect close nulls that handle value out so it does not reuse it.

In the live debug scenario following the same procedure. In this case you have the luxury of going back and setting a breakpoint in the offending call stack that freed the handle before the reuse. You can then figure out why it was not zeroed out.

Good luck and happy debugging.