Spy++ Update

Visual CPP Team

March 22nd, 20070 0

Hi, Pat Brenner again. I recently posted an article about Spy++ internals. I have some updates that I want to share with you. Some of this article will be clearer if you first read that one.

First, Spy++ has been updated for Visual Studio codename Orcas to handle the new common controls/messages/flags added in Windows Vista. This means if you run Spy++ on Vista, you should get all the new messages, and their parameters, decoded correctly so you can more accurately determine what’s going on in your application and the system. These changes go hand-in-hand with a bunch of updates to MFC to wrap the new controls and messages (see related article here), so when you’re using these new capabilities, MFC will make it easier to code them and Spy++ will make it easier to analyze.

Second, in addition to being built as a 32-bit application, Spy++ is now also being built as a 64-bit native application for IA64 and AMD64, so you can log messages in your native 64-bit applications as well. This is because a 32-bit application can only install hooks in other 32-bit applications and a 64-bit application can only install hooks in other 64-bit applications.

Third, there is an effort underway to get some real automated testing running on Spy++. Since its inception 15 years ago, Spy++ has never been tested with any automated test harness. All testing has been manual, and ad hoc, and I’m sure some of the message and parameter cracking methods have never been tested because I’ve never been able to generate those messages during normal usage. Now, however, Visual C++ QA is attempting to reach 70% code coverage on Spy++, so there will be automated tests running on Spy++ to cause those more obscure cracking methods to be invoked. I’m excited about the new level of stability this should bring to Spy++. Ben Anderson from VC++ QA is doing that work; I will encourage him to post about that work here sometime in the future.

Last, I have finally tracked down a hook hanging issue that was causing problems on Windows Vista. I’d been running into a recurring deadlock in the hooks where a hook would block waiting for another shared resource. In the past, this would have caused a system hang, since all the Windows message queues would be stopped waiting for that hook to unblock. Recently, I made a change to Spy++ so that the hooks would time out if a shared resource was not available, but this still meant disabling all the hooks so message logging was essentially aborted. After running Spy++ a number of times, and inserting some diagnostics, I finally came to the conclusion that the synchronization of the circular queue—or more specifically, the synchronization of the write and read offsets in the circular queue—was the problem.

Essentially, the synchronization of the read and write offsets was trying to guarantee that a pending write would not overwrite an area of the circular queue that had not yet been read. So the writers would loop, resetting the read event and waiting for it to be reset, until it was sure that the write would not overwrite unread data (more detail here). I stared at the code doing this synchronization for quite a while, and finally came to an unusual conclusion: I didn’t need it at all.

I realized that with multiple writers and one reader, the queue was probably filling up very rapidly when Spy++ started logging messages, and from then on the writers were always waiting on the reader. I figured if that was the case, why store data for more than one message ever? So I essentially reduced the size of the circular queue so that it was only large enough to hold data about one message. This eliminated the need for offsets, and meant that the mutexes and the events were all I needed to accomplish the synchronization. So the sets of operations performed by the writers and reader have changed since my last post.

The writers (in the hook DLL), when a message comes through one of their hooks, now do the following:

· Wait for the read event to be set. The event is auto-reset, so once the wait is satisfied, the event is reset.

· Take the writer mutex, and then take the access mutex.

· Copy the message data to the shared data area.

· Set the written event.

· Release the access mutex, and release the writer mutex.

And the reader, which resides in a loop in a thread in the Spy++ application, now does the following:

· Wait for the written event to be set. The event is auto-reset, so once the wait is satisfied, the event is reset.

· Take the access mutex.

· Copy the message data from the message packet in the shared queue.

· Release the access mutex.

· Set the read event.

· Send the copied packet data in a message to a hidden window owned by the main thread.

This did solve my hook hang issue, but I was still seeing some strange behavior. After a lengthy discussion with the tester who was validating my fixes, we came to the realization that there were some security issues coming into play. Basically, since the hook is injected into every process, it runs at the integrity level of the process that it is injected into. This means that if the synchronization objects (in this case, the mutexes and events) are created in a process with a higher integrity level, then the hook in the lower-integrity process will not be able to open or manipulate those objects. This fact, in combination with the fact that the error checking on synchronization object manipulation was not robust, meant that some lower-integrity hooks were writing to the shared memory area without synchronizing properly.

This integrity problem was also the reason for the original hook hangs that I was seeing. Some hooks were erroneously thinking that they could obtain the mutexes, but then could not really set the events, so once they entered a loop waiting for an event to be set (which depended on them setting an event), they would deadlock the other hooks.

The fix for this is to set the MIC privilege on the synchronization objects to low. This is done using a couple of APIs, ConvertStringSecurityDescriptorTo SecurityDescriptor and GetSecurityDescriptorSacl, in combination with SetSecurityInfo. This allows the low-integrity process to open and manipulate the synchronization objects. You can see more information on this topic here (see the section on Lowering Resource Integrity).

That pretty much covers the updates I wanted to share with you. I also wanted to mention a couple of usage tips, based on a question from the last article:

· Spy++ can log messages for all windows in a process. This can be used to log all the creation messages, if you use the following trick. Step into your application in the debugger, and once you have the process identifier, start Spy++. Open a processes tree view (Spy.Processes). Select your process ID in the tree, and then do Spy.Messages. In the Message Options dialog, on the Windows tab, your process will already be the selected object. Select the messages you want to see on the Messages tab. OK the dialog, and then let your process run. Now you should see all the window creation messages for windows in your process.

· You can do the same for a single thread within your process if you wish. Start with a threads tree view instead and follow the same steps.

Thanks, and again, I welcome any questions you might have about Spy++, or feature requests for future versions.

Pat Brenner

Visual C++ Libraries Team