Debugging mysterious application crash
Today I was asked to help investigating why an application mysteriously crashes.
The symptom is very interesting. First, navigate to a directory with close to 128 characters long. In that directory, any managed WinForm application will fail to launch, while all other kinds of applications runs fine (including managed console applications, unmanaged console and GUI applications). What is more interesting is that, when you launch the WinForm application from a debugger, it works fine.
My first guess is that CreateProcess failed. I attach the debugger to cmd.exe and break on Kernel32!CreateProcessW. Unfortunately, CreateProcessW returns TRUE, which means the process is created successfully. I then realize that this problem does not repro under debugger, which means anything I do under debugger is not going to help.
I then launch FileMon.exe (downloaded from https://www.sysinternals.com) to monitor the files the application accesses. Sure enough, there are some invalid file names the application tries to access. Those invalid file names probably cause the crash. But I still have no idea why the application tries to access those files.
The next thing I do is to run the application under debugger again (where the application works fine), and look at all the DLLs the application loads. For all the DLLs, I check where it is loaded from. There is one dll loaded from temp directory. This is very suspicious. This is a very basic WinForm application (with an empty Form and does nothing). Why will it load anything from temp directory? This very much looks like a spyware or malware.
That DLL has a version resource, and there is a company name in the description. A web search shows the company has been alleged with spyware, while other sites claim that it is of legitimate use. A further search shows that that DLL is carried by a photo processing software, which is bundled with the digital camera the user purchased and installed in the machine.
Anyway this looks promising enough. I rename the DLL and run the WinForm application again. Nothing. The application still can’t be launched.
With frustration, I exam the list of the DLLs the application loads again. All of them are loaded from system32. I carefully check the version resource of every DLL. One DLL catches my eyes. From the description, it is a mouse application and is installed with the driver application comes with the user’s mouse setup CD. I do a tlist on it. Apparently all the GUI applications have it loaded and they all work fine.
Desperately, I rename the mouse DLL and re-launch the WinForm application. Boom! A form shows up in the screen. The application works correctly this time! Apparently, the mouse DLL is causing the application crash.
After uninstall the mouse software, everything works fine.
Two takeaways from this analysis:
- Handling long path is hard. Win32 has a file name length limit of MAX_PATH (which is defined as 260). People are looking for ways to extend the limit, as the file system supports a file name up to 32767 characters long. Apparently this has huge AppCompat implication. As we can see from this example, current applications may already have difficulty handling file path within the MAX_PATH limit, let alone file name with 32767 characters long.
- Injecting code into other people’s process is dangerous. In our example, the mouse application vendor apparently has a very good QA, as all the unmanaged applications work fine. Still, it misses other cases and causes huge user pain.