A story of gflags and the crashing Notepad

As you can imagine, being able to reproduce a problem on a machine you can put your hands on is the best thing to troubleshoot and resolve it, so as I guess almost all my colleagues in CSS I quite often install on one of my machines the test apps I got from customers, or make experiments to confirm/deny a theory I have… this is how this story begun.

A while ago, on my Vista x64 machine notepad.exe started to intermittently fail to load and the only "visible" symptom (of course apart the fact that Notepad was not showing up) was a very quick command prompt but unfortunately the content was impossible to read, and I had nothing in the Event Log or any other "obvious" logs. I noticed this problem because the CRM tool we use to track our work on support cases, offer the possibility to review the entire log of a call in notepad, and every now and then I happen to use that function if I want to have a quick recap of what has been done so far, specially if I got the case from another Engineer and I’m not fully aware of all the details.

As I said in my previous post I am a keyboard shortcut addict, and I like to experiment cool tools and utilities which can speed up and make easier the time I spend on a computer (which is a notable amount of hours, as you can guess smile_nerd), and one of them is Slick Run (here and here a couple of posts on it from other MS guys) which I use as a replacement of the Windows "Run" command. Well, I noted that trying to launch notepad from there I got the same behavior described above, but running it from the "Run" window worked just fine… I also noted that also running C:\Windows\System32\notepad.exe (x64 version) from the file system never got a problem, while running C:\Windows\SysWow64\notepad.exe (x86 version) always failed with the same symptoms. I first thought to use adplus to look at the exception, but notepad.exe was not even showing up in the task list smile_thinking; anyway, that was not yet enough annoying to worth invest some time to fix it.

Then a couple of days ago I got completely annoyed by that problem, so I decided to tackle it once for all smile_regular. This time I choose a different approach, so since I was not able to easily attach the debugger to notepad I tried a different path and go directly from within WinDbg (File->Open Executable or CTRL+E) and spawn Slick Run from there, but flagging the checkbox to debug also child processes (Process Explorer shows that Slick Run is always the parent of the processes he launches)

open executable

Going forward step by step (g-> Enter to unlock Slick Run, then spawn notepad.exe, then again g->Enter to unlock notepad etc…) showed the command prompt I first was not able to read, here it is:

cscript output

Ah… what is that? smile_omg There is clearly a typo in that command line…! And, by the way, where does it come from? smile_sarcastic

Ok, that smelt of some debugging test I forgot to remove and it sounds it can come from the Windows Registry. Let’s have a look, just search for that "adlus.vbs" and… there it is!

image file execution options key

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe]
"Debugger"="cscript adlus.vbs -crash -pn notepad.exe"

Need other clues? No, that’s enough for me, thanks smile_shades.

I guess this is something left from one of those debugging experiments I run on my machine (honestly I can’t remember, but since I’m the only one with access to that machine, I must be the culprit here) and for some reason I then forgot to clean up… That registry comes from gflags.exe (also here), like the following

gflags dialog

I deleted the registry key and Notepad.exe was back at work also from the CRM tool and Slick Run smile_regular

I’m not sure if that is the best troubleshooting approach (probably it’s not), I should have thought to gflags from the beginning but, hey, what’s important is the result, isn’t it? smile_wink. I then found why running notepad.exe from Slick Run failed, while it worked fine from Windows Run: Slick Run is a 32 bit process and the Notepad he was trying to execute was the one from the SysWow folder (the one which failed also if executed manually from the file system), executed the one from System32 (64 bit despite the folder name).

Lessons learnt from this experience:

  • Remember to always clean my garbage
  • Carefully check my command lines
  • Do not always look for a funky cause of the problem, sometimes following the instinct pays off, even if it drives you through an unusual path



Comments (4)

  1. MikeO [MSFT] says:

    Hey Carlo. If you like SlickRun, have you seen Start++? (http://blogs.msdn.com/mikeormond/archive/2007/03/29/first-there-was-slickrun-now-there-s-start.aspx). Very nice integration of magic words in the Vista start menu. Mike

  2. Hi Mike, I discovered it from your blog and I’m just downloading it, I’m quite curious 😉

  3. Yesterday I was working on a sample project got from a customer, when Visual Studio 2005 showed this

  4. Yesterday I was working on a sample project got from a customer, when Visual Studio 2005 showed this