Why I should not be writing applications ;)
The first driver I owned when I started at Microsoft in 1997 was i8042prt.sys, the driver that controls your PS2 mouse and keyboard. I had the job of upgrading it from an NT4 legacy style driver to a PnP enabled Windows 2000 driver. Of course my dad asked "didn't keyboards and mice work before you got there? what are you doing over there anyways?" My manager at the time said all I had to do was get the mouse and keyboard working after resume from Sx and that would be it. I ended up owning the driver for 5+ years and not the advertised couple of months ;), but I learned a lot.
One of things I did while owning this driver was unify crash dump support for all builds. Previous to my owning the driver, Microsoft Far East Asia PSS added the ability to crash (aka crash dump) the machine for any given key and modifier combination (see I8xServiceCrashDump() for initialization of state and I8xProcessCrashDump() on processing the keys at runtime in 6000\src\input\pnpi8042 in the WDK for the gory details). This code was under #ifdefs and was only live for far east builds of the driver. This was a universally useful feature and I was asked to make it available for all builds. I decided to make it easy for folks to enable it as well. Configuration was not easy. First, you had to know the scan code for the trigger key (instead of the character itself). Second you had to describe the modifiers using a set of flags that were only documented in the driver's header. Third, you had to know where to put these registry values (which typically meant you also had to create a reg key as well). Yuck, not friendly.
To fix this I created one registry value (CrashOnCtrlScroll) that setup the default crash dump key sequence for you, right ctrl + scroll lock. This KB article describes how to set the key. That worked well ... except that many laptops did not have a right control key! I didn't want to create another key for crashing on left ctrl + scroll lock (or something as arbitrary but completely different) so I created a GUI application that let you type the key and select the modifiers. Not rocket science, but functional. Here is a snapshot
Issue #1: While functional, it's ugly
Issue #2: I named it cdsetup.exe (c[rash]d[ump]setup.exe). Logical name except that many cdrom setup applications are also called cdsetup.exe so there was a bit of confusion. To add to the confusion, I think my version of cdsetup.exe was a part of the Windows 200 resource kit. The one saving grace to this name is that it automatically requests elevation on Vista (probably because it has "setup" its name).
Issue #3: After recompiling the application yesterday to test it on Vista the red X, Cancel and OK buttons all stopped working which meant I could not dismiss the dialog box. Weird. I found an old build and it worked just fine so something I did during the recompile altered the behavior. To get the application to build and run I had to add the following 2 lines to the sources file
USE_MSVCRT=1
_NT_TARGET_VERSION=$(_NT_TARGET_VERSION_WIN2K)
Specifying the taret version has no effect on the runtime so it must have been the conversion to the MS VC runtime instead of the old C runtime that shipped with Windows. I decided to debug the application's DlgProc. It is rather simple:
LRESULT CALLBACK CrashDumpSetup::s_DlgProc(HWND hDlg, UINT message, WPARAM wParam, LPARAM lParam)
{
CrashDumpSetup *cdi;
if (message == WM_INITDIALOG) {
cdi = new CrashDumpSetup(hDlg);
if (!cdi)
return FALSE;
SetWindowLongPtr(hDlg, DWLP_USER, (ULONG_PTR) cdi);
return (LRESULT) cdi->Initialize();
}
cdi = (CrashDumpSetup *) GetWindowLongPtr(hDlg, DWLP_USER);
if (cdi) {
return cdi->DialogProc(message, wParam, lParam);
}
return FALSE;
}
LRESULT CrashDumpSetup::DialogProc(UINT message, WPARAM wParam, LPARAM lParam)
{
switch (message) {
case WM_COMMAND:
switch (LOWORD(wParam)) {
case IDC_DUMP_DISABLE:
case IDC_DUMP_DEFAULT:
case IDC_DUMP_ALT:
ToggleAltUI(LOWORD(wParam) == IDC_DUMP_ALT);
break;
case IDOK:
if (Apply() == FALSE) {
return FALSE;
}
__fallthrough;
case IDCANCEL:
delete this;
EndDialog(m_hDlg, LOWORD(wParam));
break;
}
}
return FALSE;
}
When pressing the red X, Cancel or OK keys my WM_COMMAND processing code ran and called EndDialog() as I expected, but like I said before, the dialog didn't dismiss itself. I looked at the docs for EndDialog and saw that it returns a BOOL, so I rewrote EndDialog to capture the return value and it was returning FALSE! Ugh. Since I was single stepping through the function, !gle came to the rescue
0:000> !gle
LastErrorValue: (Win32) 0x578 (1400) - Invalid window handle.
LastStatusValue: (NTSTATUS) 0xc0000034 - Object Name not found.
An invalid window handle? m_hDlg worked before, maybe some component was overwriting it. So dumped the variable and then the hDlg from s_DlgProc,
0:000> dt this m_hDlg
Local var @ 0x1ff798 Type CrashDumpSetup*
0x00a01f88
+0x000 m_hDlg : 0x00a064d0 HWND__
0:000> .f+
01 001ff7cc 775af512 cdsetup!CrashDumpSetup::s_DlgProc+0x8f [d:\work\cdsetup\cdsetup.cpp @ 576]
0:000> dt hDlg
Local var @ 0x1ff7d4 Type HWND__*
0x004e05f0
+0x000 unused : 49595140
Looking at the 2 values, it is obvious that m_hDlg was altered. But how? After staring at the code I wrote 7 years ago, I thought about what I changed (using the MS C runtime) and I immediate saw it. I was deleting the object and then touching freed memory. The old C runtime left the freed memory as it was before itw as freed so the old code had the bug the entire time, it was just that the new runtime caught made the mistake apparent immediately. The fix was simple, capture the value before deleting the object and then calling EndDialog (I guess I could have called EndDialog in the destructor but that made cleanup in the failure to initialize the object potentially problematic). After applying the fix, all of the buttons now worked as expected ;).
case IDCANCEL:
HWND h = m_hDlg;
delete this;
EndDialog(h m_hDlg, LOWORD(wParam));
break;