Getting 64 bit Vista to open my Inbox the way I want it to
A bit over a year ago I had to figure out why my Mail key started behaving differently on Vista vs XP and wrote about how I fixed it. Well, my dev box was so slow that I was able to employ enough sympathy that I got a new one. While my old box was a 32 bit machine, the new one came preinstalled with 64 bit Vista. Cool. I finally had a 64 bit box that was powerful enough to develop on (previously all I had a very noisy 64 bit test machine).
When I get a new machine that I will be using for dev work I do the following:
- Blow away the preinstalled OS and install the latest OS release (beta or RTM) that I can do work on. In this case, I left the OS alone ;).
- Enlist in windows sources
- Install the latest WDK
- Install Office Pro and Visio Pro (for the lovely WDF pnp/power/power policy state machines)
- Apply the mail key registry change so Outlook works the way I want it to.
Well steps 1 through 4 went by flawlessly and I updated the registry to fix the mail key. All happy with myself I made the change and pressed the key. Nothing happened. The focus did not change nor did Outlook move to the default folder. Phooey. I double checked the changes in the registry and lo and behold I mistyped the value name and fixed it. Pressed the mail key. Still nothing happened. Rats .
So now my only recourse was to debug it ;). First searched for "RegisteredApp" in the source tree (via content indexing) to find the function which processed the WM_APPCOMMAND message. Once found, I traced it into the internal shlwapi function which looked up the appropriate locations in the registry and traced through it as well. Not the most enjoyable task either....first, I do not own the code so I am figuring out it intent while determining if it is behaving correctly...and second, it is optimized code so debugging with source was not an option. Thank god it was not IA64 assembly though. Phew. The one positive through this part of the debugging was that it was 100% reproducible, so when I got lost, I could just hit 'g' and press the mail key again to restart.
After looking at this for a half an hour or so, the logic was this
- Query for the default mail client in HKCU
- if (WIN64) { If that failed, 1) open the default mail client in the 32 bit HKLM software key and 2) query for the default mail client }
- If that failed, query for the default mail client in the normal HKLM software key (which would be the 64 bit HKLM on 64 bit, the 32 bit HKLM on 32 bit).
Upon successfully querying for the default mail client, attempt to execute its open verb as registered in the key 64 bit HKLM\Software\Clients\Mail\<application>\shell\open\command. If this failed, attempt the same thing under the 32 bit version of the key.
The bug was that the result of the 2nd bullet (query on the 32 bit key on 64 bit machines) was being misinterpreted. The code treated the return value as if a RegQueryValueEx call was made, but the call was to RegOpenKeyEx. Because the return value was not being evaluated correct, step 2) was never performed (querying for the client on the new key) which led to an error when formatting the string that was used to query for the verb.
OK, great. Now at least I knew where the bug was, but I still wanted my mail key to work and I was definitely not going to run my own custom shlwapi.dll on my own machine. It is used by too many components for me to have confidence in making that kind of change. I needed a solution that was as minimal as possible and the least invasive. In the end I ended up attaching windbg and using the assembler (the 'a' command) to remove the incorrect checks. My initial removal was to nop the entire sequence...
0:004> u xxxxxxxxx`xxxxxx6c (somewhere in the guts of shlwapi.dll)
$ the call whose result will be misinterpreted
xxxxxxxxx`xxxxxx6c ff153e08fcff call qword ptr [SHLWAPI!_imp_RegOpenKeyExW (000007fe`fee918b0)]
$ incorrect tests
xxxxxxxxx`xxxxxx72 85c0 test eax,eax
xxxxxxxxx`xxxxxx74 8bf8 mov edi,eax
xxxxxxxxx`xxxxxx76 7474 je xxxxxxxxx`xxxxxxec
xxxxxxxxx`xxxxxx78 3bc3 cmp eax,ebx
xxxxxxxxx`xxxxxx7a 742c je xxxxxxxxx`xxxxxxa8
$ call to query the value
xxxxxxxxx`xxxxxx7c 488b4c2438 mov rcx,qword ptr [rsp+38h]
xxxxxxxxx`xxxxxx81 4c8d4c2430 lea r9,[rsp+30h]
xxxxxxxxx`xxxxxx86 4c8d442440 lea r8,[rsp+40h]
xxxxxxxxx`xxxxxx8b 33d2 xor edx,edx
xxxxxxxxx`xxxxxx8d c7442430a0000000 mov dword ptr [rsp+30h],0A0h
xxxxxxxxx`xxxxxx95 ff150d08fcff call qword ptr [SHLWAPI!_imp_RegQueryValueW (000007fe`fee918a8)]
$ now nop out the check, always do the query. Stop @ xxxx7c since that is the start of the query call
0:004> a xxxxxxxxx`xxxxxx72
xxxxxxxxx`xxxxxx72 nop
xxxxxxxxx`xxxxxx73 nop
xxxxxxxxx`xxxxxx74 nop
xxxxxxxxx`xxxxxx75 nop
xxxxxxxxx`xxxxxx76 nop
xxxxxxxxx`xxxxxx77 nop
xxxxxxxxx`xxxxxx78 nop
xxxxxxxxx`xxxxxx79 nop
xxxxxxxxx`xxxxxx7a nop
xxxxxxxxx`xxxxxx7b nop
$ and dump out the old code again to make sure the right thing was nop'ed
0:004> u xxxxxxxxx`xxxxxx6c
xxxxxxxxx`xxxxxx6c ff153e08fcff call qword ptr [SHLWAPI!_imp_RegOpenKeyExW (000007fe`fee918b0)]
xxxxxxxxx`xxxxxx72 90 nop
xxxxxxxxx`xxxxxx73 90 nop
xxxxxxxxx`xxxxxx74 90 nop
xxxxxxxxx`xxxxxx75 90 nop
xxxxxxxxx`xxxxxx76 90 nop
xxxxxxxxx`xxxxxx77 90 nop
xxxxxxxxx`xxxxxx78 90 nop
xxxxxxxxx`xxxxxx79 90 nop
xxxxxxxxx`xxxxxx7a 90 nop
xxxxxxxxx`xxxxxx7b 90 nop
xxxxxxxxx`xxxxxx7c 488b4c2438 mov rcx,qword ptr [rsp+38h]
xxxxxxxxx`xxxxxx81 4c8d4c2430 lea r9,[rsp+30h]
xxxxxxxxx`xxxxxx86 4c8d442440 lea r8,[rsp+40h]
xxxxxxxxx`xxxxxx8b 33d2 xor edx,edx
xxxxxxxxx`xxxxxx8d c7442430a0000000 mov dword ptr [rsp+30h],0A0h
xxxxxxxxx`xxxxxx95 ff150d08fcff call qword ptr [SHLWAPI!_imp_RegQueryValueW (000007fe`fee918a8)]
...but my final (and more refined) approach was the actual comparisons in the assembly. After the in place edits, my mail key now works again! The only issue is that I have to reapply every time explorer is started, I can live with that. I also filed a bug for the next release of Windows, so I will not have to make the edits in the future and neither will you ;).