Update: An official fix for this issue has been released to Windows Update, although I must say that I think my patch has more style than the official one. You do not need to patch your binary. Just keep your copy of Windows 8 up to date and you'll be fine.
For the five remaining Microsoft Money holdouts (meekly raises hand), here's a patch for a crashing bug during import of account transactions or when changing a payee of a downloaded transaction in Microsoft Money Sunset Deluxe. Patch the mnyob99.dll file as follows:
- File offset 003FACE8: Change 85 to 8D
- File offset 003FACED: Change 50 to 51
- File offset 003FACF0: Change FF to 85
- File offset 003FACF6: Change E8 to B9
Note that this patch is completely unsupported. If it makes your computer explode or transfers all your money to an account in the Cayman Islands, well, too bad for you.
If you are not one of the five remaining customers of Microsoft Money, this is a little exercise in application compatibility debugging. Why application compatibility debugging? Because the problem seems to be more prevalent on Windows 8 machines.
Note that I used no special knowledge about Microsoft Money. All this debugging was performed with information you also have access to. It's not like I have access to the Microsoft Money source code. And I did this debugging entirely on my own. It was not part of any official customer support case or anything like that. I was just debugging a crash that I kept hitting.
The crash occurs in the function utlsrf08!DwStringLengthA:
utlsrf08!DwStringLengthA:
push ebp
mov ebp,esp
mov eax,dword ptr [ebp+8]
lea edx,[eax+1]
again:
mov cl,byte ptr [eax]
inc eax
test cl,cl
jne again
sub eax,edx
pop ebp
ret 4
The proximate cause is that the string pointer in eax
is garbage.
If you unwind the stack one step, you'll see that the
pointer came from here:
lea eax,[ebp-20Ch]
push eax
call dword ptr [__imp__GetCurrentProcessId]
push eax
push offset "Global\TRIE@%d!%s"
lea eax,[ebp-108h]
push 104h
push eax
call mnyob99!DwStringFormatA
add esp,14h
lea eax,[ebp-2E4h]
push eax
push 5Ch
push dword ptr [ebp-2E4h] ; invalid pointer
call mnyob99!DwStringLengthA
sub eax,7
push eax
lea eax,[ebp-101h]
push eax
jmp l2
l1:
mov eax,dword ptr [ebp-2E4h]
mov byte ptr [eax],5Fh
lea eax,[ebp-2E4h]
push eax
push 5Ch
push dword ptr [ebp-2E4h]
call mnyob99!DwStringLengthA
push eax
push dword ptr [ebp-2E4h]
l2:
call mnyob99!FStringFindCharacterA
cmp dword ptr [ebp-2E4h],edi
jne l1
I was lucky in that all the function calls here were to imported
functions, so I could extract the names from the imported function table.
For example, the call to DwStringFormatA was originally
call mnyob99!CBillContextMenu::SetHwndNotifyOnGoto+0x1e56a (243fc3cc)
But the target address is an import stub:
jmp dword ptr [mnyob99+0x1ec0 (24001ec0)]
And then I can walk the import table to see that this was the import
table entry for utlsrf08!DwStringFormatA.
From the function name, it's evident that this is some sort of
sprintf-like function.
(If you disassemble it, you'll see that it's basically
a wrapper around
vsnprintf.)
Reverse-compiling this code, we get
char name[...];
char buffer[MAX_PATH];
char *backslash;
...
DwStringFormatA(buffer, MAX_PATH, "Global\\TRIE@%d!%s",
GetCurrentProcessId(), name);
// Change all backslashes (except for the first one) to underscores
if (FStringFindCharacterA(buffer + 7, DwStringLengthA(backslash) - 7,
'\\',&backslash))
{
do {
*backslash = '_'; // Change backslash to underscore
} while (FStringFindCharacterA(backslash, DwStringLengthA(backslash),
'\\',&backslash));
}
(Remember, all variable names are made-up since I don't have source code access. I'm just working from the disassembly.)
At this point, you can see the bug:
It's an uninitialized variable at the first call to
StringFindCharacterA.
Whether we crash or survive is a matter of luck.
If the uninitialized variable happens to be a pointer
to readable data,
then the
DwStringLengthA will eventually
find the null terminator,
and since in practice the string does not contain
any extra backslashes,
the call to
FStringFindCharacterA fails,
and nobody gets hurt.
But it looks like their luck ran out, and now the uninitialized variable contains something that is not a valid pointer.
The if test should have been
if (FStringFindCharacterA(buffer + 7, DwStringLengthA(buffer) - 7,
'\\',&backslash))
This means changing the
push dword ptr [ebp-2E4h]
to
lea eax,[ebp-101h]
push eax
Unfortunately, the patch is one byte larger than the existing code, so we will need to get a little clever in order to get it to fit.
One trick is to rewrite the test as
if (FStringFindCharacterA(buffer + 7, DwStringLengthA(buffer + 7),
'\\',&backslash))
That lets us rewrite the assembly code as
lea eax,[ebp-2E4h]
push eax
push 5Ch
lea eax,[ebp-101h] ; \ was "push dword ptr [ebp-2E4h]"
push eax ; /
call mnyob99!DwStringLengthA ; unchanged but code moved down one byte
nop ; \ was "sub eax,7" (3-byte instruction)
nop ; /
push eax
lea eax,[ebp-101h]
push eax
The new instructions (lea and push)
are one byte larger than the original push,
but we got rid of the three-byte sub eax, 7,
so it's a net savings of two bytes, which therefore fits.
However, I'm going to crank the nerd level up another notch and try to come up with a patch that involves modifying as few bytes as possible. In other words, I'm going for style points.
To do this, I'm going to take advantage of the fact that
the string length is the return value of
DwStringFormatA,
so that lets me eliminate the call to
DwStringLengthA altogether.
However, this means that I have to be careful not to
damage the value in eax
before I get there.
lea ecx,[ebp-2E4h] ; was "lea eax,[ebp-2E4h]"
push ecx ; was "push eax"
push 5Ch
nop ; \
nop ; |
nop ; |
nop ; | was "push dword ptr [ebp-2E4h]"
nop ; |
nop ; /
nop ; \
nop ; |
nop ; | was "call mnyob99!DwStringLengthA"
nop ; |
nop ; /
sub eax,7
push eax
lea eax,[ebp-101h]
push eax
Patching the lea eax, ... to be
lea ecx, ... can be done with a single byte,
and the push eax is a single-byte instruction
as well, so the first two patches can be done with one byte each.
That leaves me with 11 bytes that need to be nop'd out.
The naïve way of nopping out eleven bytes is simply
to patch in 11 nop instructions,
but you can do better by taking advantage of the bytes
that are already there.
ffb51cfdffff push dword ptr [ebp-2E4h] 85b51cfdffff test dword ptr [ebp-2E4h],esi e8770a0000 call mnyob99!DwStringLengthA b9770a0000 mov ecx,0A77h
By patching a single byte in each of the two instructions,
I can turn them into effective nops by making them do nothing
interesting.
The first one tests the uninitialized variable against some garbage bits,
and the second one loads a unused register with a constant.
(Since the ecx register is going to be trashed by the call to
FStringFindCharacterA,
we are free to modify it all we want prior to the call.
No code could have relied on it anyway.)
That second patch is a variation of one
I called out some time ago,
except that instead of patching out the call with a
mov eax, immed32,
we're using a
mov ecx, immed32,
because the value in the eax register is still important.
Here's the final result:
lea ecx,[ebp-2E4h] ; was "lea eax,[ebp-2E4h]"
push ecx ; was "push eax"
push 5Ch
test dword ptr [ebp-2E4h],esi ; was "push dword ptr [ebp-2E4h]"
mov ecx,0a77h ; was "call mnyob99!DwStringLengthA"
sub eax,7
push eax
lea eax,[ebp-101h]
push eax
Bonus chatter: When I shared this patch with my friends, I mentioned that this patch made me feel like my retired colleague Jeff, who had a reputation for accomplishing astonishing programming tasks in his spare time. You would pop into his office asking for some help, and he'd fire up some program you'd never seen before.
"What's that?" you'd ask.
"Oh, it's a debugger I wrote," he'd calmly reply.
Or you'd point him to a program and apologize, "Sorry, I only compiled it for x86. There isn't an Alpha version."
"That's okay, I'll run it in my emulator," he'd say, matter-of-factly.
(And retiring from Microsoft hasn't slowed him down. Here's an IBM PC Model 5150 emulator written in JavaScript.)
Specifically, I said, "I feel like Jeff, who does this sort of thing before his morning coffee."
Jeff corrected me. "If this was something I used to do before coffee, that probably meant I was up all night. Persistence >= talent."
It's almost like Microsoft's code is open source after all! :)
Wow, my nerd-o-meter just exploded.
I have an unopened copy of Microsoft Money that came with a new computer many years ago. Is it worth anything?
Now this is the type of stuff I really find interesting!
For abybody who tried to follow and got lost in presentation, here are the Raymond's four bytes changes in context:
push offset "GlobalTRIE@%d!%s"
lea eax,[ebp-108h]
push 104h
push eax
call mnyob99!DwStringFormatA
add esp,14h
lea eax,[ebp-2E4h] ; > lea ecx,[ebp-2E4h] (85 > 8D)
push eax ; > push ecx (50 > 51)
push 5Ch
push dword ptr [ebp-2E4h] ; > test [ebp-2E4h],esi (FF > 85)
call mnyob99!DwStringLengthA ; > mov ecx, whatever (E8 > B9)
Raymond, is there any advantage in trying to reduce the number of changed bytes, once they are on the non-consecutive locations?
I tried to guess what the assembly changes were going to be based on the 4 changes listed at the top of the file by looking at an opcode map (pdos.csail.mit.edu/…/appa.htm).
"push eax" ==> "push ecx" looked good. "test" ==> "lea" and "call" ==> "mov ecx" looked unlikely but possible. I had no idea what to make of "Indirect Grp5" ==> "test" so I decided I must have been wrong and that it was more likely modifying non-initial bytes to change a register or immediate value. Turns out I was right.
Why change whole bytes at the time? Please don't do unnecessary bit-inverting work.
I would expect this fragment to apply:
Q: Why?
A: There is no why.
"Microsoft Money crashes"
/ Rushes to check MSFT
Indeed, crashes. Probably due to Sinofsky's departure.
Incredibly cool, Raymond! Good work, and a very good blog post which details the steps and the required knowledge. Totally awesome.
Nice and really cool, thank you for sharing!
We're on to your retirement plan, Raymond!
What would the patch look like to initialize the variable; that is, inject a "backslash = &buffer[0]" line before the first FStringFindCharacterA call?
Maurits, to add nitialization and keep the rest of the code unchanged you'd have to find some unsused bytes which aren't there — you'd hate your compiler if it would insert unnecessary spaces all around the code. To make a simple patch Raymond had to find the way to modify the existing bytes and preserve semantics, and he was able to do this because he fully understood what was going on.
Reverse-engineering is a violation of the Microsoft Money EULA. Most likely, Microsoft is unwilling to retain employees who commit crimes against the company. I'm waiting for Raymond to be fired in 5…4…3…2…
Wouldn't be simpler to replace the 11 nops by an also two byte short jmp?
@EduardoS: "Simpler" does not fall into the criteria.
@Akk: There are certain court rulings about warranty of merchantability that override the EULA in situations like this one.
Raymond, thanks for sharing this. I've done things like this in the past, and always felt like I was the only nerdy person who'd take the time to do it. I'm glad I'm not alone. ;)
Who uses the last Money? Money 2004 Small Business Edition was the last good version before MS randomly started deleting useful features (and evil Vista era began). (money.mvps.org/…/removed_items.aspx)
I know some people who will wish they had heard of the Sunset edition before today. Excellent that this was provided – I have an irreplaceable application from elsewhere, that is the only one of its kind, where the company went belly-up and left it in "online activation hell". Not so with Microsoft.
EduardoS: you could do that, but that's more work than just NOPing the instructions out.
Guess I'm glad I never upgraded — I'm still running MS-Money 2000 on a XP VM. IMHO, the last usable version of money, even with all of it warts. :)
"I was lucky in that all the function calls here were to imported functions, so I could extract the names from the imported function table."
So you didn't even have symbols?
great article Raymond. I've had to patch binaries via hex editor a few times with legacy enterprise apps that had long forgotten source code & developers. for a geek it's an ultra satisfying accomplishment :) Chris Jackson from MS app compat previously told me youre freakingly amazing with a debugger…would love to see you in action, maybe a YouTube debugging session :)
@Yuhong Bao: Raymond probably has access, but as he said, he didn't use it.
@Malcolm McCaffery: Excellent suggestion, but I don't expect we'd ever see it.
Note that Money Plus Sunset can be freely downloaded, doesn't need to be installed as an upgrade, and doesn't need to be activated. :-)
And, yes, those are very cool patching steps!
Maybe I missed it somehow, but were there any clues for why the fault happened more on Win8 systems (like a change to ASLR or something)?
@Rick C, NOPing requires patching more bytes!
I am one of the Microsoft Money Five.
I think you could save a byte by patching the 'call mnyob99!DwStringLengthA' to point at the 'ret 4' instead, but it doesn't matter. Those crazy random opcodes get infinite style points.
Hi Raymond,
I just want you to know that 4 years ago , you motivated me to drop medicine and do Computer science.
I'm really happy now, I'm Brazilian and got a scholarship to study here(Seattle) for a year, and it is almost ending my time here.
A chance to take a photo with you would represent a lot to me !
(my name at uw.edu )
Raymond,
Please reconsider publishing a video capturing one of your debugging sessions. Even if you just mumble a bit to yourself to give some vague insight and context, I'm sure there are any number of us who would watch an eight hour video. We could always jump around in the video timeline, after all. :-)
I've really enjoyed watching things like Notch's Java coding sessions, and I'm not even a Java programmer. It still provided tremendous insight into his approach to coding and how he uses the tools available to him.
I use Money 2003. The later versions were changed so they would operate for only 18 months or so (maybe 2 years) before they shut down and HAD to be replaced. (Actually, some of Money 2003's features may have stopped operating after 1 or 2 years, like importing stock quotes, but I don't use those features.)
Money 2003 runs fine on 64-bit Windows 7. I considered going to the "sunset" version, and I would have, if I had been on a *later* version than 2003 that would have expired!
I just want to comment that for anyone what to try doing something like this, check inside the debugger for previous ecx value before using it and beware of loops.
(Okay… for anyone who're able to do this, they must already know they have to check that)
Nerdiest byte sequence that I ever thought up was: EB 03 C3 yy xx
If you create a .COM file with those 5 bytes as the first ones, and look at the disassembly (I'm assuming an x86 disassembler here..), you'll see 'JMP SHORT 3', followed by 3 garbage bytes. Nothing ususual here…
If you look at a Z80 disassembly of those same bytes, that translates to 'EX DE,HL; INC BC;'. Exchange the content of 2 registers, increment another one, nothing special. The 3rd byte is 'JUMP' followed by the 16-bit address specified as yy xx.
So, if you create a .COM file with the 5 bytes as above, followed by 8088 code, followed by Z80 code (at offset xxyy + 0x100 in the source), you'll have a .COM file that runs on MS-DOS and (i.e.) CP/M…
Best would be a compatibility setting that required no modification to the source. But that happens all the time for supported products, I suppose.
I'm still using Money 99. It's the last version sold in Germany and it works pretty well on Windows 7. ;-)
There's now a discussion on this over at Hacker News: news.ycombinator.com/item
Notable is the personal thanks to Raymond Chen at news.ycombinator.com/item .
@thespiral if Raymond Chen were to post a video of him debugging, it wouldn't be eight hours long; it would be thirty seconds, and everyone would just spend eight hours re-watching it to try to understand what he did (and how he did it so fast).
I say this because I've watched him debug an internal issue in a minute, and spent half my day looking back through the debugger console to learn how he figured out the issue.
My brain just melted and is now in a puddle on the desk in front of me.
Nice, Raymond =) I've actually done assembly language patches like this to our code in an emergency to fix a critical bug. It's advisable to *not* fix things this way from a maintainability standpoint, of course, but sometimes things happen and you need it fixed in a hurry.
Count me as another Money Sunset user and kudos to Microsoft for providing this. I did run into a bizarre bug with a charting feature the other day, but there was a work-around, so all is good.
As a hardcore debugging n00b, just wondering.
How do you determine what hex bytes changes to make based on the assembler changes? I've used WinDbg, but can you actually type replacement assembler code in WinDbg and see what resultant memory bytes changes are made? Or do you need to actually know how each assembler statement maps to opcodes?
Jeff, WinDbg comes with a (simple) built-in assembler.
You can type "a <address" and start entering assembler commands. Type an empty line when you're done. Then you can use 'db' to view the assembled opcodes. Another trick people use is ".dvalloc <size>", which allocates memory in the current process. You can use this memory buffer to insert some code using 'a', instead of trashing real code in a process. (I usually use notepad as my target for this sort of thing, but any app will do.) Notepad has a nice function, notepad!InsertDateTime, that is called whenever you hit F5 (to insert a timestamp into a file), which you can use to test code you hack up in this way…
Writing a tool to take a set of binary patches like this, and a target binary file, and generate an Application Compatibility Database (.sdb) file to apply the patches would be fun. I don't think there are any public tools to do this, but internal tools exist at Microsoft. (Some of the security FixIt's use this approach…)
The APIs and data types are described here: msdn.microsoft.com/…/bb432182(v=vs.85).aspx
@Jeff: You can also go straight to the source. Similar manuals exist for other architectures.
Intel® 64 and IA-32 Architectures Software Developer’s Manual, Instruction Set Reference:
download.intel.com/…/325383.pdf
Before the call to FStringFindCharacterA() there are 5 pushes, but the pseudo C code for FStringFindCharacterA() only takes 4 arguments:
lea eax,[ebp-2E4h]
push eax <<<<
push 5Ch <<<<
push dword ptr [ebp-2E4h] <<<<
call mnyob99!DwStringLengthA
sub eax,7
push eax <<<<
lea eax,[ebp-101h]
push eax <<<<
What happens to the push of "dword ptr [ebp-2E4h]", as DwStringLengthA() doesn't appear to pop it off? (Sorry, but my assembly skills are very basic!)
@decka: That push is the one argument to DwStringLengthA. The compiler here has interleaved the 4 pushes of the arguments to FStringFindCharacterA and the 1 push of the argument to DwStringLengthA.
It is not working, I need support:
It made my computer explode and transferred all my money to an account in the Cayman Islands!