How Long Can the Command Line Argument Be for a Shim on Windows Vista?

I had a comment come up on an earlier post:

“FYI, I’ve hit a limitation with CorrectFilePaths – the maximum amount of data that can be entered in the parameter field is 512 bytes – not enough to fix more than 2 files under “Program Files” unfortunately…”

Now, I knew the limitation existed, but I had never spent the time to understand the exact limitation, or where it existed. Is the limitation in Compatibility Administrator when it generated the XML for the SDB compiler? If so, we could always be sneaky and edit the XML before the compiler was run. Is the limitation in the SDB compiler? It’s a lot more work, but if so we could always be sneaky and reverse the SDB format to add additional information. Or is the limitation in the shim engine itself, which makes being sneaky quite challenging indeed?

And the answer is that the limitation is in the shim engine itself. We can see that with a bit of reverse engineering.

The API that the shim engine calls happens to be publicly documented, so that makes our job easier. It’s SdbReadStringTag. Here’s where we actually pull the command line:

0x6C105914: 6800040000             PUSH        0x400           
0x6C105919: 56                     PUSH        ESI             
0x6C10591A: 50                     PUSH        EAX             
0x6C10591B: FF7508                 PUSH        DWORD PTR [hSDB <Void *>]; ARG:EBP+0x8
0x6C10591E: 66891E                 MOV         WORD PTR [ESI],BX_IS_ZERO
0x6C105921: E865260000             CALL        __imp__SdbReadStringTagRef@16; (0x6C107F8B) ; SdbReadStringTagRef

Since this is a __stdcall function, arguments are passed right to left. The first PUSH, therefore, corresponds to __in DWORD cchBufferSize, and this is what limits the command line.

So, we know that the command line is limited to 0x400 characters (1,024), and the limitation is implemented in the shim engine itself.

Shims that require longer command lines, such as CorrectFilePaths, you therefore need to be careful with. If you can shorten the command line by only taking the portion of it you need, that is helpful. If you can leverage file and registry virtualization for fixes, that leaves you more of your 1,024 characters to work with as well!