regcap.exe crashes while building a setup project in Visual Studio 2008

  • Article

regcap.exe crashes while building a setup project in Visual Studio 2008

I launched the devenv.exe in the debugger (https://www.microsoft.com/whdc/devtools/debugging/installx86.Mspx) and took a crash dump (https://support.microsoft.com/kb/286350) while building the setup project.

From the crash dump, I found the following information:

0:000> kpnL
# ChildEBP RetAddr
00 0012f2e8 7c90de7a ntdll!KiFastSystemCallRet(void)
01 0012f2ec 7c801e3a ntdll!ZwTerminateProcess(void)+0xc
02 0012f2fc 7858cc2e kernel32!TerminateProcess(void * hProcess = 0xffffffff, unsigned int uExitCode = 0xc0000417)+0x20
03 0012f638 78556bf4 msvcr90!_invoke_watson(wchar_t * pszExpression = 0x00000000 "", wchar_t * pszFunction = 0x00000000 "", wchar_t * pszFile = 0x00000000 "", unsigned int nLine = 0, unsigned int pReserved = 0)+0x11b
04 0012f660 467c1544 msvcr90!strcat_s(char * _Dst = 0x0012f8a8 "", unsigned int _SizeInBytes = 0x104, char * _Src = 0x00362b48 "C:\Documents and Settings\xxx\.........\bin\Release\App.tlb")+0x2c
05 0012f9b0 467c172f lregdll!DoRegistration(char * pszTLibName = 0x00362b48 "C:\Documents and Settings\xxx\.........\bin\Release\App.tlb", char * pszHelpDir = 0x0012f9c8 "???")+0x77
06 0012f9c8 00402270 lregdll!LRegTLib(char * szFileName = 0x00362b48 "C:\Documents and Settings\xxx\.........\bin\Release\App.tlb", struct InjectedDLLInformation * pOptionInfo = x0012fa48)+0xde
07 0012f9e0 00401de4 regcap!StartProcess(ProcessToExecute Process = RegTypeLib (1), char * szFileName = 0x00362b48 "C:\Documents and Settings\xxx\.........\bin\Release\App.tlb", char * szArgumentsToPassToExe = 0x00362c58 "", struct InjectedDLLInformation * pOptionInfo = 0x0012fa48)+0x48
08 0012ff7c 00402741 regcap!main(int argc = 5, char ** argv = 0x00362a10)+0x4e5
09 0012ffc0 7c817077 regcap!__tmainCRTStartup(void)+0x10f
0a 0012fff0 00000000 kernel32!BaseProcessStart(<function> * lpStartAddress = 0x00402893)+0x23

0:000> .frame 05
05 0012f9b0 467c172f lregdll!DoRegistration+0x77

0:000> dv /v
0012f9b8 pszTLibName = 0x00362b48 "C:\Documents and Settings\xxx\.........\bin\Release\App.tlb"
0012f9b0 pszHelpDir = 0x0012f9c8 "???"
0012f698 ptlib = 0x774ec060
0012f6a0 puszTLibName = wchar_t [260] ""
0012f8a8 szFileName = char [260] ""

There was a file App.tlb in the project located at "C:\Documents and Settings\xxx\.........\bin\Release\App.tlb"
The path to this file was very long so it was hitting some sort of buffer overflow. Shortening the path resolved the problem (eg. c:\test\App.tlb)

N.B.: Due to compatibility issue, we may find such issues also. In those scenarios please go into the compatibility settings for regcap.exe and make the following changes:

1. Set the tool(Regcap.exe) to run as Administrator.
2. Set the tool(Regcap.exe) to Compatibility mode for Windows Vista SP2 or Windows XP SP3.

 

image