Mixed Mode MFC Application may fail with ASSERTION when individual files are compiled with /clr switch

  • Article

Some time back we were working with a developer who was running into a mysterious ASSERTION failure when registering his MFC OCX control which was working fine for quite a few years. The change he made was to compile one of his source file with /clr switch to use some of the managed constructs. His idea was not to disturb the other part of his project but leverage some manage code only in one file. He was getting below ASSERT failure dialog when registering the OCX

clip_image001

Digging into the details, we figured out that in a Mixed Mode MFC application the initialization of MFC module state(with things such as the resource handle) is not complete, until the CLR is initialized. In this developer's case when registering the OCX the DllRegisterServer() entry point is invoked, which in turns tries to use an MFC resource handle (afxCurrentResourceHandle ) which was not yet initialized. Below is the callstack for the assertion dialog

USER32!NtUserWaitMessage
USER32!DialogBox2
USER32!InternalDialogBox
USER32!SoftModalMessageBox
USER32!MessageBoxWorker
USER32!MessageBoxTimeoutW
USER32!MessageBoxTimeoutA
USER32!MessageBoxExA
USER32!MessageBoxA
MSVCR90D!__crtMessageBoxA
MSVCR90D!__crtMessageWindowA
MSVCR90D!_VCrtDbgReportA
MSVCR90D!_CrtDbgReportV
MSVCR90D!_CrtDbgReport
mfc90d!AfxAssertFailedLine
mfc90d!AfxGetResourceHandle
mfc90d!AfxFindStringResourceHandle
mfc90d!StrTraitMFC_DLL<char,ATL::ChTraitsCRT<char> >::FindStringResourceInstance
mfc90d!ATL::CStringT<char,StrTraitMFC_DLL<char,ATL::ChTraitsCRT<char> > >::LoadStringA
mfc90d!AfxOleRegisterPropertyPageClass
mfc90d!AfxOleRegisterPropertyPageClass
<Developer's Module>!<Developer's Class>::UpdateRegistry
mfc90d!COleObjectFactory::UpdateRegistryAll
<Developer's Module>!DllRegisterServer
regsvr32!wWinMain
regsvr32!__wmainCRTStartup
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

Now what could be the work around? The neat work around is to build the whole project with /clr switch. If you cannot do that at all consider the following.

  1. As described above the root cause of the issue is that CLR is not initialized yet and it wont unless some method in the module which is compiled with/clr is invoked. So one could compile the source that has the entry point method also with /clr switch. In this developer's case it the file containing the definition of DllRegisterServer() which is invoked by the regsvr32.exe
  2. The other work around is to declare a dummy method that does nothing in the source file which is compiled with /clr switch and invoke that method from the entry point functions of the binary (In this case from DllRegisterServer())

Eg:-

// part of source file that defines DllRegisterServer()

DllRegisterServer()

{
     // This will cause the CLR to initialize the managed CRT/MFC startup 
      CLRLoadDll();
}

// part of source file built using /CLR

void CLRLoadDll()

{
        return;
}