Fatal Execution Engine Error on x64 Framework


I got two FEEE cases in a row last week, both of them for the same cause so I thought to blog about it hoping to save some stress and headshakes to someone else.


The affected platform is 64 bit Framework (32 bit works fine), either Web or Windows application: randomly the application crashes and if you check your event log you’ll find messages like the following:



Event Type:      Error
Event Source:   .NET Runtime
Date:                12-02-2009 10:58:36
User:                N/A
Computer:       <computername>
Description: .NET Runtime version 2.0.50727.3082 – Fatal Execution Engine Error (000006427F8A5DC8) (80131506)


The stack of the faulting thread looks like this:


0:025> kpL1000
Child-SP RetAddr Call Site
00000000`0643e350 00000642`78acb013 mscorwks!COMCryptography::_DecryptData+0x329
00000000`0643e590 00000642`801f4b87 mscorlib_ni!System.Security.Cryptography.CryptoAPITransform.TransformFinalBlock(<HRESULT 0x80004001>)+0x123
00000000`0643e620 00000642`801f25c7 CryptoSample!MyCryptoClass.Decrypt(<HRESULT 0x80004001>)+0xf7
00000000`0643e960 00000642`bc8e449b System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute(<HRESULT 0x80004001>)+0x257
00000000`0643ea10 00000642`bc8f2215 System_Web_ni!System.Web.HttpApplication.ExecuteStep(<HRESULT 0x80004001>)+0xab
00000000`0643eab0 00000642`bc8e3553 System_Web_ni!System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(<HRESULT 0x80004001>)+0x1a5
00000000`0643eb60 00000642`bc8e7874 System_Web_ni!System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(<HRESULT 0x80004001>)+0xd3
00000000`0643ebc0 00000642`bc8e745c System_Web_ni!System.Web.HttpRuntime.ProcessRequestInternal(<HRESULT 0x80004001>)+0x1c4
00000000`0643ec50 00000642`bc8e608c System_Web_ni!System.Web.HttpRuntime.ProcessRequestNoDemand(<HRESULT 0x80004001>)+0x7c
00000000`0643ec90 00000642`7f602322 System_Web_ni!System.Web.Hosting.ISAPIRuntime.ProcessRequest(<HRESULT 0x80004001>)+0x18c
00000000`0643edc0 00000642`7f503bb3 mscorwks!CallDescrWorker+0x82
00000000`0643ee20 00000642`7f5251f8 mscorwks!CallDescrWorkerWithHandler+0xd3
00000000`0643eec0 00000642`7f525563 mscorwks!ForwardCallToManagedMethod+0x160
00000000`0643ef60 00000642`7f544738 mscorwks!COMToCLRWorkerBody+0x35b
00000000`0643f1c0 00000642`7f50c8ae mscorwks!COMToCLRWorkerDebuggerWrapper+0x50
00000000`0643f230 00000642`7f60249e mscorwks!COMToCLRWorker+0x366
00000000`0643f520 00000642`fff58293 mscorwks!GenericComCallStub+0x5e
00000000`0643f5d0 00000642`fff58633 webengine!HttpCompletion::ProcessRequestInManagedCode+0x2a3
00000000`0643fa80 00000642`fff9abf4 webengine!HttpCompletion::ProcessCompletion+0x63
00000000`0643fac0 00000642`7f48dc77 webengine!CorThreadPoolWorkitemCallback+0x24
00000000`0643faf0 00000642`7f4a289a mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x157
00000000`0643fb90 00000642`7f41f0ac mscorwks!ThreadpoolMgr::WorkerThreadStart+0x1ba
00000000`0643fc30 00000000`77d6b6da mscorwks!Thread::intermediateThreadProc+0x78
00000000`0643ff80 00000000`00000000 kernel32!BaseThreadStart+0x3a

0:025> !clrstack
OS Thread Id: 0x1258 (25)
Child-SP RetAddr Call Site
000000000643e590 00000642801f4b87 System.Security.Cryptography.CryptoAPITransform.TransformFinalBlock(Byte[], Int32, Int32)
000000000643e620 00000642801f25c7 MyCryptoClass.Decrypt(System.String, System.String, Boolean)
000000000643e960 00000642bc8e449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000643ea10 00000642bc8f2215 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000643eab0 00000642bc8e3553 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
000000000643eb60 00000642bc8e7874 System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
000000000643ebc0 00000642bc8e745c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
000000000643ec50 00000642bc8e608c System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
000000000643ec90 000006427f602322 System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)


If in your code you are calling Convert.FromBase64String (or CryptoAPITransform.TransformFinalBlock directly) assure you are not passing an empty string as the value to convert, for example:


Public Function Decrypt(ByVal Source As String) As String

Dim byteArr As Byte() = System.Convert.FromBase64String(Source)

[…]

End Function


should instead look like this:


Public Function Decrypt(ByVal Source As String) As String
‘convert from Base64 to binary

If IsNullOrEmpty(Source) Then
Return String.Empty
End If

Dim byteArr As Byte() = System.Convert.FromBase64String(Source)

[…]

End Function


Carlo

Quote of the day:
The world is full of people whose notion of a satisfactory future is, in fact, a return to the idealized past. – Robertson Davies

Comments (10)

  1. Dan Davies Brackett says:

    I think there might be a typo in your example.  You have the correct code returning String.Empty when the Source string is not null or empty – thus ensuring that, by the time FromBase64String is called, the string is either null or empty…which would trigger this bug.

  2. Thanks Dan, fixed 🙂

  3. Nathan says:

    THANK YOU for the post! You saved me a lot of headache tracking this down. Kudos!

  4. jimmykang says:

    You said that this does not affect the 32 bit framework. However, I have consistently reproduced it on windows 2003 32bit. However, I only get this error when I am running my asp.net app. under a custom user account (not the network service account). I can switch back and forth and fix/break the app by changing the user account that runs the app pool back and forth between network service and the custom user account.

    All of my other apps run fine under the custom account, but this app crashes with the above mentioned error.

    Oddly, I have noticed that if I am logged in as my custom account when the worker process gets created then the error does not occur. However, if I am not logged in as the custom account and I refresh the worker process, when it gets re-created I have this problem again.

    Just hope this saves someone the 6 hours of headache I spent figuring this out.

    REgards

  5. softlion says:

    Hi,

    could you explain me how you managed to get the .NET crash dump on x64 ?

    I’m having headeaches trying to find out and I didn’t 🙁

    reply to: name@name.com where replace name by softlion

    thks !

  6. Hello, everything is exactly the same as for x86 processes, you only have to use a 64 bit debugger (either adplus/Windbg or DebugDiag): what do you get when you try?

  7. Anand Vinod says:

    Hi Carlo,

    Do you think the Error Code here: 000006427F8A5DC8 has a significance to this particular code? Do you think similar usage of this API can cause a runtime crash with a different error code?

    I'm looking at a similar problem and this error occurs after about 60+hrs of running automation tests on our product software.

    Other links pointed to CustomMarshalers and XSL usage as root cause to this error. But i did not find any references of these in our code. But I do see, usage of FromBase64String without a null or empty check as you have described. Do you think these 2 can be related.

    connect.microsoft.com/…/fatal-execution-engine-error-when-doing-xslt-transform-on-64bit-server

    connect.microsoft.com/…/clr-crashes-on-64bit-platform-when-marshaling-delegates-that-have-custom-marshalled-parameters

    I'm currently running ADPLUS but no luck in reporducing this issue so far. Any help on this topic would be helpful and appreciated.

    Error Description


    Event Type:       Error

    Event Source:    .NET Runtime

    Event Category: None

    Event ID:           1023

    Date:                6/2/2010

    Time:                4:01:43 AM

    User:                N/A

    Computer:         SC2000SC2

    Description:       .NET Runtime version 2.0.50727.3082 – Fatal Execution Engine Error (000006427F44A4EE) (80131506)

    For more information, see Help and Support Center at go.microsoft.com/…/events.asp.

    Thanks,

    Anand

  8. Hello Anand, both code 000006427F8A5DC8 you mention and 000006427F44A4EE I see in the error you pasted mean CDERR_DIALOGFAILURE: does this have anything to do with your application? At first sight I don't this this is related to the XSL issue you mention but if you see encryption of an empty string then this is definitely I would look into… Did you consider opening a call with Microsoft Support? If you're based in EMEA I may be able to take it

  9. Eric says:

    Thank you, THANK YOU!  

    I recently migrated one of our websites to new 2008 R2 (x64) web servers from older x86 2003 web servers.  IIS has been blowing up with "A process serving application pool 'AppPool20' terminated unexpectedly. The process id was '2432'. The process exit code was '0x80131506'."  

    I spent days trying to debug the problem, thinking it was our 3rd party DLLs.  However it turns out it was my own code – ouch! 🙂