Scenario 1 : ZipEngine role is looping between Restarting and Busy state throwing an unhandled exception: Could not load file or assembly - An attempt was made to load a program with an incorrect format
Referring to my blog on Azure Cloud Service Troubleshooting Series, this is the 1st scenario of the lab. Please make sure you have followed the lab setup instructions for Compressor application as per this, to recreate the problem.
Symptom
ZipEngine role instance of Compressor application is constantly looping between Restarting and Busy state throwing the below unhandled exception in the azure portal blade:
Unhandled Exception: Could not load file or assembly 'WorkerAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format. at ZipEngine.WorkerRole.OnStart() at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum) at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.
<InitializeRole>
b__0() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart() ' [2018-08-12T11:28:39Z] Last exit time: [2018/08/12, 11:28:39.434].
Troubleshooting
If your role does not start, or is recycling between the initializing, busy, and stopping states, your code may be throwing an unhandled exception within one of the lifecycle events each time the role restarts. So if you look into the above callstack carefully you would notice that an unhandled exception is being through from OnStart() method of your worker role. The best place to begin troubleshooting for these kind of scenarios is to check Windows Azure Event Logs which contains key diagnostic output from the Windows Azure Runtime, including information such as Role starts/stops, startup tasks, OnStart start and stop, OnRun start, crashes, recycles, etc.
System.BadImageFormatException
Process ID: 5132
Process Name: WaWorkerHost
Thread ID: 4
AppDomain Unhandled Exception for role ZipEngine_IN_0
Exception: Could not load file or assembly 'WorkerAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format.
at ZipEngine.WorkerRole.OnStart()
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum)
at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.
<InitializeRole>
b__0()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
So...now we got a bit more detail about the exception from Windows Azure Event Logs which says that process hosting worker role is not able to load the assembly 'WorkerAssembly' due to System.BadImageFormatException. In general when a process is not able to load an assembly, it's always a good practice to capture Fusion logs. Make the following registry key changes under the path [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion] to enable fusion logging. Give Everyone - Full Control permission to the fusion log path folder (C:\FusionLogs).
Upon checking the fusion log for 'WorkerAssembly' I got some interesting facts to further dig into.
*** Assembly Binder Log Entry (8/12/2018 @ 12:51:00 PM) ***
The operation failed.
Bind result: hr = 0x8007000b. An attempt was made to load a program with an incorrect format.
Assembly manager loaded from: D:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Running under executable E:\base\x64\WaWorkerHost.exe
--- A detailed error log follows.
=== Pre-bind state information ===
LOG: DisplayName = WorkerAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
(Fully-specified)
LOG: Appbase = file:///E:/approot
LOG: Initial PrivatePath = E:\approot
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = RoleManager
Calling assembly : ZipEngine, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: E:\approot\ZipEngine.dll.config
LOG: Using host configuration file:
LOG: Using machine configuration file from D:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///E:/approot/WorkerAssembly.DLL.
LOG: Assembly download was successful. Attempting setup of file: E:\approot\WorkerAssembly.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: WorkerAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
ERR: Invalid assembly platform or ContentType in file (hr = 0x8007000b).
ERR: Run-from-source setup phase failed with hr = 0x8007000b.
ERR: Failed to complete setup of assembly (hr = 0x8007000b). Probing terminated.
The above highlighted error message in fusion logs states that something is wrong in the bitness of the assembly ! If we look into the MSDN documentation of BadImageFormatException, then the most probable cause of this error correlates to this -
"A DLL or executable is loaded as a 64-bit assembly, but it contains 32-bit features or resources. For example, it relies on COM interop or calls methods in a 32-bit dynamic link library. To address this exception, set the project's Platform target property to x86 (instead of x64 or AnyCPU) and recompile. "
In order to find out the bitness of the assembly you can run any .NET decompiler of your choice. I prefer ILSpy and this is what I found after decompiling 'WorkerAssembly' .... Bingo ! This is 32-bit assembly (x86).
// C:\WorkerAssembly.dll
// WorkerAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
// Global type:
<Module>
// Architecture: x86
// Runtime: .NET 4.0
Azure is a 64-bit environment. Therefore, .NET assemblies compiled for a 32-bit target won't work on Azure. To address this exception, set 'WorkerAssembly' project's Platform target property to x64 (instead of x86 or AnyCPU) and recompile.
If you review the code of WorkerRole.cs for ZipEngine role you would notice below two lines of code which was actually loading the assembly 'WorkerAssembly' and performing some function. Since it was a 32-bit assembly the WaWorkerHost.exe was not able to load that assembly.
WorkerAssembly.WorkerAssembly workerAssembly = new WorkerAssembly.WorkerAssembly();
workerAssembly.DoWork();
I hope you have got an idea how to troubleshoot issues related to assembly that targets wrong platform in Azure Cloud Service.
Happy Learning !