System.CodeDom.Compiler error!

Got an interesting one today...

Customer had a Windows 2000 box and lots of websites. Everything use to run fine till a moment when they started getting the following error out of the blue. Interestingly, recycling the Application pool would solve this problem every time.

=============================
Server Error in '/AppName/SomeApp' Application.
--------------------------------------------------------------------------------
Cannot execute a program. The command being executed was "c:\windows\microsoft.net\framework\v1.1.4322\vbc.exe"
@"C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files\SomeCodeDir\17cc8924\7736bc14\b3rfp0co.cmdline".
Description: An unhandled exception occurred during compilation using the CodeDomProvider 'Microsoft.VisualBasic.VBCodeProvider'. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Runtime.InteropServices.ExternalException: Cannot execute a program. The command being executed was
"c:\windows\microsoft.net\framework\v1.1.4322\vbc.exe" @"C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET
Files\SomeCodeDir\17cc8924\7736bc14\b3rfp0co.cmdline".

Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:
[ExternalException (0x10b): Cannot execute a program. The command being executed was "c:\windows\microsoft.net\framework\v1.1.4322\vbc.exe"
@"C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files\SomeCodeDir\17cc8924\7736bc14\b3rfp0co.cmdline".]
   System.CodeDom.Compiler.Executor.ExecWaitWithCaptureUnimpersonated(IntPtr userToken, String cmd, String currentDir, TempFileCollection tempFiles, String& outputName, String& errorName, String trueCmdLine) +2048  System.CodeDom.Compiler.Executor.ExecWaitWithCapture(IntPtr userToken, String cmd, String currentDir, TempFileCollection tempFiles, String& outputName, String& errorName, String trueCmdLine) +259   System.CodeDom.Compiler.CodeCompiler.Compile(CompilerParameters options, String compilerDirectory, String compilerExe, String arguments, String& outputFile, Int32& nativeReturnValue, String trueArgs) +383   System.CodeDom.Compiler.CodeCompiler.FromFileBatch(CompilerParameters options, String[] fileNames) +548  System.CodeDom.Compiler.CodeCompiler.FromDomBatch(CompilerParameters options, CodeCompileUnit[] ea)   System.CodeDom.Compiler.CodeCompiler.FromDom(CompilerParameters options, CodeCompileUnit e)   System.CodeDom.Compiler.CodeCompiler.System.CodeDom.Compiler.ICodeCompiler.CompileAssemblyFromDom(CompilerParameters options, CodeCompileUnit e) System.Web.Compilation.SourceCompiler.CompileAndCache()
--------------------------------------------------------------------------------
Version Information: Microsoft .NET Framework Version:1.1.4322.2032; ASP.NET Version:1.1.4322.2032

It was quite tricky at first but here is what we tried and ultimately fixed the issue.

First thing was that we needed to find out what is causing this issue in the first place. We checked the C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files folder and looked at two .out files in the same folder C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files\SomeCodeDir\17cc8924\7736bc14

We found that the aspnet_wp.exe process directory should start with "C:\WINDOWS\system32>" but somehow it changed to \\SomeServer\DATA at the time the problem was occurring. Obviously that folder didn't have the necessary ACLs and that is why it was throwing the error message. We needed to find out how that folder location was changed. Thankfully, the customer knew where they are changing the folder using SetCurrentDirectory and ChDir in their code. Once we came to know about the pages, it was pretty easy to prove.

  1. Wait for the issue to happen. Let's say it is happening for page X.
  2. Once it happens, give an IISReset and go to the page which is calling ChDir or SetCurrentDirectory
  3. Go back to page X and it should throw the above error message.
  4. Now give another IISReset and go to page X directly. This should work!

The resolution is to either not use that API or give appropriate permissions to the Remote share. If it is a Remote share, you might use a domain account for ASP.NET.

Cheers,
Rahul