Customer has an ASP.Net application uses XmlSerializer. The XML serialization infrastructure dynamically generates assemblies(by calling CSC.EXE) to serialize and deserialize specified types. By default, the assemblies will be created under %TEMP%. The file name is random generated by CSC to avoid conflict.
The problem customer was experiencing is every time XmlSerializer called, user got an exception like “Could not file the file ‘c:\windows\temp\<random name>.dll’”.
Looking at the c:\windows\temp and with process monitor, confirmed that the assembly was not generated. And there was no permission issue as well. In addition, process monitor showed that CSC.EXE was started and exited at once without doing anything.
Then, by attaching a debugger to CSC.EXE and I found User32.dll was failed to initialize with error code 0xc0000142. Usually, this means problem related to desktop heap. Then we reviewed the desktop heap settings, and increased the desktop size as well. Unfortunately, this doesn’t help.
User32.dll or Kernel32.dll fails to initialize
Another interesting finding is problem disappeared if we add the application pool identity to local administrators group. So, this turns out to be some permission issue. However, process monitor and security audit failed to report any permission issue.
Again, I attached a debugger to the worker process, and found user32.dll failed to initialize because it failed to initialize GDI. In GDI32!NtGdiInit, the kernel call returned 0x0 instead of 0x1 which is the expected result. So, STATUS_NO_MEMORY was returned and this error code was converted to 0xc0000142 eventually.
if (NtGdiInit() != TRUE)
So, what’s the problem? With the help of a platform engineer, we found out the problem is due to a 3rd party component.
When the w3wp.exe process starts, it specifically attaches to the window station __X78B95_89_IW, (which should be created by inetinfo.exe). Only the “Administrators” group has the full control to the window station __X78B95_89_IW. Normally this scenario has no problem because the w3wp.exe process is NOT a Console process but its child process CSC.exe is a Console process. When CSC.exe process is being created, because its parent process w3wp.exe is not a Console process, it creates a Console information from scratch using the default window station Service-0x0-luid$, to the application pool identity which has full control. So everything is running fine in this scenario.
One of customer’s web page loads a 3rd party component and this component calls the API AllocConsole(), which turns the w3wp.exe process into a Console process. The Console information uses the window station __X78B95_89_IW that already associated with the w3wp.exe process. Therefore, after this point, when the w3wp.exe creates the CSC.exe process again, the CSC.exe process will NOT create new Console information anymore, but directly inherits the Console information from w3wp.exe. Because the Console information is using the window station __X78B95_89_IW, which the application pool identity does not have full control to the window, it fails to initialize. That is also why if we add the application pool identity to the “Administrators” group, the problem will disappear.
See you next time.