How to troubleshoot ngen issues

  • Article

Hi,

Earlier this week, I had a need to ngen several TFS assemblies on TFS 2010 AT so that I can get rid of broken call stacks in PerfView profiler. How hard could it be? Just run ngen.exe on these assemblies. That's what I throught as well. Well, I was wrong. :)

Tip #1. Pick the right version of ngen.exe

These are TFS assemblies I was trying to ngen:

  • Microsoft.Teamfoundation.Framework.Server.dll
  • Microsoft.Teamfoundation.VersionControler.Server.dll
  • Microsoft.TeamFoundation.VersionControl.Server.Proxy.dll

Well, these assemblies are compiled against .NET V2.0. But appPool is configured to run with .NET V4.0 as a 64-bit. So we need to use the NGen.exe under d:\windows\microsoft .net\framework64\v.4.0.30319. 

To avoid typing the full path, I navigated to the above path and Set Path=%CD%;%PATH%

 

Tip #2: Use /nodepdencies

So I navigated to web service\bin folder where TFS assemblies are located and ran "ngen.exe install %TfsAssemblyName%". This failed with a bunch of dependenies not found warnings.

Warning: ngen'ed image is still getting created though. So make sure you uninstall them properly so that your ngen folder is not clogged with a bunch of "bad" images that won't get used by your application anyway. And to uninstall them properyl, you need to use the same version of ngen.exe you used to install it, and pass in the same parameters.  Anyway, I forgot to do this and later on I just couldn't uninstall a few of them from NGEN folder. So I just use brutal force and navigated to these folders and deleted them manually. (It didn't seem to cause any lingering issues. well so far. :))

Well, I'm aware of /nodependencies option for ngen.exe. So I ran "ngen.exe install %TfsAssemblyName% /nodependencies".

This still failed and I noticed it is looking up .NET V2.0 dependencies. Hmmm, so how do I tell ngen.exe to use .NET V4.0 for these TFS assemblies which are compiled against .NET V2.0?

I know MS.TF.Framework.Server.dll is used by TfsMgmt.exe as well and TfsMgmt.exe is compiled with .NET V4.0. It is doing so with these entries in the TfsMgmt.exe.config file:

<configuration>
<startup useLegacyV2RuntimeActivationPolicy="true">
<supportedRuntime version="v4.0"/>
</startup>
</configuration>

So maybe we can pass this along to ngen.exe?

 

Tip #3: Use /ExeConfig

Now I'm running this command: "ngen.exe install %TfsAssemblyName% /nodependencies /execonfig:" D:\Program Files\Microsoft Team Foundation Server 2010\Tools\Tfsmgmt.exe"".

This seemed to work!!!! :)

 

Tip#4: Use "ngen.exe display /verbose"

Now I did a "IISreset" and use Process Explorer to verify my ngen assemblies are loaded. Unfortunatelly, MS.TF.VersionControl.Server.dll is still not using the ngen'ed version. :(

It turns out Ngen.exe has a nice command "ngen.exe display /verbose > out.txt" that give you a lot of good information on any ngen'ed assemblies. It tells you which version of ngen images are installed and which options were
used while it was installed:

  • ScenarioNoDependencies: this means /NoDependencies is used
  • ConfigExe: this means /configExe is used
  • RunTime Version: this tellyou which version of Ngen.exe was used.
  • Processor or ngen install path: This tells you if 32 bit or 64-bit ngen was used
  • ** Missing dependency assembly **: this tells you while ngen image was created, it failed to locate a dependent assembly.  This ngen image will failed to load by
         your application.

With this, I found that there are two missing dependency dlls for MS.TF.VersionControl.Server.dll while other two ngen'ed assemblies have no missing dependencies:

  • Microsoft.TeamFoundaton.VersionControl.Server.Proxy.dll
  • Microsoft.TeamFoundation.WorkItemTracking.Server.DataAccessLayer.dll

But both these dlls are there on the "web Service\bin" directory, the same directory where the MS.TF.VersionControl.Server.dll is. So why does ngen.exe failed to find these assemblies?

 

Tip #5: Use FusLogVw.exe to troubleshoot

To figure out why ngen.exe failed to locate these two dlls while ngen'ing MS.TF.Versioncontrol.Server.dll, I enabled FusLogVw.exe. (copied D:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools from a VS client machine).

The log revealed the reason. Becuase I was using /execonfig option on ngen.exe: ngen.exe install Microsoft.Teamfoundation.VersionControler.Server.dll /nodependencies /execonfig:" D:\Program Files\Microsoft Team Foundation Server 2010\Tools\Tfsmgmt.exe"

So why wasn't thin an issue for other two assemblies? (Framework.Server and Proxy). Because their dependent dlls are also used by TfsMgmt.exe so they are all avaailbe under D:\Program Files\Microsoft Team Foundation Server 2010\Tools!!!

Armed with this knownledge, I fixed the ngen issue on MS.TF.VersionControl.Server.dll with these steps:

1. Copy TfsMgmt.exe and TfsMgmt.exe.config from D:\Program Files\Microsoft Team Foundation Server 2010\Tools to D:\Program Files\Microsoft Team Foundation Server 2010\Web Services\bin

2. ngen.exe install Microsoft.Teamfoundation.VersionControler.Server.dll /nodependencies /execonfig:" D:\Program Files\Microsoft Team Foundation Server 2010\Web Services\bin\Tfsmgmt.exe"

Now I have ngen'ed TFS assemblies loaded in my appPool!!!

 

 *** Assembly Binder Log Entry  (8/8/2012 @ 1:52:33 PM) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from:  D:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Running under executable  D:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscorsvw.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: User = DDNET\VSTSLAB
LOG: DisplayName = Microsoft.TeamFoundation.VersionControl.Server.Proxy, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
 (Fully-specified)
LOG: Appbase = file:///D:/Program Files/Microsoft Team Foundation Server 2010/Tools/ LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = mscorsvw.exe
Calling assembly : (Unknown).
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: D:\Program Files\Microsoft Team Foundation Server 2010\Tools\Tfsmgmt.exe.config LOG: Using host configuration file: 
LOG: Using machine configuration file from D:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///D:/Program Files/Microsoft Team Foundation Server 2010/Tools/Microsoft.TeamFoundation.VersionControl.Server.Proxy.DLL. LOG: Attempting download of new URL file:///D:/Program Files/Microsoft Team Foundation Server 2010/Tools/Microsoft.TeamFoundation.VersionControl.Server.Proxy/Microsoft.TeamFoundation.VersionControl.Server.Proxy.DLL. LOG: Attempting download of new URL file:///D:/Program Files/Microsoft Team Foundation Server 2010/Tools/Microsoft.TeamFoundation.VersionControl.Server.Proxy.EXE. LOG: Attempting download of new URL file:///D:/Program Files/Microsoft Team Foundation Server 2010/Tools/Microsoft.TeamFoundation.VersionControl.Server.Proxy/Microsoft.TeamFoundation.VersionControl.Server.Proxy.EXE. LOG: All probing URLs attempted and failed.

 

Tip #6: You can also use FusLogVw.exe to find out why an ngen'ed image is not been used.

a. Make sure you check "Native Image" option in settings.

b. For a failed ngen assembliy, you will see something like this:

WRN: Dependency assembly was not found at ngen time, but is found at binding time. Disallow using this native image.
WRN: No matching native image found.
LOG: Bind to native image assembly did not succeed. Use IL image.
LOG: IL assembly loaded from D:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\ad61b6f7\f7e77928\assembly\dl3\76ac4941\00c26601_b5c7ca01\Microsoft.TeamFoundation.VersionControl.Server.dll.

Have a good day!

Jiange