This question arose out of a support case that my colleague Susanne and I handled recently for one of our customers. The situation was that the first request to an ASPX page was taking in excess of 15 seconds whereas subsequent requests were very fast – less than a second or so.
The obvious possibilities at first thought were things like dynamic page compilation, process startup costs etc but we gradually eliminated those one by one. ASP.NET page tracing revealed that the time appeared to be lost during page pre-rendering:
PagePreRenderEnter – 11:22:09
PagePreRenderLeave – 11:22:25
This gave a clue that it could be to do with the controls that were hosted on that page (there was nothing in the PreRender for the Page itself) but it did not really narrow the search down very much.
Our next idea was to use Process Monitor which can not only capture registry events, file system events and key process events such as thread start and exit, process start and exit but also the kernel and user mode call stacks for all of these.
Diagnosing problems is often not about certainties but about looking for clues. Now in this case we were looking to account for a gap of around 15 seconds. Filtering the trace to only show activity associated with w3wp.exe and scanning down the timestamp column quickly located the general area where the time was being lost – it began at around 11:22:10. So this is where we started to look for clues.
At 11:22:10 we find something to do with WinHTTP happening:
Then just before that we see cryptnet.dll make a network IO request:
So that tells us that something to do with cryptography is initiating a network connection for some reason.
Furthermore, right clicking on an entry lets you bring up a call stack. Even if you’ve forgotten to configure Process Monitor to use the correct version of dbghelp.dll on the machine where the trace was being gathered (as we had) you still get a call stack made up of module names and offsets which can give you clues as to what was going on.
In this case that was enough:
From this we could infer that the .NET runtime was calling into wintrust.dll which was then calling into crypt32.dll which in turn was making the calls into cryptnet.dll.
This immediately began to ring bells (from past experience) about loading of .NET assemblies and the checking of certificate revocation lists.
A key fact in this customer’s situation was that the server experiencing the problem did not have access to the internet. Therefore any attempt to check the validity of a certificate with the issuing authority was doomed to failure.
The delay occurs because in the .NET Framework 2.0 when the CLR loads an assembly into a process, by default if it finds an Authenticode signature it generates Publisher Evidence from it and in doing so it has to fully validate the certificate by (among other things) contacting the issuing authority to ensure the certificate has not been revoked. This is all fine and good if you have access to the internet but if not then a delay followed by a timeout will occur – hence the 15 second delay.
This was recognised as a potential problem in some cases so a hotfix was produced that allows the application to be configured to not do this inference of Publisher Evidence from the Authenticode signature:
This fix is of course included in .NET 2.0 Service Pack 1.
This is now also covered in the relevant documentation.
So the search was on for which assembly was causing this, as this is not an issue we come across all that often. Perhaps with web applications people do not always notice a startup delay because most of the time the server is in a “steady state” serving requests.
Browsing the Process Monitor log revealed that just prior to all this network activity triggered by WinTrust.dll was a load of a 3rd party assembly: Infragistics2.WebUI.UltraWebGrid.v7.3.dll . However, when we checked that it seemed that this assembly was not in fact Authenticode signed:
>signtool verify /pa Infragistics2.WebUI.UltraWebGrid.v7.3.dll
SignTool Error: No signature found.
SignTool Error: File not valid: Infragistics2.WebUI.UltraWebGrid.v7.3.dll
Number of errors: 1
However the customer then found that there was another assembly used by the Infragistics component that was Authenticode signed:
>signtool verify /pa Aspose.Cells.dll
Successfully verified: Aspose.Cells.dll
So, to eliminate the delay, all that had to be done was turn off auto-generation of publisher evidence as documented by adding the appropriate configuration element:
For a normal .NET Windows client application you could put this into an application specific config file, like myapp.exe.config. But for a web application you cannot put it into web.config because web.config defines settings that are only AppDomain wide, not process wide (since aspnet_isapi.dll setups of the hosting environment for the runtime).
Therefore, for ASP.NET, you have two options. The first is to create a file called w3wp.exe.config (assuming you are using IIS6) or aspnet_wp.exe.config (if using IIS5 or 5.1 or running IIS6 in IIS5 compatability mode). This will of course affect all .NET based web applications on the system. However the only other alternative is to specify this in machine.config but of course this affects every .NET application on the machine, not only those hosted in w3wp.exe. Furthermore, for some reason it does not appear to be possible to set this attribute to false at the machine.config level and then subsequently override it with true in an application specific config file.
The question then arises, how secure is it to turn off the generation of publisher evidence like this?
Well, according to the help for generatePublisherEvidence in most situations you do not need it:
“The common language runtime (CLR) tries to verify the Authenticode signature at load time to create Publisher evidence for the assembly. However, by default, most applications do not need Publisher evidence. Standard CAS policy does not rely on the PublisherMembershipCondition. You should avoid the unnecessary startup cost associated with verifying the publisher signature unless your application executes on a computer with custom CAS policy, or is intending to satisfy demands for PublisherIdentityPermission in a partial-trust environment. (Demands for identity permissions always succeed in a full-trust environment.)”
We recommend that services use the <generatePublisherEvidence> element to improve startup performance. Using this element can also help avoid delays that can cause a time-out and the cancellation of the service startup. “
Only you as the application developer can know whether you rely on it or not. But if you are changing the setting in such a way that you may affect other applications on the system then you may need to give some thought to it.
Another question you might ask is why had the vendor of this component bothered to Authenticode sign the assembly. Well, one of the main reasons is that having Authenticode signatures for all executables is that the Windows Vista Logo Program requires it :
“All executable files must be signed with an Authenticode certificate. This includes files with the following extensions: exe, dll, ocx, sys, cpl, drv, scr”
Note that no distinction is made here between managed and unmanaged DLLs.
This whole area of Authenticode signatures is something I am not that familiar with and during my investigations I found this very comprehensive document on Code Signing Best Practices which is well worth a read if you want to know more about this topic.
Hope that helps!