Thanks to a couple of my European support colleagues for sharing this one, which I know could also affect many of our customers world-wide if they are running servers that are not internet connected. Great work by Jorge Puig Altozano from our Project support team in Madrid, and special thanks (and all the credit - according to Jorge) to Hector Calvarro for the SharePoint team – also in Madrid. If Spanish is your preferred language then read on at http://blogs.technet.com/b/elfarodeprojectserver/archive/2012/02/22/project-server-2010-elevados-tiempos-de-respuesta-al-cargar-paginas-de-sitios-pwa.aspx and also Hector’s blog at http://blogs.technet.com/b/hablamoss/. Although we are just talking about PWA and SharePoint here – consideration also needs to be given to other 3rd Party assemblies that may be installed too – and this could affect the loading of PDPs.
On to the English description – translation thanks to Jorge, with some additional input from Catalin Olteanu from one of our partners, UMT, as they also experienced the same slow downs.
*** Update 3/30/2012 - Before you read too far here is another suggested fix from Catalin
The fix is exporting the
“SharePoint Root Authority” certificate from SharePoint and import to the
Trusted Root Certification Authorities store.
To export the certificate,
use these commands in SharePoint Management Shell:
| Set-Content C:\SharePointRoot.cer -Encoding byte
The exported certificate then
needs to be imported to the Trusted Root Certification Authorities on each of
the servers in the farm and that should eliminate the 15 seconds delay.
End Update ***
We had a Project Server 2010 server, with no Internet connection, that was returning high response times when we would try to load any PWA page – slow response was seen to be due specifically to the calls to SecurityCheckUserPagePermisison, and CheckUserProjectPermissions. We observed the information n the SharePoint Developer Dashboard, and decided to take a look at the Certificates behavior.
BriSmith note – I’m sure lots of analysis went into this decision by Hector – and for anyone interested in understanding more about the problem Catalin found the lookup for www.download.windowsupdate.com from a netmon trace, and the event viewer showing a 4102, and a couple of 4107 CAPI2 errors helped join the dots…
We disabled the timeouts for the certificates verification in the SharePoint server
On Windows Server , this component is on by default and , whenever an application is presented with a certificate that is not present in the trusted root store, it will attempt to contact Microsoft download servers to get the latest root chain.
The SharePoint OOB certificates can induce this as they are stored in a particular repository (SharePoint- Under Certificate management -Local Computer), as opposed to the trusted root. The decision not to have SharePoint code creating and installing a root cert in the Trusted Root store was taken for security reasons (ex if an application could install a certificate into the TRC store might compromise the security of the system).
Can this behavior be avoided? ( ie. bypass this check for subsequent validations).
Disable automatic update of root certificates on SharePoint Servers
• Launch gpedit.msc as admin on the box
• Go to Computer Configuration –> Windows Settings ->Security settings ->Public Key Policies -> Certificate Path validation settings
• In Network retrieval tab -> Define the policy and uncheck “Automatically update certs from Microsoft root cert program”
• Run gpupdate /force for policy to take effect immediately.
Additionally , the cert management plan needs to be implemented as per below article:
Manage Certificate Path Validation:
• It is not unusual for enterprises to disable auto-root update. If they opt to do it, they will have to manage distribution of third-party roots that they need in their enterprise via group policy.
• The customer will want to monitor new releases (KB931125) quarterly and update their trust as required.
Implications of disabling :
There should be no specific implications to SharePoint since we are using self-signed certs and manage them ourselves. The SharePoint certs do have an expiry and we do have a health rule that watches for that IIRC and will warn the admin to update/re-roll them.
The main aspect to think through is for “other” certs used on the box (like SSL certs, certs to trust download packages or for SAFER policy etc etc) which are issues from certs chained to those in the TRC store. But note there is nothing “new” about these issues with this setting; since the boxes in question cannot access the Internet … they actually “require” more hands on.
We also got rid of the verifications for Code Access Security and some other certificates (Certificates Revocation List and Authenticode signatures) doing the following:
We have to edit this file: C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG\machine.config
And add/change the value:
The explanation for this key:
This element was introduced in the .NET Framework version 3.5 and applies only to that version. It has no effect in later versions of the .NET Framework.
The common language runtime (CLR) tries to verify the Authenticode signature at load time to create Publisher evidence for the assembly. However, by default, most applications do not need Publisher evidence. Standard CAS policy does not rely on the PublisherMembershipCondition. You should avoid the unnecessary startup cost associated with verifying the publisher signature unless your application executes on a computer with custom CAS policy, or is intending to satisfy demands for PublisherIdentityPermission in a partial-trust environment. (Demands for identity permissions always succeed in a full-trust environment.)
If we don't use code signed by Authenticode, and work with CLR validations, we can work with the mentioned option. This is a more detailed explanation about this:
A little background – CAS is feature in .NET that allows you to have more granular control over what code can execute in your process. Basically there are 3 parts:
1.Evidence – Information that a module/code presents to the runtime. This can be where the module was loaded from, the hash of the binary, strong name, and importantly for this case the Authenticode signature that identifies a modules publisher.
2.Permissions Set – Group of Permissions to give code (Access to File System, Access to AD, Access to Registry)
3.Code Group – The evidence is used to provide membership in a code group. Permission Sets are granted to a code group.
So when a module loads it presents a bunch of evidence to the CLR and the CLR validates it. One type of evidence is the “publisher” of the module. This evidence is validated by looking at the Authenticode signature which involves a Certificate. When validating the Certificate the OS walks the chain of Certificates and tries to download the Certificate Revocation List from a server on the internet. This is where the slowdown occurs.
A lot of servers do not have access to make calls out to internet. It is either explicitly blocked, the server might be on a secure network, or a proxy server might require credentials to gain access to the internet. If the DNS/network returns quickly with a failure the OS check will move on but if the DNS/network is slow or does not respond at all to the request we have to timeout.
This can occur for multiple modules because we create this evidence for each module that is loaded. However if we have looked for a CRL and failed we will not recheck. However different certificates have different CRLs. For instance a VeriSign Certificate may have one CRL URL but a Microsoft Certificate will have a different one.
Since this probe can slow things down it is best to just avoid the probe if you do not need it. For .NET the only reason you would need it is if you are setting Code Access Security based on the module Publisher. Because this can cause potential slow downs and you do not need to occur this penalty you can just disable the generation of the Publisher Evidence when your module is loaded. To disable this use the <generatePublisherEvidence> Application configuration. Just set the enabled property to false and you will avoid all of this.
Now for ASP.NET applications it was not immediately obvious how to do this but it turns out that you cannot add this to an applications Web.Config but you can add it to the ASPNET.CONFIG file in the Framework directory. For other applications just add the attribute to the APP.CONFIG file.
More information here:
- Site slowness due to SharePoint STS Certificate CRL checking