“Exception message: Error in the application..” application event log error from TFS Service

Coop (Chris Cooper) bring us this little tidbit from a recent support case of his…

 

When looking at your application event logs, you may notice the following error from time to time:

Event Type:        Error
Event Source:    TFS Services
Event Category:                None
Event ID:              3071
Date:                     9/19/2009
Time:                     3:02:19 AM
User:                     N/A
Computer:          TFSSERVER
Description:
TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC): 9/19/2009 3:00:19 AM
Machine: TFSSERVER
Application Domain: /LM/W3SVC/3/Root/Services-3-128977417369957288
Assembly: Microsoft.TeamFoundation.Server, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
Process Details:
  Process Name: w3wp
  Process Id: 2296
  Thread Id: 3796
  Account name: XYTECHLA\tfsservice

Detailed Message: TF200035: One or more errors occurred when Team Foundation Server attempted to synchronize with the following Active Directory identity: Team Foundation Valid Users. Number of errors that occurred: 1.
An error occurred when synchronizing the following Active Directory identity: Team Foundation Valid Users. Exception message: Error in the application..
For more information, see Help and Support Center at go.microsoft.com/fwlink/events.asp.

When I saw it for the first time, I was perplexed. I only had a limited number of users and nobody had ever complained about access to the server. I had no trouble adding new users and I would only see this error once a day. So I began to dig a little deeper. Looking at the system event log, I saw the following error at the same time:

“A worker process with process id of XXXX serving application pool 'Microsoft Team Foundation Server Application Pool' has requested a recycle because the worker process reached its allowed processing time limit. “

So I looked at the application pool settings and noted that our application pool for Team Foundation Server was set to recycle every night at 3:00 am. The default setting is normally set to recycle every 1740 minutes (29 hours) but we had changed this reproducing another customers issue.

So here’s what’s going on. Synchronization with AD is performed by a background thread run by the TFS web application. We typically see this error due to scheduled restarts, because the restart occurs on the hour, and so does the sync. If you are setup this way, you will see it on the restart because just before the restart, identity sync was already in progress, then the process running identity sync is shut down. So this is basically a thread abort, that we report as “Error in the application”.