Unexpected Error has Occurred
A customer was browsing their SharePoint site and randomly seeing a white page with "An Unexpected Error Has Occurred" in black, unformatted text being returned [your text may vary]. If you run across this, it indicates that the Error.aspx page threw an exception while processing. The Error.aspx page hooks the Page_Error event, so if an error occurs during processing of this particular page, you don't get stuck in an infinite loop or crash the process [error occurs, so it redirects to error.aspx which fails, then redirects to error.aspx, on and on]. In the event handler, the following HTML is returned to the client:
<html>
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
</head>
<body>
An unexpected error has occurred.
</body>
<script type="text/javascript" language="JavaScript">
var gearPage = document.getElementById('GearPage');
if(null != gearPage)
{
gearPage.parentNode.removeChild(gearPage);
document.title = "Error";
}
</script>
</html>
We were able to narrow down the above error and symptoms to a specific web application on a specific web front end. The issue appeared random due to the way requests were being load balanced. If you hit this box, you would get the error. We attached windbg.exe and setup a breakpoint on managed exceptions (sxe clr). We then browsed the page to see that an access violation was occurring trying to access the ContentService object which was pointing to an bad address. We then found the following exception in the dump that indicates a SQL Deadlock occurred while SharePoint was trying to get the Alternate URL Collection.
Exception type: System.Data.SqlClient.SqlException
Message: Transaction (Process ID 3002) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
InnerException: <none>
StackTrace (generated):
Function
System.Data.SqlClient.SqlConnection.OnError
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning
System.Data.SqlClient.TdsParser.Run
System.Data.SqlClient.SqlDataReader.HasMoreRows
System.Data.SqlClient.SqlDataReader.ReadInternal
Microsoft.SharePoint.Administration.SPPersistedObjectCollection`1[[System.__Canon, mscorlib]].get_BackingList()
Microsoft.SharePoint.Administration.SPPersistedObjectCollection`1[[System.__Canon, mscorlib]].GetEnumerator()
Microsoft.SharePoint.Administration.SPAlternateUrlCollectionManager.FlushCaches()
Microsoft.SharePoint.Administration.SPFarm.FlushCache
Microsoft.SharePoint.Administration.SPFarm.FireChangeEvent
Microsoft.SharePoint.Administration.SPConfigurationDatabase+SPFileSystemCacheWatcher.OnChanged
The particular web application had been up for a couple days [automatic recycling had been disabled previously to help diagnose another problem], so we used the IIS logs to determine when requests started failing. We would see a request to a resource with a status code of 200, but the time taken and response size was very small due to the request failing and the small bit of HTML from above being sent. Once we had a time we then checked the application event viewer on the particular web front end, and confirmed the above deadlock exception was logged in the Event Viewer on this box at the same time as the IIS entries we found.
Event ID: 6398
Event Source: Windows SharePoint Services 3
Event Message: Microsoft.SharePoint.Administration.SPConfigurationRefreshJobDefinition
dd509b0c-6b7e-4581-b523-658b03c336eb
Transaction (Process ID 74) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
We already had deadlock tracking enabled on SQL, and were able to go back to the SQL Error Logs for the time of the IIS log failures and Event Viewer entry to see that a deadlock had occurred between two stored procedures in the configuration database: proc_putObject and proc_getObjectsByBaseClass.
In order to resolve the error, you have to recycle the application pool.