Suddenly TFS Reports Throwing "An internal error occurred on the report server. See the error log for more details. (rsInternalError)"
The other day I was changing the service account passwords on my VPCs that I use for troubleshooting customer issues and decided just to run down the Team Explorer nodes and make sure everything was still running fine. In addition to the password changes that may have caused issues, our corporate policies here will sometimes delete your machine account if your PC has not authenticated with the domain in a certain number of days, and this can happen easily with VPCs as I usually leave them in a SAVED state when not in use. This machine account deletion can cause weird behavior in TFS as you can well imagine. So anyway, I was just making sure that hadn't happened, my password change had gone off fine, and all was still running by the 9s when I came to find out that my reports were acting up. I tested the "Remaining Work" report in a brand new team project based on Agile on my TFS 2008 / SQL 2005 / WSS 3.0 ATDT VPC. It rendered fine on the first run, but when I clicked the "View Report" button a second time, I was given this:
An internal error occurred on the report server. See the error log for more details. (rsInternalError)
I had to do a bit of searching around to find out where this error log actually was (it's not a log you see in Event Viewer, as I first thought). It's actually located in \Program Files\Microsoft SQL Server\MSSQL.3\Reporting Services\LogFiles. Looking in there I found some dumps (.mdmp) in addition to the log files I was looking for. Opening SQLDmpr0001.log I found this:
w3wp!processing!1!9/25/2008-09:12:24:: a ASSERT: Assertion failed! Call stack:
Microsoft.ReportingServices.ReportProcessing.ReportProcessing.UpdateTimingInformation
(RunningJobContext jobContext, Int64 totalDurationMs, Int64
reportDataRetrievalDurationMs, Int64 reportRenderingDurationMs)
Microsoft.ReportingServices.ReportProcessing.ReportProcessing.ProcessAndRenderReport
(IRenderingExtension renderer, Boolean snapshotProcessing, DateTime
executionTimeStamp, ProcessingContext pc, RenderingContext rc, GetReportChunk
getReportChunkCallback, GetReportChunk getDataCallback)
Microsoft.ReportingServices.ReportProcessing.ReportProcessing.ProcessAndRenderSnapshot
(ProcessingContext pc, RenderingContext rc, GetReportChunk
getReportChunkCallback)
Microsoft.ReportingServices.Library.RSService.RenderAsSnapshotNoCache
(CatalogItemContext reportContext, Guid reportId, DateTime executionDate,
DateTime expirationDate, ClientRequest session, ReportSnapshot snapshotData,
String description, RenderingResult& result, Boolean&
transientRefcountNeedsDecreasing)
Microsoft.ReportingServices.Library.RSService.RenderAsSnapshot(CatalogItemContext
reportContext, Guid reportId, DateTime executionDate, DateTime expirationDate,
ClientRequest session, ReportSnapshot snapshotData, String description, Boolean
foundInCache, Warning[]& warnings, ParameterInfoCollection& effectiveParameters)
Microsoft.ReportingServices.Library.RSService.RenderAsLiveOrSnapshot
(CatalogItemContext reportContext, ClientRequest session, Warning[]& warnings,
ParameterInfoCollection& effectiveParameters)
Microsoft.ReportingServices.Library.RSService.RenderFirst(CatalogItemContext
reportContext, ClientRequest session, Warning[]& warnings, ParameterInfoCollection&
effectiveParameters, String[]& secondaryStreamNames)
Microsoft.ReportingServices.Library.RenderFirstCancelableStep.Execute()
Microsoft.ReportingServices.Diagnostics.CancelablePhaseBase.ExecuteWrapper()
Microsoft.ReportingServices.Library.RenderFirstCancelableStep.RenderFirst(RSService rs,
CatalogItemContext reportContext, ClientRequest session, JobType type, Warning[]&
warnings, ParameterInfoCollection& effectiveParameters, String[]&
secondaryStreamNames)
Microsoft.ReportingServices.Library.RenderReportAction.Render(String format, String
deviceInfo)
Microsoft.ReportingServices.WebServer.ReportExecution2005Impl.InternalRender(String
Format, String DeviceInfo, Stream& Result, String& Extension, String& MimeType,
String& Encoding, Warning[]& Warnings, String[]& StreamIds)
Microsoft.ReportingServices.WebServer.ReportExecutionService.InternalRender(String
Format, String DeviceInfo, Stream& Result, String& Extension, String& MimeType,
String& Encoding, Warning[]& Warnings, String[]& StreamIds)
Microsoft.Reporting.WebForms.ServerReport.InternalRender(String format, String
deviceInfo, String& extension, String& mimeType, String& encoding, String[]&
streams, Warning[]& warnings)
Microsoft.Reporting.WebForms.ServerReport.Render(String format, String deviceInfo,
NameValueCollection additionalParams, String& mimeType, String& extension)
Microsoft.Reporting.WebForms.ServerReportControlSource.RenderReport(String format,
String deviceInfo, NameValueCollection additionalParams, String& mimeType, String&
fileExtension)
Microsoft.Reporting.WebForms.ReportControlSource.RenderReportHTML4(String
deviceInfo)
Microsoft.Reporting.WebForms.ReportControl.RenderReport(ReportControlSource
controlSource, String instanceIdentifier, String navigationID, ZoomMode zoomMode,
Int32 zoomPercent, Int32 pageNumber, String prefix, String searchText, String
linkTarget, String replacementRoot, String alertMessage, Int32 searchStartPage,
Boolean sizeToContent)
Microsoft.Reporting.WebForms.ReportAreaPage.OnPreRender(EventArgs e)
System.Web.UI.Control.PreRenderRecursiveInternal()
System.Web.UI.Page.ProcessRequestMain(Boolean
includeStagesBeforeAsyncPoint,Boolean includeStagesAfterAsyncPoint)
System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean
includeStagesAfterAsyncPoint)
System.Web.UI.Page.ProcessRequest()
System.Web.UI.Page.ProcessRequestWithNoAssert(HttpContext context)
System.Web.UI.Page.ProcessRequest(HttpContext context)
System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter
writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath
filePath, String physPath, Exception error, String queryStringOverride)
System.Web.HttpServerUtility.Transfer(IHttpHandler handler, Boolean preserveForm)
Microsoft.Reporting.WebForms.ReportAreaPageOperation.PerformOperation
(NameValueCollection urlQuery, HttpResponse response)
Microsoft.Reporting.WebForms.HttpHandler.ProcessRequest(HttpContext context)
System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.
HttpApplication.IExecutionStep.Execute()
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean&
completedSynchronously)
System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(Exception error)
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest
(HttpContext context, AsyncCallback cb, Object extraData)
System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
System.Web.HttpRuntime.ProcessRequestNoDemand(HttpWorkerRequest wr)
System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr ecb, Int32 iWRType)
w3wp!library!1!9/25/2008-09:12:24:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details., un-named assertion fired for component processing;
Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details.
I searched around and finally found this KB article that describes the issue detailed in the log. That article lead me to another article, which is a list of SQL Server 2005 builds that were released after SQL Server 2005 Service Pack 2 was released. From there I got a link to "Cumulative update package 9 for SQL Server 2005 Service Pack 2", the most recent cumulative update at time of writing. Using the link on the article I was able to get the HOTFIX and apply it to my TF server (well, to SQL Server 2005, to be specific). After a reboot and a quick test, my reports seem fine now.
Why am I telling you all this, you may ask? Well to help you out, for one <g> but more specifically because I found it interesting that this server had been running flawlessly since install, and I could not figure out what might have caused this odd issue to raise it's ugly head. Reading 946082 however set off an alarm... specifically this bit:
"You install Microsoft SQL Server 2005 Reporting Services (SSRS) on a computer that has multiple processors..."
Why did this resonate with me? Well, I had recently moved my virtualization scheme from Virtual Server 2005 R2 to a new Windows Server 2008 / Hyper-V machine, and transferred all my TFS VPCs over there. One of the key features in Hyper-V is the support for multiple processors, which this "failing" TFS now had in it's new virtualization home - I had configured the processor setting for the VPC to "2". :-)
So there you have it. I hope someone out there finds this useful. As for me, I have to go off and put this HOTFIX on my 12 other VPCs. :-/
Trevor Hancock