Intermittently System.Web.HttpExceptions(A field or property with the name ’ ‘ was not found on the selected data source) is thrown when the application is under heavy load

Working on another interesting case where we were occasionally failing with HttpException and after hitting refresh on the browser we were back in business.Next stop was to look at event viewer where bunch of unhandled exceptions like below were reported:

Event Type: Warning
Event Source: ASP.NET 2.0.50727.0
Event Category: Web Event
Event ID: 1309
Date:  5/18/2009
Time:  3:29:10 PM
User:  N/A
Computer: MachineName
Description:
Event code: 3005
Event message: An unhandled exception has occurred. Application information:
    Application domain: /LM/W3SVC/1/Root/Web-1-128871464389591756 Process information:
    Process ID: 3428
    Process name: w3wp.exe
    Account name: NT AUTHORITY\NETWORK SERVICE
Exception information:
    Exception type: HttpException
    Exception message: A field or property with the name 'Contact_Type' was not found on the selected data source.
Thread information:
    Thread ID: 1
    Thread account name: NT AUTHORITY\NETWORK SERVICE
    Is impersonating: False
    Stack trace:    at System.Web.UI.WebControls.BoundField.GetValue(Control controlContainer)
   at System.Web.UI.WebControls.BoundField.OnDataBindField(Object sender, EventArgs e)
   at System.Web.UI.Control.OnDataBinding(EventArgs e)
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at System.Web.UI.Control.DataBind()
   at System.Web.UI.Control.DataBindChildren()

OR sometime we get error message like:

Exception information:
    Exception type: HttpException
    Exception message: A field or property with the name 'Request_ID' was not found on the selected data source.

Thing to note here was that every time a different column name was specified in the message. This was a good indication to run SQL profiler trace and we found that we were executing a different query (which was of course not having the required columns) and it was not the one which was bound to the control(GridView). At this point question tickled my thought process as to why were the wrong columns returned by Dataset/DataSource or more importantly why was a different query being executed?

Further reviewing the code we figured out that we were having Data Access Layer which are actually Static Classes and all the SQLConnection And SQLCommand objects were declared as static in those classes.

private static void OpenSqlConnection()
private static void CreateCommand(string queryString,string connectionString)

Aha now that’s our culprit !

Static function would show up SQL query whatever was the last one executed. Now that reminds me of an excellent post by one of our Escalation Engineers, Brian Booth, which talks  about real world example which can lead to this unexpected results.

Static(c#) or Shared(vb.net) are global variable across App Domain and would be accessible throughout the process uptime. It’s value would be retain across pages and would be available to all the users!!!That means each user would be able see each other data….

Till then Bye bye…