Operation has timed out from class library in COM+

  • Article

In a recent MS internal performance gig we encountered an interesting issue with the maxconnection setting in the Machine.Config. Essentially the application we were testing consisted of a web application using classic ASP, COM+ business objects and a .NET wrapper proxy that consumed web services on a separate web server using integrated authentication and SSL.

We used VSTS to apply load to the web server and measure throughput, after a minute we noticed that the server throughput dropped and ASP requests per seconds fell to zero. RPS would stay at zero for exactly 1 minute 40 seconds then they would resume, this behavior would repeat itself every minute or so. Even more troubling were the exceptions in the application event viewer. They looked like this: System.Net.WebException: The operation has timed out…

Fast forwarding a bit, we decided to try live debugging. Fortunately for us we found an excellent blog entry that contained a step by step walkthrough on a similar issue:https://blogs.msdn.com/tess/archive/2006/02/23/asp-net-performance-case-study-web-service-calls-taking-forever.aspx

While the system was under load we took a dump of the DLLHOST process. We used WINDBG to analyze the memory dump and very similar to the blog entry above, we found that many of the threads in our process are waiting. Here’s an example.

img1

Switching to thread 39 we can see multiple System.Net.HttpWebRequest objects that are used to make our Web Service calls.

img2

Looking at one of the HttpWebRequest objects we see something interesting called _ServicePoint. The ServicePoint class provides connection management for HTTP connections.

img3

We verified that we were looking at the correct object by checking the _Uri value. It was also interesting to note in the System.Net.HttpWebRequest properties that the _Timeout value of 100000 milliseconds corresponds with the timeout we are seeing during our load testing. After 1 minute 40 seconds the timeout occurs.

img4

Finally we took a look at ServicePoint and found that the m_ConnectionLimit and m_CurrentConnectins were set to 2 in System.Net.ServicePoint.

img5

Immediately we checked the Machine.Config on the web server and located the connectionManagement section. In it we saw that the maxconnection was set to 24 per the best practice (12 * # of processors)

<connectionManagement>

<add address="*" maxconnection="24"/>

</connectionManagement>

We also tried a simple test, since we were able to reproduce the issue on demand; we ran our VSTS load test and reproduced the problem again. On the web server we used the netstat command to check the number of connections to the web services server; we found that 2 connections were made on port 443 to the web services server. Somehow the connection settings in the machine.config were being ignored and the webserver was constrained to 2 total web service connections to our web services server.

Since the maxconnection setting was ignored we needed a way to set the maxconnections programmatically in our code . The way to do this is with the System.Net.ServicePointManager class.  Here is a sample of how to set the DefaultConnectionLimit in code:

System.Net.ServicePointManager.DefaultConnectionLimit = nn;

//Where nn is the recommended connection value in integer.

Using this class in our .NET wrapper code we set the .DefaultConnectionLimit property before any HTTP/HTTPS web service calls are made. After making the code change we retested and the problem was resolved!

NOTE: Special thanks to Tess Ferrandez for the awesome blog mentioned above, Edmund Wong for Windbg help and PSS Developer Support for help with the resolution.

For More information on ServicePointManager see the following: https://msdn2.microsoft.com/en-us/library/system.net.servicepointmanager.defaultconnectionlimit(vs.80).aspx

James Beeson

Performance Technologist

Microsoft – ACE Team