Web Services: Concurrent Connections


The issue of concurrent outbound Web Service calls being throttled to two per endpoint comes up again and again with unmanaged and managed code – and it embarrassingly bit me last week in a BizTalk Performance and Scaling exercise despite me harking on about it all the time with standard .NET development L

The HTTP 1.1 Specification, specifies that a “single-user client SHOULD NOT maintain more than 2 connections with any server or proxy”.  This is to ensure that a user cannot overload a remote Web Server, this is all understood and the various Microsoft HTTP stacks all respect this restriction as per the specification.

This is fine for user scenarios, but when you’re calling a Web Service from an ASP.NET Site, BizTalk Orchestration (via SOAP Adapter), etc. you will also hit this limit for the whole of your application rather than for each user of your application.  This concurrent connection restriction is applied to an AppDomain which in the ASP.NET world means a Virtual Directory, which then means all users concurrently calling ASP.NET Pages will have any outbound Web Service calls throttled down to two concurrently, pending requests will then be queued.

This as you’d imagine causes a massive scalability and performance problem, and unfortunately this default behavior makes its way into live solutions more often that not, but fortunately it’s easy to fix.

In your code you can override this as per this article, in your initialization you need to set:   ServicePointManager.DefaultConnectionLimit = 4;

This will then be hardcoded inside your application so my recommendation would be to control it via your .config file:

<system.net>

            <connectionManagement>

      <addaddress=”*”maxconnection=”40″/>

</connectionManagement>

</system.net>

More background on this here.  As for the value you should use it’s really up to your scenario, as long as the remote Web Server can handle it (and is under your control) you probably want it as close to the number of possible threads available on the requesting machine.  If it’s an ASP.NET page calling the Web Service you ideally want it set to 25*NumberofProcessors which matches the ASP.NET Worker Thread pool.

A quick and easy way to see if you’ve hit this issue is to monitor the number of Concurrent Connections to your Web Server, if you’re running IIS the “Concurrent Connections” counter under the Web Service performance object is the one to watch.  If you never see this go about 2 * NumberOfRequestingMachines then you might be hitting the limitation. 

I was in the performance lab and we were only seeing 4 concurrent connections to the IIS Box hosting a Web Service, we had two BizTalk 2004 Servers (Quad Procs) and were putting a lot of load through the system and should have seen far higher connections.  We put the above change into the BTSNTSVC.exe.config file and low and behold the number of Concurrent Connections got into the 30/40s J

For some reason I just assumed the BizTalk 2004 Install would “overcome” this limitation but it doesn’t, and the more I think about it – it shouldn’t either.  The Installer doesn’t know how many connections your remote Web Service can handle – that’s something you need to tune.

Anyway, we live and learn – Hopefully this will help someone out there and save them some head scratching J 

If your calling Web Services from .NET, BizTalk Server 2004 and you have lots of concurrent users: Your almost certainly experiencing this problem!

Comments (14)

  1. RobJ says:

    Hi Darren, good post – a real gotcha! The best article I’ve seen on ASP.Net threading is this one http://support.microsoft.com/default.aspx?scid=%2Fservicedesks%2Fwebcasts%2Fen%2Ftranscripts%2Fwct060503.asp

    Rob.

  2. Hi,

    Thanks for posting this information. We were having the exact same problem and did not know how to fix this. We put in the change that you suggested in the BTSNTSVC.exe.config and voila we are seeing more concurrent connections.

    Thanks once again.

    Radha.

  3. KP says:

    Darren this is great stuff. I’m assuming this also applies to the HTTP adapter? Do you know how (or if) this correlates with the HTTP adapter HttpOutMaxConnection setting in the performance white paper, that has a default value of 5?

  4. Darren Jefford says:

    The HTTP Adapter team seem to have exposed the maxConnection setting via a Registry Key, that should override anything you put in a config file, but your best trying it to be sure.

  5. KP says:

    Thanks Darren, our experiences seem to show the HttpOutMaxConnection registry key does indeed override it.

  6. Ben Harper says:

    This is great information. I am developing a BizTalk solution that works with Axapta through a webservice and was frustrated to find only 2 simultaneous connections. After playing with a registry tweak for Internet SettingsMaxConnectionsPerServer, it’s nice to have found an application-specific solution to this. I was even considering playing with the wininet.dll and doing some very tedious WinAPI coding. You saved me many frustrating hours.

  7. After reading a post from a coworker I found two links of interest regarding configuration changes required…

  8. Fix for security errors when installing BizTalk 2004 on Windows XP SP2 MSDN Biztalk Developer Center

  9. Anders06 says:

    “single-user client SHOULD NOT maintain more than 2 connections with any server or proxy”.

  10. Chris Williams says:

    How does this apply to thick desktop clients.   I would think the same would apply to outgoing connections as well.

    I see 2 TCP/IP reg keys  MaxConnectionsPerServer and MaxConnectionsPer1_0Serv

    Are these it?

    I have a C# application that calls a Java Webservice suite.  It makes around 23 webservice calls at the beging of the sessions and i want to optimize the client to make sure its making them all aysnc and not chunking them.