17 hours of work because of remoting.......

Last night we stayed 17 hours at work trying to figure
out why we can't load the remoted server with more than 250 Remote calls per
sec.

we had a server running asp.net which called another server thru
remoting.

we could get the 2nd server to load, we even tried more than 1 web server at
the time.

it seemed we had a bottleneck somewhere in the 2nd server which led us to
believe Remoting had a limit.

on every asp.net hit we've noticed 10 times Remoting Calls, we were sure no
more than 1 remoting call is performed from the asp.net page for every hit.

we debugged some more, and we dicovbered that because the 2nd server exposes
it's Com+ objects thru remoting and calls inner com+ objects , the total number
of calls is greater than we anticipated,

a small test (replacing all the Serviced compoenent with MarshalByRef) did
decrease the number of remoted calls.

but still we couldn't find why we can't get more than 250 calls/sec and the
CPU was not loaded.

we wrote a small remoting Server that returned the time,

tried to hit it thru asp.net (iis6 and 5.1) , both of them died after 30 sec.
of full stress,

but even for the 30 sec there was no serious stress on the Remoted
server.

I thought it was connected to the ThreadPool (that has a limitation of 25
threads per CPU) knowing that remoting is using the ThreadPool, I've add more
thread (with the help of Thread Pool Control Sample from DevelopMentor.com )

still nothing, it didn;t help, we tried to chnage the asp.net server
WorkerThreads thru the machine.config and still nothing.

last but not least we wrote a winform client that calls the remoted server,

and then..... we got about 2000 hits per sec.....

seems like the real bottleneck was asp.net......