You are doing too much at one time: EWSMaxConcurrency – “too many concurrent connections opened”

  • Article

At any given time, you can have up to a specific amount of connections involving an Exchange mailbox or involving a delegate account accessing mailboxes. For an analogy, you can have 12 eggs in a carton and only up to 12. You can change out the eggs and have less, but you cannot add more than 12 at any given time – you will end up with a cracked egg if you try to go over the limit. So, for the carton the max concurrent number of eggs it can contain is 12. Hitting the EWSMaxConcurrency limit is like trying to cram one too many eggs in a carton.

The cause of the EWSMaxConcurrency error is that based-upon the type of access that at a moment in time there are too many connections to a mailbox or with an account accessing mailboxes. The type of access (Owner, Delegate, Impersonation) will make a difference in how to view the limits of the EWSMaxConcurrency throttling policy. Based upon type of access the active connections will either be charged to the service account (owner and delegate) or against the mailbox which is being accessed (Impersonation). This EWSMaxConcurrency limit is 10 for Exchange 2010 and 27 for Exchange Online, however it can vary depending upon the server version and if this limit was overridden by an admin (not reccomended.

    EWS throttling in Exchange
https://msdn.microsoft.com/en-us/library/office/jj945066(v=exchg.150).aspx

A change to the throttling policy should only be considered if there is no way or no time to work-around the issue in the near term. For Exchange Shared mailboxes a throttling policy change EWSMaxConcurrency is not possible with the possible exception being that you are migrating to Exchange Online and a bypass of this policy is not going to be forever. In the end applications need to self-throttle and be built to scale.

Note: This error is a very specific one which is tied to EWSMaxConcurrency and has nothing to do with other polices such as ServerBusyException and FindCountLimit – those are very different polices.

Accessing your own mailbox:

When accessing your own mailbox using your owner account, your account will be charged for each connection to the mailbox. Usually applications do not run into issues with this type of access except for some special purpose applications which have high activity against a single mailbox as its owner.

Delegate access to mailboxes:

Access to mailboxes using delegate access (folder level and full mailbox delegate access) will cause the connections to be charged to the service account. Often applications using this method of access are doing calls to many mailboxes, so there is a chance of getting throttled during high activity with multithreaded calls unless the application is self-throttling.

Keep in mind that its possible to mix delegate access with Impersonation. Sometimes developers write code to initially to do delegate access then change to impersonation but don't clean up their code and the calls end up mixing impersonation and delegate access. So, watch out for this scenario if you are using Impersonation but still getting EWSMaxConcurrency throttled. To check, look at your code and look at the request body – you may see either an STMP address for another mailbox (explicit delegate access) or an item/folder id for something in another mailbox (implicit delegate access).

EWS Impersonation to mailboxes:

Access using EWS Impersonation will cause the connection count to be charged to the budget of the mailbox being accessed. Since mailboxes have a high limit for simultaneous connections under the EWSMaxConcurrency policy limit, the chances of being throttled on EWSMaxConcurrency are very low. As an example, a mailbox by default can have up to 27 simultaneous connections at any given point and that far more than most will ever need. The primary exception to this is with types of special purpose applications where there is a very, very high against one mailbox.

You might run into the scenario where many applications or even one hitting the same mailbox at the same time. In this case the EWSMaxConcurrency throttling will be due to the collective in-process calls to the mailbox. Keep in mind that if they mix in delegate access then the story changes and you need to also consider the throttling involved with delegate access.

Work-around suggestions:

There are ways to prevent hitting the EWSMaxConcurrency while accessing a mailbox. The best overall approach is to use EWS Impersonation access when possible.

In general:

  • Optimize your code so that calls do not run a long time. Examples are to optimize what you can for searches, only pull back the needed properties, simplify calls. The shorter the time of the calls the less chance of calls overlapping in the same time. This is also a best practice as it lowers the load on Exchange and also can provide a better time in overall processing.
  • Add a time delay between calls. This may help in that it gives breathing space between calls so that calls done on other threads or by other applications can do their work and not contribute to throttling. The delay does not need to be very long. You will need to decide if this approach will work for you based upon the processing needs of your application.
  • Limit the number of threads doing calls at one time.
  • A processing queue could be used: This goes back to old-school queue processing. The idea is to put objects into a queue for each EWS call to be made. An application would then read the queue and do EWS calls at a specified rate. This method can be used for limiting calls made at a time and over time and can be used for working around several throttling limits. It's not a popular approach, however it is an option.

Owner access:

  • If you run into this limit you may be forced to choose between delegate access using multiple accounts and increasing the limit.

Delegate access:

  • Recoding to use EWS Impersonation is usually the best approach to avoid this type of throttling since the connections are charged to the throttling budget of the mailbox being accessed and not the service account and its unlikely that the total active number of connections to reach the EWSMaxConcurrency policy limit for a mailbox.
  • For delegate access, alternate accounts used for doing calls. This works in some cases; however, some find they need keep adding code to add additional accounts and a pool of accounts is helpful in that case.
  • Use a round-robin queue of accounts for doing the calls. This allows a higher level of scaling with less maintenance when a new account needs to be added.
  • Use an account which is only used by the application. If another application is using the same account, then it will add to the connection limit for the account you are using.

Impersonation:

  • If you run into this issue while using Impersonation, then look at your code and what goes over the wire to be sure that there is not a mixture of Impersonation and delegation access. Keep in mind that if you mix the two that you are subject to the throttling polices of both.

History and perspective:

In older versions of Exchange (2007 and prior) there was almost no throttling. Customers would have to pay attention to applications they were running against Exchange and to be sure that applications were not dominating servers or otherwise causing performance issues. When there were few applications going against Exchange it usually worked out. However, the world changed and a lot more applications were going against Exchange with Exchange Web Services (EWS), Exchange Server ActiveSync (EAS) and other APIs. Some of those applications were written in ways only taking in consideration of the performance of that application not considering the impact on the servers overall. Performance issues were often not found because programmers were not going against real life production servers in testing and checking performance impact or issues were introduced release by updates. The traffic from those applications was enough to bring Exchange servers to their knees and effectively taking them down. Something had to change to improve situation and to help prevent call traffic which would have such undesirable impact on Exchange servers. In Exchange 2010 throttling polices were added to force applications to be better behaving so as to not cause such issues. Exchange 2013 added more polices and improvements. Now applications need to behave by self-throttling so that Exchange won't have to.

Please review my blog article on throttling and each of the links it contains:

Exchange throttling is your friend… well, more like a police officer.
https://blogs.msdn.microsoft.com/webdav_101/2015/11/18/exchange-throttling-is-your-friend-well-more-like-a-police-officer/

Streaming Notifications:

Code doing streaming notifications should have separate accounts used for handling the subscription versus processing the subscription.  As an example, the subscription thread would spawn worker threads which would use an account that would process the notification.  The subscription handling the account will be under special throttling handling and the account(s) which will use the notification information returned from Exchange would be under normal throttling policies.  Note that Impersonation should be used for Streaming notifications.

Delegate access results in a MAPI connection limit errror (MapiExceptionSessionLimit):

Yes, this says MAPI - the throttling limits for delegate access with EWS are tied the same limits for MAPI on Exchange.  These errors are usually intermittent.   Below are the reasons which I this error show up often from our customers:

  • The code is exceeding the limit.
  • The limit is smaller than you think it is.  Recheck it and adjust as needed.
  • There are other applications using the same account which your application is using.  Its best that service style applications have their own account to run under to avoid this situation.
  • The code is mixing EWS Impersonation and delegate access and don't realize it.
  • The code is accessing an item by ID and it actually is for an item in a mailbox other than you think it is and implicit delegate access is being used.
  • There is a load balancer which is causing the issue.  With some load balancers this can happen when its CPU hits 85%.  This happens when the load balancer drops the connection and Exchange is not notified, so it keeps the connection alive based-upon the keepalive setting - the default is two hours.  Try a test by bypassing the load balancer to see if the issue goes away so that you can rule it out - this is a necessary troubleshooting test.

Using Impersonation instead of Delegate access will help prevent seeing both MapiExceptionSessionLimit and EWSMaxConcurrency throttling.

Why you might not have been throttled before:

Many applications which use EWS may not see any throttling issues for a long time then start to encounter them. This is most likely due to changing conditions which could include the number of items being processed and the type of data involved changing. Mailboxes grow, and companies add users – as this happens, applications which service mailboxes will usually generate a higher load against Exchange. The calls also take a longer time as a server's load increases, which increases the number of outstanding calls because the calling applications must wait longer (unless they are all self-throttled). Code modifications can of course increase load. Exchange is always for watching performance issues related to calls being made to it. So, at some point limits get exceeded an application gets throttled. So, writing code to be highly efficient in their call and doing things like using an around robin queue of service accounts are important to do from the start.

Overall Best Practices to help with troubleshooting and prevent issues:

When there is a failure in an application there should be the ability to turn detailed logging at any time. This is critical for troubleshooting. Implementation of detailed logging and adding things to requests to help with tracking calls can make a huge difference in resolving issues. An issue which might be resolvable in hours might take days or longer if such things are not implemented from the start.

All code should have the ability to log full information on the EWS call.  This includes at least:

  • The request body and headers.
  • The response body and headers.
  • Error message.
  • The error call stack.
  • The HTTP response code.
  • The time and time zone of each call.

The following should always be set to provide better efficiency and tracing of the calls:

Each request should always have the User-Agent header set to identify it in traffic.

oExchangeService.UserAgent = "MyApplication";   

The "client-request-id" and "return-client-request-id" headers (ClientRequestId and ReturnClientRequestId on the EWS Managed API ExchangeService object) should be set for every call and logged.  Those headers identify a call to the server and back to the client and are often needed for troubleshooting – especially with Exchange online.  "client-request-id" will need to be set to a new GUID per call.

oExchangeService.ClientRequestId = Guid.NewGuid().ToString();  // Set a new GUID on each call.

oExchangeService.ReturnClientRequestId = true

The "X-AnchorMailbox" header with the SMTP address of the target mailbox should always be added.  "X-PublicFolderMailbox" and "X-AnchorMailbox" should always be set when accessing a public folder.  "X-AnchorMailbox" this may make a difference in performance and is sometimes needed for EWS calls.   Examples:

oExchangeService.HttpHeaders.Add("X-AnchorMailbox ", "mytargetmailbox@contoso.com");  // Set on every call – be sure to set it to the target mailbox or the Anchor mailbox for Streaming notifciations.

oExchangeService.HttpHeaders.Add("X-PublicFolderMailbox",  "mypublicfolder@contoso.com");  // For a call to a public folder

Those headers might seem like unnecessary work; however, they are important to set. The X-headers can increase performance and are often needed to get calls to work. The User-Agent, "client-request-id" and "return-client-request-id" are important as they help identify the traffic against the server and are critical for finding the severs which handled the call. The response headers have information like the names of the servers which processed a request. Time and time zone information logged is often critical in being able to find the server logs and their entries related to EWS calls. Mailbox servers are often in different time zones from an application and many server's failover to sites in other time zones.

IIS logs are seldom helpful for resolving issues with APIs/protocols like EWS – they just don't have enough information. So, be sure to be able to log needed info on the fly.

References:

EWS throttling in Exchange
https://msdn.microsoft.com/en-us/library/office/jj945066(v=exchg.150).aspx

Best Practices - Throttling
https://blogs.msdn.microsoft.com/webdav_101/2015/05/03/best-practices-ews-throttling/

EWS throttling in Exchange (EWS Managed API | Exchange Online | Exchange Server 2013 | Office 365)
https://msdn.microsoft.com/en-us/library/office/jj945066(v=exchg.150).aspx

Exchange throttling is your friend… well, more like a police officer.
https://blogs.msdn.com/b/webdav_101/archive/2015/11/18/throttling-is-your-friend-well-more-like-a-police-officer.aspx

New-ThrottlingPolicy
https://technet.microsoft.com/en-us/library/dd351045(v=exchg.150).aspx

Understanding Client Throttling Policies (Exchange Server 2010 SP3, Exchange Server 2010 SP2)
https://technet.microsoft.com/en-us/library/dd297964(v=exchg.141).aspx

EWS Best Practices – Searching
https://blogs.msdn.microsoft.com/webdav_101/2015/05/03/ews-best-practices-searching/

Message properties indexed by Exchange Search
https://technet.microsoft.com/en-us/library/jj983804(v=exchg.150).aspx

Best Practices – EWS Authentication and Access Issues
https://blogs.msdn.microsoft.com/webdav_101/2015/05/11/best-practices-ews-authentication-and-access-issues/

How to: Perform paged searches by using EWS in Exchange
https://msdn.microsoft.com/en-us/library/office/dn592093(v=exchg.150).aspx

Throttling Policies and the EWSFindCountLimit
https://blogs.msdn.com/b/exchangedev/archive/2010/03/12/throttling-policies-and-the-ewsfindcountlimit.aspx

EWS ServerBusyException – The server is too busy – for you
https://blogs.msdn.microsoft.com/webdav_101/2018/03/20/ews-serverbusyexception-the-server-is-too-busy-for-you