EWS Streaming Notifications with Exchange 2013
Below are some pointers on EWS streaming events with Exchange 2013/Exchange Online.
The basics:
Affinity in Exchange 2013/Wave 15 is handled the client application using cookies and headers. Subscription code needs to gather information using AutoDiscover, and create batch list of mailboxes to subscribe to then and set appropriate headers for the initial subscription request and for each subsequent EWS request related to the subscription. Subscriptions do break, so it’s extremely important to have a heavy amount of error and recovery checking. Also keep in mind that what goes over the wire and comes back on the wire is of equal or greater importance than your actual code – so be able to log traffic and review it when you run into issues.
It’s important that every developer read the following articles and go over their content several times to be sure they understand how writing code for steaming notifications works:
How to: Stream notifications about mailbox events by using EWS in Exchange
https://msdn.microsoft.com/EN-US/library/office/dn458792(v=exchg.150).aspx
Note that while this article has helpful information that it does not cover the needed affinity settings.
How to: Maintain affinity between a group of subscriptions and the Mailbox server in Exchange
https://msdn.microsoft.com/EN-US/library/office/dn458789(v=exchg.150).aspx
Note : This is probably the most important article on EWS Streaming notifications.
Handling notification-related errors in EWS in Exchange
https://msdn.microsoft.com/EN-US/library/office/dn458788(v=exchg.150).aspx
Notification subscriptions, mailbox events, and EWS in Exchange
https://msdn.microsoft.com/EN-US/library/office/dn458791(v=exchg.150).aspx
GetStreamingEvents operation
https://msdn.microsoft.com/EN-US/library/office/ff406172(v=exchg.150).aspx
Review this code to see how an application might be written:
EWS: Streaming Notification Sample
https://blogs.msdn.com/b/emeamsgdev/archive/2013/04/16/ews-streaming-notification-sample.aspx
Be sure to fully review this code before writing your own code - it may save you a good amount of and some grief.
Since EWS Impersonation will be used in code it’s important to understand how it works code-wise:
Impersonation and EWS in Exchange
https://msdn.microsoft.com/en-us/library/office/dn722377(v=exchg.150).aspx
Working with impersonation by using the EWS Managed API 2.0
https://msdn.microsoft.com/en-us/library/office/dd633680(v=exchg.80).aspx
The importance of EWS Impersonation while using an application account.
https://blogs.msdn.com/b/webdav_101/archive/2012/06/27/the-importance-of-ews-impersonation-while-using-an-application-account.aspx
Need to continue the connection every 30 minutes:
EWS Notifications hold an open connection to Exchange for 30 minutes in order to receive notifications for the subscription. After 30 minutes that connection will timeout and the connection close. Code needs to be written so that it catches the connection close event and reopens the connection. A closing connection won't kill the subscription but rather closes the connection in order to keep from holding up the server and max connections. So, if you lose the connection just reconnect. The general rule is don't have a connection open and listening for events unless you need to. Keep in mind that there is an open connection limit.
Prepare the service account and mailboxes:
Create an account for doing EWS Impersonation and be sure that its configured properly to access the mailboxes the code is going to be doing subscriptions against. Impersonation access is charged to the mailbox being accessed and not to the service account – this allows one account to access many mailboxes without encountering concurrency throttling limits.
How to: Configure impersonation
https://msdn.microsoft.com/en-us/library/office/dn722376(v=exchg.150).aspx
You should check to be sure that impersonation is working correctly using EWSEditor or by using the Microsoft Remote Connectivity Analyzer.
Microsoft Remote Connectivity Analyzer
https://testconnectivity.microsoft.com/
EWS Editor
https://ewseditor.codeplex.com/
Coding:
There are many steps involved in coding for streaming subscriptions. At times code will need to do autodiscover for an existing batch subscription from scratch and possibly create separate subscriptions. Subscriptions can and due break – so be sure to write fault tolerant code and be sure to have the ability to log traffic. Code written for Exchange 2010 won’t work properly with 2013 or Exchange Online since how streaming subscriptions work has changed.
With the EWS Managed API one Service object can be created and used with the impersonating account to do the needed subscription calls and for gathering the changes. So, one service object can handle the subscription or many mailboxes.
Code should be written to handle subscriptions on one thread and use other threads for processing the results. So, after calling GetStreamingEvents you can spawn new thread with the results in order to do processing. Be sure to NOT use EWS Managed API object across threads as they are not designed for such usage. Note that if you define an object such as a service object as a public static then you are exposing it across threads – so, don’t do that. When you need to open a new thread to do work you need to create a new service object to do the work.
You can use any language on any platform for using EWS calls. However, you will likely find that most samples are written in C# and most applications use the EWS Managed API.
EWS Managed API... open source? YES! It's now Open Source!!!
https://blogs.msdn.com/b/webdav_101/archive/2014/09/29/ews-managed-api-open-source-yes-it-s-now-open-source.aspx
Creating a subscription list:
Do AutoDiscover against every mailbox which needs to be subscribed to and gather needed info per the above document. Using this information create an initial list of mailboxes to subscribe to.
- Mailbox SMTP address.
- External EWS URL
- Grouping Information (AD site)
Build lists of mailboxes to batch subscribe with:
- This list needs to be sorted by External EWS URL first then by Grouping information.
- Now create separate lists per External EWS URL + Grouping information. Each of these lists will be used for individual batch subscriptions. If one of these new lists have over 200 items, then create more lists of up to 200 items. Each entry in the list always needs to have the same External EWS URL and Grouping information.
Note that the first mailbox in each of the resulting subscriptions lists is going to be used in several ways. This first mailblox will be the anchor mailbox and used in the X-AnchorMailbox header. The code will also be impersonating as that mailbox when subscribing.
The headers:
There are several headers which MUST always be set correctly in each EWS requests in order to for the subscriptions to work. Some developers gloss over these headers or don’t implement them correctly and even don’t implement them because they second-guess their importance – the end result is the code eventually fails.
- X-AnchorMailbox (this should be the first mailbox in the subscription list).
- X-PreferServerAffinity (set to true)
- X-BackEndOverrideCookie (This is returned from the initial subscription to the first mailbox in each subscription list – ie the anchor mailbox.)
When subscribing you be sure to specify an anchor mailbox to ensure the subscription code works. This is done by setting a "X-AnchorMailbox" header on every EWS request involved in the subscription to the SMTP address of one of the mailboxes in each list and the SMTP address which should be used is the STMP address of the first mailbox in the list. Setting the "X-AnchorMailbox" header helps Exchange route the traffic of the requests correctly.
Logging:
Be sure to add the ability to trace/log traffic when needed. This will likely help greatly when you run into issues and can be invaluable in a production environment. Exchange IIS logs can provide some information; however, in most cases involving subscription code the full content of the request and response (both headers and bodies) are needed in order to troubleshoot issues.
EWS Best Practices - Tracing and logging
https://blogs.msdn.com/b/webdav_101/archive/2015/05/03/ews-best-practices-tracing-and-logging.aspx
Be sure to set ReturnClientRequestId on the service object being used so that the client id will be in the traffic logged in traces and logs. This allows a person reviewing the traffic to tie client side traffic and server side traffic together. Set SendClientLatencieson the service object so that server information is returned when subscribing against an Office 365 mailbox. When SendClientLatencies is set on the request some diagnostic information will be in the EWS responses which can greatly help in diagnosing issues – such as the FQDNs of the mailbox and CAS server which processed the calls.
Instrumenting client requests for EWS and REST in Exchange
https://msdn.microsoft.com/en-us/library/office/dn720380(v=exchg.150).aspx
Errors:
As noted above, it is very important to code for errors. In some case you may need to re-subscribe from the start – yes all the way back to the initial AutoDiscover and batching. An example of this is when ErrorProxyRequestNotAllowed is thrown and which is usually caused by a failover or mailbox move. Office 365 will do failovers once a day or so to balance the load, for patching and maintenance - this will cause ErrorProxyRequestNotAllowed to be thrown as failovers are done to different sites and another site would have a different Grouping Information value.
In addition to streaming related errors, your code could run into Exchange throttling policy issues – so be sure to read on the subject. Keep in mind that throttling is your friend and it’s there to help be sure that programs don’t dominate a server. Every program should be able to self-throttle so that the fail-safe measures of Exchange throttling don’t kick in.
Handling notification-related errors in EWS in Exchange
https://msdn.microsoft.com/EN-US/library/office/dn458788(v=exchg.150).aspx
EWS throttling in Exchange
https://msdn.microsoft.com/en-us/library/office/jj945066(v=exchg.150).aspx
Exchange workload management
https://technet.microsoft.com/en-ca/library/jj150503(v=exchg.150).aspx