Best Practices - EWS Authentication and Access Issues
Authentication and access to a mailbox is an often misunderstood area. I'm going to cover Authentication and type of access (impersonation vs delegate access vs direct access) and common problems developers run into in this article.
Authentication vs Authorization.
Lets start from the beginning with some basic information on authentication and authorization, The first thing to keep in mind is that Exchange itself does not authenticate - IIS handles that part. Authorizing access to things in a mailbox is handled by Exchange. So, if you got an issue like using NTLM or BASIC authentication in an EWS POST then consider how the Exchange IIS is configured, how the client code is written to authenticate and what's happening between the client application and Exchange's IIS server that might mess with he call. If Impersonation or delegation don't work but you can authenticate then the issue is most likely going to be with Exchange or the client code and not with IIS as it would be a authorization issue.
The basics on EWS Authentication and Authorization:
You should read about the basics of authentication with EWS:
Authentication and EWS in Exchange
https://msdn.microsoft.com/en-us/library/office/dn626019(v=exchg.150).aspx
Impersonation and EWS in Exchange
https://msdn.microsoft.com/en-us/library/office/dn722377(v=exchg.150).aspx
With Impersonation a service account has full access to a defined set of mailboxes. What it can access in those mailboxes (such as specific folders) cannot be filtered or defined. Only an Exchange Admin can configure an EWS Impersonation account for impersonating and configure its mailboxes to allow the impersonation.
Delegate access and EWS in Exchange
https://msdn.microsoft.com/en-us/library/office/dn641957(v=exchg.150).aspx
Delegate access allows a user to access certain folders in another user's mailbox Delegate permissions can be set by a mailbox owner or administrator using an app or other app code.
Controlling client application access to EWS in Exchange
https://msdn.microsoft.com/en-us/library/office/jj900165(v=exchg.150).aspx
Exchange 2013 101 Code Samples
https://code.msdn.microsoft.com/office/Exchange-2013-101-Code-3c38582c
Look at the "Exchange 2013 Authenticate with EWS" sample.
So, when it comes to credentials used with EWS:
- Code needs to obtain credentials at runtime based-upon its runtime security context AND that context needs to be valid with the server/app its code is going against. This is what is used when default credentials with EWS are used – ie windows authentication. However, UseDefaultCredentials won’t work with Exchange 365.
ExchangeServiceBase.UseDefaultCredentials property
https://msdn.microsoft.com/en-us/library/microsoft.exchange.webservices.data.exchangeservicebase.usedefaultcredentials(v=exchg.80).aspx
You cannot use the default credentials of the logged on user if the user’s mailbox is hosted in Exchange Online or Exchange Online as part of Office 365. Instead, use the Credentials property to set the user’s credentials. The user’s credentials must be in user principal name (UPN) form for Exchange Online.
- The code needs to get a security token/context which will work with the server/application its code is going against by betting it from a third party using its current runtime context. This is what the oAuth approach does.
How to: Authenticate an EWS application by using OAuth
https://msdn.microsoft.com/en-us/library/office/dn903761(v=exchg.150).aspx
Authentication and EWS in Exchange
https://msdn.microsoft.com/en-us/library/office/dn626019(v=exchg.150).aspx
- The code needs to obtain stored credentials which come from a store. This is the usage of Basic and NTLM Authentication. One method to store credentials is to use Credential Manager. There are other ways to store credentials - some customer solutions I've seen are pretty creative.
Getting a 401 with NetworkCredentials and not WebCredentials:
Its best to use WebCredentials. Look at the samples in MSDN and you will see they use WebCredentials - see below:
Get started with EWS Managed API client applications
https://msdn.microsoft.com/en-us/library/office/dn567668(v=exchg.150).aspx
Setting the Exchange service URL by using the EWS Managed API 2.0
https://msdn.microsoft.com/en-us/library/office/dd633692(v=exchg.80).aspx
How to: Set the EWS service URL by using the EWS Managed API
https://msdn.microsoft.com/en-us/library/office/dn509511(v=exchg.150).aspx
Authentication with Exchange Online and UseDefaultCredentials:
The EWS Managed API's UseDefaultCredentials will not work with Exchange Online. You will need to specify credentials. Try using a WebCredential.
Connecting to EWS by using the EWS Managed API 2.0
https://msdn.microsoft.com/en-us/library/office/ff597939(v=exchg.80).aspx
Be sure to safely store the password or prompt for it.
How to store user credentials (XAML)
https://msdn.microsoft.com/en-us/library/windows/apps/xaml/hh465069.aspx
Windows Data Protection
https://msdn.microsoft.com/en-us/library/ms995355.aspx
Autodiscover is not working and I'm using an older version of the EWS Managed API, should I upgrade?:
Of course! We do fix stuff. Older versions of the EWS Managed API had issues with Autodiscover. As an example the EWS Managed API 1.2.1 fixes two Autodiscover redirect issues for scenarios in which an Exchange Server 15 user targets a request to an Exchange 2010 server.
Exchange Web Services Managed API 1.2.1 – now released!
https://blogs.msdn.com/b/exchangedev/archive/2012/06/18/exchange-web-services-managed-api-1-2-1-now-released.aspx
While the EWS Managed API is a pretty solid API there have been bugs which have been fixed. So, anytime you run into an issue with this API and are using an older version then you should test with the latest release to be sure your not dealing with something already fixed.
AutoDiscover is Slow with the EWS Managed API.
Yes, it can take a while. The best practice is to store the EWS web service URL set on the service object and use it next time instead of doing AutoDiscover again.
If you’re code is going against Exchange Online or have a custom domain then the URL is probably not going to change and you may get away with not doing AutoDiscover in the first place, However, you should have a fallback of redoing AutoDiscover if it does.
On the ExchangeService object there is a property called EnableScpLookup. By default, it is set to true. There are two types of Autodiscover used by the EWS Managed API. The first is the SCP AutoDiscover process, which will run if EnableScpLookup is set to true. SCP AutoDiscover will ping AD for information on possible servers to discover against then will do POSTs to try to get discovery information. If your server is in-network with your client application, then the SCP AutoDiscover process should be able to find the EWS web service URL for the mailbox. If SCP AutoDiscover fails then the EWS Managed API will then try POX AutoDiscover, which uses patterns based-upon the SMTP address to build URLS to do POSTs with in order to try to find the EWS web service URL. POX is used for out of network autodiscover – so it should be used with Exchange Online or when the client application is otherwise not in-network with the Exchange server. If EnableScpLookup is set to false then the SCP AutoDiscover process is skipped and the POX AutoDiscovery process is run.
So, why is this important? Well, AutoDiscover processes are expensive. So, only do it when you need to, but do it when its needed. If you are not caching the AutoDiscover results, then you are waiting time by doing it when it’s not needed. If you do SCP AutoDiscover when going against Exchange Online, then you are wasting time. As a real world example I’ve seen, if you AutoDiscover go against Exchange Online with EnableScpLookup set to true and the call takes 20 seconds to finish then around 18 seconds may have been taken up by the SCP AutoDiscover process doing LDAP calls against AD and the final two seconds were for POX AutoDiscover. If the server had been in-network, then it would have completed far faster. However, if EnableScpLookup was set to false then the AutoDisocvery call would have finished far faster since it would only have had to do POX Autodiscover. So, be sure to set EnableScpLookup correctly for every ExchangeService object you instance.
Autodiscover for Exchange
https://msdn.microsoft.com/en-us/library/office/jj900169(v=exchg.150).aspx
How to: Generate a list of Autodiscover endpoints
https://msdn.microsoft.com/en-us/library/office/dn467397(v=exchg.150).aspx
UPN works but not the SMTP address:
With a lot of code samples and articles it looks like you would use your SMTP address for doing authentication. However, this is not correct thinking. All credential matching with a user account is done with a UPN and not a SMTP address. So, what is actually happening is that the credential match is being against a UPN which matches that of your SMTP address. The way to resolve this is to add a UPN suffix which matches the SMTP address to Active Directory. See the article below on how to add a UPN suffix.
Add User Principal Name Suffixeshttps://technet.microsoft.com/en-us/library/cc772007.aspx
Always set X-AnchorMailbox when using EWS Impersonation:
When EWS Impersonation is used the X-AnchorMailbox always should be correctly set. Without doing so you may get 500 or 503 errors at times. It is critical for performance and also for notifications with Exchange Online/Exchange 2013. Not setting it can double or more the time it takes to complete the call. In some cases you can also get timeouts. The rule is to always set this header when using impersonation - this will make your EWS Impersonated code from Exchange 2007 work better with Exchange 2013. It should be set to be set to the mailbox being accessed with the exception of when streaming notifications are being done and in that case it should be set to the first mailbox in the subscription group. With Exchange Online there are additional headers which need to be set for affinity.
Example: service.HttpHeaders.Add("X-AnchorMailbox", targetSmtp);
Also see:
How to: Route public folder hierarchy requests
https://msdn.microsoft.com/en-us/library/office/dn818490(v=exchg.150).aspx
EWS Impersonation is not working:
There can be many causes for this. I’ll cover some of them.
Incorrect credentials:
Be sure the credentials are correct.
Try accessing the mailbox of the authenticating account using the owner’s credentials using EWS and OWA.
Try accessing the mailbox being accessed using the its owner’s credential using EWS and OWA.
Is EWS Impersonation setup?:
How to: Configure impersonation (2013)
https://msdn.microsoft.com/en-us/library/office/dn722376(v=exchg.150).aspx
Configuring Exchange Impersonation in Exchange 2010
https://msdn.microsoft.com/en-us/library/office/bb204095(v=exchg.140).aspx
Exchange Impersonation vs. Delegate Access https://blogs.msdn.com/b/exchangedev/archive/2009/06/15/exchange-impersonation-vs-delegate-access.aspx
SMTP Address not working with Impersonation but UPN does.
It needs to be a UPN. An SMTP address is not an UPN. However, you can have a UPN which matches. See the article below on how to add a UPN suffix.
Add User Principal Name Suffixes
https://technet.microsoft.com/en-us/library/cc772007.aspx
Did you set X-AnchorMailbox? Notice in this article that I mention setting this many times.
Example: service.HttpHeaders.Add("X-AnchorMailbox", targetSmtp);
Combining EWS Impersonation and Delegate Access is not working:
When code uses both EWS Impersonation and delegate access the call is subject to limitations and restrictions for both.
Be sure to try the following:
Check the Impersonation and general EWS throttling settings to be sure your under the limits set.
Because delegate access is in the mix your code will be subject to delegate access limitations, which include the maximum MAPI connection limit (MapiExceptionSessionLimit) which kick-in when too many delegate operations happen at the same time. If you don't need to be using delegate access in the EWS call then consider removing it.
Be sure to set the X-AnchorMailbox header to the mailbox being accessed or you might run into errors on some EWS calls when working with items - such as creating a meeting in a delegate calendar.
Be sure that EWS Impersonation and Delegate access work without a combined call.
EWS Impersonation used against 2016 CAS with a 2010 mailbox server fails:
This is another area where you should try setting the X-AnchorMailbox header to the mailbox being accessed.
Delegate access results in a MAPI connection limit errror (MapiExceptionSessionLimit):
Yes this says MAPI - the throttling limits for delegate access with EWS are tied the same limits for MAPI on Exchange. These errors are usually intermittent. In the response Below are the reasons which I see show up often from our customers:
- The code is exceeding the limit.
- The limit is smaller than you think it is. Recheck it and adjust as needed.
- There are other applications using the same account which your application is using. Its best that service style applications have their own account to run under to avoid this situation.
- The code is mixing EWS Impersonation and delegate access and don't realize it.
- The code is accessing an item by ID and it actually is for an item in a mailbox other than you think it is and implicit delegate access is being used.
- There is a load balancer which is causing the issue. With some load balancers this can happen when its CPU hits 85%. This happens when the load balancer drops the connection and Exchange is not notified, so it keeps the connection alive based-upon the keepalive setting - the default is two hours. Try a test by bypassing the load balancer to see if the issue goes away so that you can rule it out - this is a necessary troubleshooting test.
Impersonation credentials on a thread while using EWS does not work:
Thread impersonation with messaging APIs is not supported and that includes EWS. Exchange was neither designed nor tested for calls which are done with thread impersonation.
Getting intermittent 500 or 503 errors when using EWS Impersonation:
These can be caused if the X-AnchorMailbox has not been set. When EWS Impersonation is used the X-AnchorMailbox always should be correctly set when going against Exchange 2013/Exchange Online/Office 365. The rule is to always set this header when using impersonation. It should be set to be set to the mailbox being accessed with the exception of when streaming notifications are being done and in that case it should be set to the first mailbox in the subscription group.
Example: service.HttpHeaders.Add("X-AnchorMailbox", targetSmtp);
EWS using Impersonation is slower then when not impersonating.
These can be caused if the X-AnchorMailbox has not been set when going against Exchange 2013/Exchange Online/Office 365. Usually the performance hit is seen when the mailboxes are in different stores and can be quite profound when they are on different stores in different continents. When EWS Impersonation is used the X-AnchorMailbox always should be correctly set. The rule is to always set this header when using impersonation. It should be set to be set to the mailbox being accessed with the exception of when streaming notifications are being done and in that case it should be set to the first mailbox in the subscription group.
Example: service.HttpHeaders.Add("X-AnchorMailbox", targetSmtp);
I cannot find a way to have EWS Impersonation only access specific folders:
EWS Impersonation does not do this. You cannot lock down access to specified folders or limit rights. When using EWS Impersonation the service account is effectively at the level of the mailbox owner. Keep in mind that EWS Impersonation is for apps to have full access to a mailbox to do whatever is needed and is not subject to a user changing things like delegate permissions on a folder. Remember, you can limit the scope of mailboxes that an impersonation account can reach and secure that account as you would any other admin type account.
The importance of EWS Impersonation while using an application account.
https://blogs.msdn.com/b/webdav_101/archive/2012/06/27/the-importance-of-ews-impersonation-while-using-an-application-account.aspx
EWS cannot access public folders or their items with Exchange 2013/ExchangeOnline/Office 365:
Check to see if you have set the X-AnchorMailbox and X-PublicFolderMailbox headers - they will be needed for Exchange 2013/Exchange Online/Office 365.
How to: Route public folder hierarchy requests
https://msdn.microsoft.com/en-us/library/office/dn818490(v=exchg.150).aspx
How to: Route public folder content requests
https://msdn.microsoft.com/EN-US/library/office/dn818491(v=exchg.150).aspx
How to access the archive folder when user’s mailbox is on-premise and the archive mailbox is in the cloud
When a mailbox is on-premise and the archive mailbox is in the cloud you have to do some extra work to access the cloud archive. The following article covers this:
Cannot access the public folder with EWS:
When accessing public folders you should set two headers to help in assuring success. These two headers are X-AnchorMailbox and X-PublicFolderMailbox. Please read the articles below.
Public folder access with EWS in Exchange
https://msdn.microsoft.com/en-us/library/office/jj945067(v=exchg.150).aspx
How to: Route public folder hierarchy requests
https://msdn.microsoft.com/en-us/library/office/dn818490(v=exchg.150).aspx
Note that if your using EWSEditor for testing that you can set X-AnchorMailbox on the main login window and via the Global Options button on the log-in window (there is a place to set any 3 headers).
When doing AutoDiscover for a mailbox which is under Exchange 2010 and the CAS is 2013 you get back GroupingInformation.
GroupingInformation (the site name) is only set for mailboxes under Exchange 2013 and later. A mailbox under Exchange 2010 has no GroupingInformation. for Exchange 2010 and earlier you will need to use PowerShell calls or maintain a list/database of where the mailbox resides.
If an impersonating account with a mailbox under 2013 is used with a call to get Autodiscovery information on a mailbox under a 2010 and the x-anchormailbox header is not set to that 2010 maibox, you will get back the grouping information for the impersonating account's mailbox. If you set the x-anchormailbox when to the 2010 mailbox then you will get back nothing for GroupingInformation, which is the correct value. If the Impersonating account does not have an Exchange mailbox but rather has an associated SMTP address then GroupingInformation will be properly returned when doing an AutoDiscover against a 2010 mailbox.
I don't want to store a password or use oAuth but want to access Exchange Online.
Well, that's not going to be possible. You need to either pass a password which you have stored somewhere or use oAuth.
How can I safely store my password and retrieve it?
There are different ways to store a password and one way is to use the credential manager. I'm not a security expert, so I'm not going to make recommendations and approaches to securing passwords. However, I have seen and heard of this this approach used often. Below are samples on working with Credential Manager.
CredUIPromptForCredentials function
https://msdn.microsoft.com/en-us/library/windows/desktop/aa375177(v=vs.85).aspx
Asking the User for Credentials
https://msdn.microsoft.com/en-us/library/windows/desktop/ms717794(v=vs.85).aspx
Exchange 2013 101 Code Samples
https://code.msdn.microsoft.com/office/Exchange-2013-101-Code-3c38582c
Look at the "Exchange 2013 Authenticate with EWS" sample.
Please consider using the Microsoft Threat Modeling Tool for assessing your current security. This will help you follow The Microsoft Security Development Lifecycle (SDL).
Microsoft Threat Modeling Tool 2014
https://www.microsoft.com/en-us/download/details.aspx?id=42518
I don't want to use Basic authentication because its not secure:
Yes, Basic authentication is very easy to decode... so, why aren't you encrypting the traffic? SSL encryption should always be used to secure traffic. This is what clients including our own do. Protocols such as EWS and EAS (Exchange Server ActiveSync) often are used with Basic Authentication and the encryption is SSL encrypted. Commercially sold phones usually use EAS use Basic authentication with SSL encryption.
.NET has classes and methods for storing data. Data can also be further encrypted. If you decide to store the password then be sure to secure it.
Isolated Storage
https://msdn.microsoft.com/en-us/library/3ak841sy(v=vs.110).aspx
OAuth could be configured for Exchange Online servers, however you need to have additional setup with Azure services.
Authentication and EWS in Exchange
https://msdn.microsoft.com/en-us/library/office/dn626019(v=exchg.150).aspx
How to: Authenticate an EWS application by using Oauth
https://msdn.microsoft.com/en-us/library/office/dn903761(v=exchg.150).aspx
Have you looked at oAuth 2.0?
oAuth lets your application leverage Azure services to handle the credentials so that it can access Exchange Online. This avoids your application passing a password to Exchange. oAuth currently only works against Exchange in Office 365. The application used needs to have full mailbox access permission. This type of authentication does need a bit of setup in order to work.
These articles explain how oAuth can be used with Exchange Online:
Office 365 app authentication concepts
https://msdn.microsoft.com/en-us/office/office365/howto/common-app-authentication-tasks
oAuth authentication
https://msdn.microsoft.com/en-us/library/office/dn626019(v=exchg.150).aspx#sectionSection0
Using OAuth2 to access Calendar, Contact and Mail API in Office 365 Exchange Online
https://blogs.msdn.com/b/exchangedev/archive/2014/03/25/using-oauth2-to-access-calendar-contact-and-mail-api-in-exchange-online-in-office-365.aspx
How to: Authenticate an EWS application by using OAuth
https://msdn.microsoft.com/en-us/library/office/dn903761(v=exchg.150).aspx
Using an ISAPI filter to change credentials for doing single sign-on:
This is not a supported approach.
Using ISAPI Extensions to change-out OWA Credential is not supported
https://blogs.msdn.com/b/webdav_101/archive/2007/05/04/using-isapi-extensions-to-change-out-owa-credential-is-not-supported.aspx
Microsoft does not support using ISAPI extensions or filters to modify Outlook Web Access credentials on a server that is running Exchange Server
https://support.microsoft.com/en-us/kb/938609/
If you want to test authentication and access types then then you can use the following:
EWSEditor
https://ewseditor.codeplex.com/
Remote Connectivity Test (RCA)
https://testconnectivity.microsoft.com
This is a Microsoft web site used for testing connectivity. It has a web based tester and also one for running from a desktop.
Exchange 2013 101 Code Samples
https://code.msdn.microsoft.com/office/Exchange-2013-101-Code-3c38582c
Look at the "Exchange 2013 Authenticate with EWS" sample.
How do I get delegate access to work between Exchange Online and my on-premise server?
That won't work:
Cross-premises Calendar editing is unavailable in a hybrid deployment of Exchange Online in Office 365 and on-premises Exchange Server
https://support.microsoft.com/en-us/kb/2807149
My JAVA code does not work when using NTLM against Exchange; however, EWSEditor and other .NET code works... why?
The IIS server is what does authentication, so the issue here does not really involve EWS and the issue should repro with any other app running on the same version of IIS which has the same auth settings. The versions of IIS used by Exchange need to be authenticated with at least NTLMv2. Older versions of JAVA libraries don't support NTLMv2, so they won't work. And yes, that includes older Apache libraries. The JAVA Managed API also has a minimal version of JAVA it needs in order to work. So, be sure your using libraries which support NTLMv2. If your JAVA code forces you to stick with an older version of libraries then consider creating a middle man component which can accept the calls from your existing app and do calls against Exchange using NTLMLv2 or try using Basic authentication or oAuth.
In almost all cases I can recall all times when there is an issue with JAVA code doing calls with NTLM and it failing against Exchange it has been an issue with out dated libraries or with bugs in those libraries or the developer decided to create their own libraries by mixing code and creating something which did not work. The exceptions to those scenarios is where something between the client and server was causing issues - I haven't seen that happen for a long time though.
Using EWS to access an Exchange 2007 or 2010 public folder when you have a 2013 mailbox.
If you are using EWS with a 2013 mailbox and try to access a public folder on an Exchange 2007 or 2013 server then you will see the following error:
There are no public folder servers available.
The reason you get this error is that Exchange 2013 does not support that access scenario.
Please review the following:
Considerations when deploying public folders
https://technet.microsoft.com/en-us/library/dn957481(v=exchg.150).aspx
This is the part to focus on:
Exchange 2013 no longer supports public folder databases.
Therefore, there’s no coexistence with legacy public folders. As a result,
Exchange 2013 is unable to read from the hierarchy stored in a public folder
database on Exchange 2010 or Exchange 2007 servers.
Cannot access the Journal folder with delegate access:
These can be caused if the X-AnchorMailbox has not been set. Set this header to the destination mailbox.
Example: service.HttpHeaders.Add("X-AnchorMailbox", targetSmtp);
Cannot access a shared calendar folder with delegate access:
These can be caused if the X-AnchorMailbox has not been set. Set this header to the destination mailbox.
Example: service.HttpHeaders.Add("X-AnchorMailbox", targetSmtp);
Deleted item still shows in GetUserAvailabiltiy:
Set the X-AnchorMailbox header to the destination mailbox.
Example: service.HttpHeaders.Add("X-AnchorMailbox", targetSmtp);
How do I troubleshoot?:
Here are general troubleshooting steps:
Tools and resources for troubleshooting EWS applications for Exchange
https://msdn.microsoft.com/EN-US/library/office/dn720300(v=exchg.150).aspx
Use the Exchange Remote Connectivity Analyzer (EXRCA) to see if the same issue happens. Note that in addition to the web client that there is a downloadable desktop tester. This app is a Microsoft base line tester for connectivity. If the same issue your code is running into reproduces with EXRCA then the issue is most likely not with your code.
Exchange Remote Connectivity Analyzer
https://technet.microsoft.com/en-CA/library/ff701693(v=exchg.150).aspx
Try to reproduce the issue with EWSEditor. This is an extensive open sourced sample which uses the EWS Managed API and raw EWS POSTs. You can use it to submit EWS posts with the same EWS body using its EWS POST window.
EWSEditor
https://ewseditor.codeplex.com/
Try using EWSEditor or a repro application at different places between your application and Exchange. Note that EWSEditor is an unsupported sample.
On the same machine as your code.
On the Exchange CAS server.
Bypass load balancers and run from the machine running your code. Load balancers can cause issues, so ruling them out is important.
Run the code from a machine other than the one your code runs from.
If your applications machine is in network then try running your code out of network. If your application is an out of network machine then try running on an in-network machine.