Potential risks of using unsecure RPC (No Authentication Required setting) in MSDTC security configuration

  • Article

When your MSDTC machine is accessible from an un-authenticated client, i.e. machine outside your domain/firewall. Then when such a machine tries to talk to this MSDTC server, the MSDTC server will accept such requests, so in all it could allow requests from a machine outside the domain(as long as it can reach the MSDTC server/ports) to start/end a distributed transaction. Which in the most simple cases can cause overload on the MSDTC process or can make it unnecessarily slow.

The more potent version of the attack would be if an vulnerability in RPC is exposed, then an outside domain machine might have access to the process (which in this case will be running as a low privilege network access account).

So in the worst case the MSDTC service will be overloaded with requests and when you try to start a new transaction you will get a “Denial of Service” error and your request might not be honored.

To give you the background, on windows 2000 server, there were no security features available in MSDTC service and it used to run under the high privilege Local System account. Thereafter, these new security features were introduced on windows 2003 that gives you the option to select different levels of security. However, with that we also changed the account under which DTC service would run, i.e. NT Authority\NetworkService account (903944)which is a low privilege account. Therefore, even if we open the security (No Authentication Required) which is a requirement with the cluster environment, it would not cause much problems on the server.

For more information about these MSDTC security settings you may refer to the following kb article :

New functionality in the Distributed Transaction Coordinator service in Windows Server 2003 Service Pack 1 and in Windows XP Service Pack 2

https://support.microsoft.com/?id=899191

Per this article :

In a clustered environment, the computer account for the Distributed Transaction Coordinator service specifies the cluster node's host name. In a clustered environment, the Distributed Transaction Coordinator authentication does not use the transaction mode's host name. In a clustered environment, the transaction mode's host name is the name of the virtual service. Therefore, you cannot use the Mutual Authentication Required transaction mode in a clustered environment. You can use the Mutual Authentication Required transaction mode between two nonclustered computers that are running Windows Server 2003 SP1 or between two computers that are running Windows XP SP2.
You must use the Incoming Caller Authentication Required transaction mode between Windows Server 2003-based computers in a clustered environment.
You must use the No Authentication Required transaction mode where one or more of the following conditions are true:

  • The network access is between computers that are running Microsoft Windows 2000.
  • The network access is between two domains that do not have a mutual trust configured.
  • The network access is between computers that are members of a workgroup.

Even for the Incoming Caller Authentication Required setting to work in the cluster environment you need to have the environment configured correctly for the Kerberos authentication. And again this might break if there is a Windows 2000 server coming into picture as it would only work if we have set No Authentication Required throughout on all the Windows 2003 server machines.

Based on the explanation above, we recommend having “No Authentication Required” setting on the cluster and the other standalone server interacting with the cluster to avoid running into any compatibility issues with older operating systems and authentication failures.