Using MSDTC between Vista clients and Windows 2000 servers

Background

Consider the following two scenarios:

1.       You have implemented a .NET application accessing a specific COM+ service with automatic transaction processing.  You derived a class from ServicedComponent, set the Transaction attribute for the class, and applied the Automatic attribute to a method which opens a SqlConnection.

2.       Inside a TransactionScope, you wanted to open a SqlConnection enlisting to a distributed transaction.

In both scenarios, since you’ve used the services of the Microsoft Distributed Transaction Coordinator (MSDTC), you need to configure and start the MSDTC service. Assuming SQL Server is running on a Windows 2000 machine, everything works fine until you upgrade your client to Windows Vista. You’re going to see the following exception:

Unhandled Exception: System.Transactions.TransactionException: The transaction has already been implicitly or explicitly committed or aborted. —> System.Runtime.InteropServices.COMException (0x8004D00E): The transaction has already been implicitly or explicitly committed or aborted (Exception from HRESULT: 0x8004D00E)

Solution

The solution to the problem is simpler than you might imagine: make a simple change to the MSDTC settings in your Vista client. The tricky part, however, is how to set it. I’d like to share the steps:

a.       Run dcomcnfg command (don’t tell me you couldn’t find the run command:)

b.      Expand the “Component Services” node, then the “My Computer” node, then the “Distributed Transaction Coordinator” node

c.       Right-Click on the “Local DTC” node and select “Properties”

d.      On the “Local DTC Properties”, select the “Security” tab

e.      Select Network DTC Access, Allow Inbound, Allow Outbound, No Authentication Required, and Enable XA Transactions

So here is the difference. In Windows XP SP2 or Windows 2003 SP1, you only need to choose Incoming Caller Authentication Required. In Vista, however, when you’re trying to talk to a Windows 2000 Server through MSDTC service, you have to lower the security level of Transaction Manager Communication to No Authentication Required.

Now you’ve solved your problem. If you need to know why or have concerns about the security (it’s supposed to be more secure with Vista, isn’t it?), please continue to read.

Explanation

Since I’m working on ADO.NET, I’d like to quote the explanation from a developer Jim Carley in MSDTC team:

In Vista, we tightened down the MSDTC security even more than in W2k3SP1. We added code that, after doing an RpcImpersonateClient, obtained the token of the caller from the thread and checked to see if that token was in the Authenticated Users group.  The tokens that come in from W2k systems are “anonymous”, so aren’t considered in the Authenticated Users group.

On W2k3SP1, we only do a RpcImpersonateClient. We don’t check for Authenticated Users membership.

That is why “Incoming Caller Authentication Required” works on W2k3 SP1 and not on Vista.

W2k does not have the code in MSDTC to establish the right security settings expected by MSDTC in later releases, thus the need to lower the RPC security level.

If XPSP2 or WS03 SP1 systems are able to exchange transactions with a W2k system at the “incoming authentication required” level of security, they are getting lucky. There are situations where this configuration needs to be lowered to “no authentication required” also.

You probably don’t understand every technical detail (it’s very hard for people outside MSDTC team to completely understand how everything works), but the important fact to know is that in Vista, MSDTC security is enhanced and you have to lower the security level to talk with a Windows 2000 server.

Regarding the security concern, according to what Jim said, lowering the security level does open up opportunities for anonymous attacks on the RPC interfaces. Fortunately, Jim also mentioned that in Vista, there have been other changes that make the RPC-facing code more robust, thus strengthening the security even if the caller gets through the RPC security stuff.