Issue with Incoming Caller Authentication for MS DTC on Windows 2008 and Windows 2008 R2 Cluster

Issue:
When you run an application that tries to connect to the SQL cluster on Windows 2008 or R2 for distributed transactions, you may get the following error if Incoming Caller Authentication Required is set for MS DTC:

"New transaction cannot enlist in the specified transaction coordinator"

You can try No Authentication Required as a workaround but it may not work. You may encounter the problem even with the no authentication required setting though the errors in DTC CM trace may be slightly different. The client operating system could be either Windows 2003 or Windows 2008.

Cause:
If you enable MS DTC tracing (TRACE_CM) on the cluster, you may see the messages similar to these:

12-21-2009 17:36:25:700 : [33a8. e34] 0x00000000 [ TRACE_CM] CRpcIOManagerClient::SetRpcSecurity (d:\vistasp1_gdr\com\complus\dtc\dtc\cm\src\iomgrclt.cpp@636): We are a cluster resource - Impersonate the virtual server name CLUS1Dtc
12-21-2009 17:36:25:707 : [33a8. e34] 0x8007138f [TRACE_CLUSTER] MtxCluGetVirtualServerToken (d:\rtm\com\complus\dtc\shared\mtxclu\src\clusterutil.cpp@5159): OpenClusterResource failed for virtual server name CLUS1Dtc
12-21-2009 17:36:25:707 : [33a8. e34] 0x8007138f [ TRACE_CM] CRpcIOManagerClient::SetRpcSecurity (d:\vistasp1_gdr\com\complus\dtc\dtc\cm\src\iomgrclt.cpp@641): Error from MtxCluGetVirtualServerToken
12-21-2009 17:36:25:707 : [33a8. e34] 0x8007138f [ TRACE_CM] CRpcIOManagerClient::SecureBuildContextWrapper (d:\vistasp1_gdr\com\complus\dtc\dtc\cm\src\iomgrclt.cpp@760): [Remote:APPS01 07adda78] Call to SetRpcSecurity failed
12-21-2009 17:36:25:707 : [33a8. e34] 0x8007138f [ TRACE_CM] CRpcIOManagerClient::Bind (d:\vistasp1_gdr\com\complus\dtc\dtc\cm\src\iomgrclt.cpp@324): [Remote:APPS01 07adda78] SecureBuildContextWrapper call failed. This is usually due to security/network configuration issues.
12-21-2009 17:36:25:709 : [33a8. e34] 0x80070005 [ TRACE_CM] CRpcIOManagerClient::BuildContextWrapper (d:\vistasp1_gdr\com\complus\dtc\dtc\cm\src\iomgrclt.cpp@847): [Remote:APPS01 07adda78] Exception raised in the rpc call C_BuildContextW.

OpenClusterResource fails because it could not find the network name resource called CLUS1Dtc in the cluster resource group that has the Microsoft Distributed Transaction Coordinator (MS DTC) resource. CLUS1Dtc here is actually the DNS name or the private name of the DTC network name resource. The error 0x8007138f means ERROR_RESOURCE_NOT_FOUND. This is a bug in MS DTC. You encounter this problem if the display name of the network name resource that the DTC resource depends on is different from the name resource’s DNS name (private name.) In this example the MS DTC resource is in a group that has two Name resources. The display name of the network name resource for DTC is MSDTC Network Name (Dev01) and its corresponding DNS name is CLUS1Dtc. In Failover Cluster Manager you can follow the steps listed below to find the display name of a network name resource that DTC depends on in any group such as MS DTC, SQL Server, MSMQ or any application that has a DTC resource and uses it for managing distributed transactions.

  • Open the Properties page for MS DTC resource (MSDTC-CLUS1Dtc)

image 

  • Find the network Name resource dependency as CLUS1Dtc

image 

  • Open the Properties page for the Name resource (CLUS1Dtc)

image

  •  Find the resource display name as MSDTC Network Name (Dev01)

image 

Run the “Cluster.exe res” command to display all cluster resources:

C:\Users\Administrator.233947DOM>Cluster res
Listing status for all available resources:

Resource Group Node Status
--------------------------------------------------------------------------------

MSDTC Network Name (Dev01) CLUS1Msmq 233947NODE1 Online
MSDTC-CLUS1Dtc CLUS1Msmq 233947NODE1 Online 

Resolution:
To use Incoming Caller Authentication Required for MS DTC on a Windows 2008 or R2 cluster, the display name of the network name resource for MS DTC must be same as its DNS name.

The following command will change the display name of the server network name resource from MSDTC Network Name (Dev01) to CLUS1Dtc:

Cluster.exe res “MSDTC Network Name (Dev01)” /ren:CLUS1Dtc

Windows 2008/Windows 2008 R2 can have multiple DTC instances on a cluster. Each DTC instance can be configured in any resource group. This change is not limited to one DTC resource group or one SQL Server resource group that has a MSDTC resource. The same change needs to be made in any SQL resource group that has a MSDTC resource or any other group that has a MSDTC resource which you use for managing distributed transactions.

A common configuration is you have a SQL cluster resource group and then you add a DTC resource to the same group. Both the SQL and DTC resources depend on the same network name. In the default SQL installation the resource display name is like SQL Network Name (XYZ). In this case you need to change SQL Network Name (XYZ) to XYZ if its DNS name is XYZ.

Cluster.exe res “SQL Network Name (XYZ)” /ren:XYZ

For more info on configuring MS DTC instances on a Windows 2008 cluster, please read this article: How to Configure Multiple Instances of Distributed Transaction Coordinator (DTC) on a Windows Server Failover Cluster 2008