How to troubleshoot MSDTC communication failure (I)
MSDTC is used in distributed environment, because many factors were involved, when MSDTC communication got failure, such as:
New transaction cannot enlist in the specified transaction coordinator.
Enlist of MSDTC transaction failed: Result Code = 0x8004d00e
Error 8004d00a. Distributed Transaction error
error 0x8004D025 (XACT_E_PARTNER_NETWORK_TX_DISABLED)
0x8004D01B (The Transaction Manager is not available.)
We generally have three typical things to verify, including:
1. MSDTC installation or configuration incorrectly on both sides .
2. Firewall settings which should not block two-directions of MSDTC/RPC communication
3. MSDTC fully depends on RPC communication, whether RPC dynamic port can be opened without issues
There are some documents discussed above scenarios from different views, but some of them are out of date now or some words are not properly for production environments. Here we would like to summarize the typical troubleshooting check list and case studies into three sections, the target is current Win2003/Win2008 MSDTC service, hopefully this article will help you efficiently start solving similar issues when MSDTC communication is down.
Section A: Check if MSDTC service installation for configuration is ready on two side machines
1. Open Services.msc, ensure the "Distributed Transaction Coordinator" is in the running status.
2. Ensure the name resolution of machine names between each machine works. If the backend server is cluster, the cluster name (including SQL, MSDTC resource name) can be resolved by MSDTC client.
For Windows 2008 cluster, please follow this article to avoid name configuration issue:
2172085 MSDTC communication is not working on a Windows 2008 and R2 Cluster using Incoming Caller Authentication
https://support.microsoft.com/default.aspx?scid=kb;en-US;2172085
3. Ensure the "Network DTC Access" is selected for MSDTC.
For Win2003, exactly follow:
817064 How to enable network DTC access in Windows Server 2003
https://support.microsoft.com/default.aspx?scid=kb;EN-US;817064
For Win2008 server, In "Server Manager", click "Add Role" to add Application Server role. The screenshot is like below, the "Incoming Remote Transactions" and "Outgoing Remote Transactions" must be installed:
Then open DCOMCNFG.msc in Windows 2008, open Computers -> My Computer -> Distributed Transaction Coordinator -> Local DTC, check it's security properties, should have similar settings as those required in Win2003:
4. If you are running on SQL cluster, MSDTC cluster resource must be created and brought online.
For Win2003:
301600 How to configure Microsoft Distributed Transaction Coordinator on a Windows Server 2003 cluster
https://support.microsoft.com/default.aspx?scid=kb;EN-US;301600
For Win2008:
How to configure shared MSDTC resource for Windows 2008 Failover Cluster
How to configure individual MSDTC resource with SQL cluster application in SQL Server 2008 Failover Cluster
5. If you setup dynamic Port range for RPC communication, please pay attention to:
a. The dynamic port range change will not take effect before machine reboots.
b. For a server, the dynamic port range should be at least above 100. 5000~5200 is suitable for many environments.
c. We can set the port range in DCOMCNFG.msc, click Computers -> My Computer , open its properties, click the "Default Protocols" tab, select the "Connection-oriented TCP/IP" item, click Properties. It should be similar to this:
d. If the DTC server is a cluster, after changed the UI for port range, please make sure the Registry key does match the values, otherwise, please manually change them:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Rpc\Internet
6. If the client process doesn't have enough permission to query the MSDTC service status, the "Transaction Manager is not available" error will also happen. After confirming the MSDTC service status is running, if the error still happens, we can check the service ACL and grant correct permissions. I resolved several problems with below steps:
sc sdshow msdtc
If the Authenticated Users group doesn't have query permission on the MSDTC service object, this means most users have no permission to get the MSDTC service status, for example:
(A;;CR;;;AU)
We need to run this command to grant enough permission for the Authenticated user, and then restart DLLHOST.exe (before do this, please backup the output information of "sc sdshow msdtc"):
sc sdset msdtc D:(A;;CCLCSWRPLOCRRC;;;S-1-2-0)(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;IU)(A;;CCLCSWLOCRRC;;;SU)(A;;CCLCSWLOCRRC;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)(A;;CCLCSWRPLORC;;;NS)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)
Note: the key part is: (A;;CCLCSWLOCRRC;;;AU), Other strings is taken from the "sc sdshow msdtc" result
For more information about the Description of ACL, please refer to:
914392 Best practices and guidance for writers of service discretionary access control lists
https://support.microsoft.com/default.aspx?scid=kb;EN-US;914392
Note: this permission change is for Service Object ACL, will not affect NTFS file properties.
Thanks!
Freist from Microsoft APGC DSI Team