Troubleshooting MSDTC issues with the DTCPing tool


Every day the Distributed Services support team in Microsoft helps customers in troubleshooting some of the most common Distributed Transaction errors which are a direct result of MSRPC (Microsoft Remote Procedure Call) communication failing in a network because of some Security\Firewall settings. On an application layer (like SQL), these are the common error messages that will be bubbled up.

  • Server: Msg 7391, Level 16, State 1, Line 2 The operation could not be performed because the OLE DB provider ‘SQLOLEDB’ was unable to begin a distributed transaction. OLE/DB provider returned message: New transaction cannot enlist in the specified transaction coordinator.
  • New transaction cannot enlist in the specified transaction coordinator (0x8004d00a)
  • The transaction has already been implicitly or explicitly committed or aborted (0x8004d00e)

If you encounter one of the above error messages while using Distributed Transactions from your application, feel free to use the DTCPING tool to find out where the problem lies. This blog explains how to use the DTCPing tool to narrow down the source of the problem and how to fix it.

Read the full post here…

Comments (33)

  1. Numa says:

    Great one. Groups together a lots of bits and pieces I’ve found after hitting successive walls.. and brings more :)

    Thanks a lot!

  2. Jay Mehta says:

    I’m getting the following error "WARNING:the CID values for both test machines are the same while this problem won’t stop DTCping test, MSDTC will fail for this"

    I’m pretty sure the CID values for both test machines are not the same.  What else could be causing this problem?

    Thanks for your help.  I’ve attached the log of my test.

    ++++++++++++hosts      ++++++++++++

    127.0.0.1       localhost

    172.16.6.2 reportserver.omnicare.com

    ++++++++++++++++++++++++++++++++++++++++++++++

        DTCping 1.9 Report for OCRSQL01  

    ++++++++++++++++++++++++++++++++++++++++++++++

    RPC server is ready

    06-30, 09:50:08.864–>RPC server:OCRSQL01 received following information:

    Network Name: ocrsql01

    Source  Port: 1781

    Partner LOG: 13792.log

    Partner CID: CD523431-12C1-49D7-AB1E-38C61097DF31

    ++++++++++++Start Reverse Bind Test+++++++++++++

    Received Bind call from

    Network Name: ocrsql01

    Source  Port: 1781

    Hosting Machine:OCRSQL01

    06-30, 09:50:09.083–>Trying to Reverse Bind to …

    Test Guid:CD523431-12C1-49D7-AB1E-38C61097DF31

    Name Resolution:

    –>172.16.6.2–>reportserver.omnicare.com

    Received reverse bind call from

    Network Name: ocrsql01

    Source  Port: 1781

    Hosting Machine:OCRSQL01

    Reverse Binding success: OCRSQL01–>

    ++++++++++++Reverse Bind Test ENDED++++++++++

    06-30, 09:50:09.083–>Called POKE from Partner:

    Network Name: ocrsql01

    Source  Port: 1781

    Hosting Machine:OCRSQL01

    ++++++++++++Validating Remote Computer Name++++++++++++

    06-30, 09:50:58.442–>Start DTC connection test

    Name Resolution:

    ocr2k3ovaprod01–>172.16.6.219–>OCR2K3OVAPROD01

    06-30, 09:50:58.458–>Start RPC test (OCRSQL01–>ocr2k3ovaprod01)

    WARNING:the CID values for both test machines are the same

    while this problem won’t stop DTCping test, MSDTC will fail for this

    06-30, 09:50:58.458–>RPC server:OCRSQL01 received following information:

    Network Name: ocrsql01

    Source  Port: 1781

    Partner LOG: OCRSQL016472.log

    Partner CID: 3E0DE9BB-93C1-491D-804C-841313AADD60

    RPC test is successful

    Partner’s CID:3E0DE9BB-93C1-491D-804C-841313AADD60

    ++++++++++++RPC test completed+++++++++++++++

    ++++++++++++Start DTC Binding Test +++++++++++++

    Trying Bind to OCRSQL01

    06-30, 09:50:58.458–>OCRSQL01 Initiating DTC Binding Test….

    Test Guid:3E0DE9BB-93C1-491D-804C-841313AADD60

    ++++++++++++Start Reverse Bind Test+++++++++++++

    Received Bind call from OCRSQL01

    Network Name: ocrsql01

    Source  Port: 1781

    Hosting Machine:OCRSQL01

    06-30, 09:50:58.458–>Trying to Reverse Bind to OCRSQL01…

    Test Guid:3E0DE9BB-93C1-491D-804C-841313AADD60

    Name Resolution:

    OCRSQL01–>172.16.6.2–>reportserver.omnicare.com

    Received reverse bind call from OCRSQL01

    Network Name: ocrsql01

    Source  Port: 1781

    Hosting Machine:OCRSQL01

    Reverse Binding success: OCRSQL01–>OCRSQL01

    ++++++++++++Reverse Bind Test ENDED++++++++++

    Binding success: OCRSQL01–>OCRSQL01

    ++++++++++++DTC Binding Test END+++++++++++++

  3. puneetg says:

    Hi Jay,

    Are you running DTCPING on two nodes of the same cluster? If yes, then you can ignore the warning that the tool is showing you.

    Also, I am not sure where exactly are you checking the value of the CID key. Please make sure that you are looking at the value of the CID inside HKEY_CLASSES_ROOTCID. Under this key, you may notice more than one GUID. Locate the GUID whose underlying Description key is MSDTC and this GUID should be different on both the machines.

    I hope this helps. If it doesnt, please paste the value of HKEY_CLASSES_ROOTCID so that we can check what might be causing this warning.

  4. Hung says:

    Hi,

    We have BizTalk 2009 installed into Windows 2008 server for some proof of concepts (MQSeries adapter). The Windows MQ server is based on Windows 2003.

    We’ve tried to DTCPing but faced with this message:

      The RPC server is unavailable (RPC test failed).

    We’ve tried to do everything like you write above, we’ve even turned the firewall of from both side without result.

    Telnet was ok (after install and start the service in Windows 2008).

    All RPC services was started.

    Can you help us to dig deeper?

    Thank,

    Hung

  5. davidqiu says:

    You can install Microsoft Network Monitor 3.3 on both servers and get simultaneous traces when you run DTCPing.

    http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=983b941d-06cb-4658-b7f6-3088333d062f

  6. davidqiu says:

    Note the computer name "ocr2k3ovaprod01" is 15 characters long.

    DTCPing 1.9 has some bugs when computer name length is MAX_COMPUTERNAME_LENGTH (15)

    I suggest you run DTCTester. If it works you can safely ignore the warning.

    http://support.microsoft.com/kb/293799

  7. szmatyi84 says:

    Hello,

    I’d like to ask, where can I find the "PART II" of this article, which deals with the situation, where DTCPing works fine, but the transactions do not work?

    Thank you in advance

  8. puneetg says:

    Hi szmatyi84,

    I never got a chance to work on writing the PART II of this article. As soon as I do so, I will update the blog to contain that link.

    For the time being, if DTCPing has worked in your environment and Distributed Transactions are not working, then you can check the MSDTC Security settings on both the machines. You can check them by going to Administrative Tools-> Component Services -> My Computers ->(Right Click)Properties -> MSDTC -> Security configuration.

    1) Make sure that "Network DTC Access" is checked in both the machines.

    2) Make sure that "Allow Inbound" and "Allow Outbound" is selected.

    3) If 1 and 2 don’t help, you can select "No Authentication Required" for testing purposes on both the machines and see if that makes a differnce.

    NOTE: On a cluster you cannot use "Mutual Authentication". The minimum authentical level supported on a cluster is Incoming Authentication Required.

    If none of the above steps help, you can turn on some Transaction tracing and see if that shows you some interesting error messages. You can refer to http://blogs.msdn.com/distributedservices/archive/2009/02/07/the-hidden-tool-msdtc-transaction-tracing.aspx for details around Transaction Tracing

  9. Dan says:

    Have you seen the case when there is no error description for the RPC test failure?  This is what I am getting:

    ++++++++++++++++++++++++++++++++++++++++++++++

        DTCping 1.9 Report for DEVDB01  

    ++++++++++++++++++++++++++++++++++++++++++++++

    RPC server is ready

    ++++++++++++Validating Remote Computer Name++++++++++++

    07-29, 16:39:50.683–>Start DTC connection test

    Name Resolution:

    DEVWEB01–>10.4.97.105–>devweb01.prod.firstlook.com

    07-29, 16:39:50.730–>Start RPC test (DEVDB01–>DEVWEB01)

    RPC test failed

  10. puneetg says:

    Hi Dan,

    I have seen that error typically if you are running this tool on Windows 2008/VISTA machine. Can you let us know the details of the operating systems where you were running the DTCPING tool when you got the "RPC Test Failed" error. Also are you getting the same error when you are running DTCPing from DEVWEB01?

  11. Ken says:

    Very useful, very helpful, appreciate the effort.  I have an active/passive cluster that started having DTC problems after the latest round of server patches – at least that’s my story at the moment.

  12. Tim says:

    I am recieving the same issue as Dan.

    ++++++++++++++++++++++++++++++++++++++++++++++

        DTCping 1.9 Report for APR-SFLD-DEV02  

    ++++++++++++++++++++++++++++++++++++++++++++++

    RPC server is ready

    02-22, 10:33:19.321–>RPC server:APR-SFLD-DEV02 received following information:

    Network Name: apr-sfld-dev02

    Source  Port: 2048

    Partner LOG: APR-FRMG-IT014860.log

    Partner CID: 00000000-0000-0000-0000-000000000000

    ++++++++++++Validating Remote Computer Name++++++++++++

    02-22, 10:33:22.305–>Start DTC connection test

    Name Resolution:

    apr-frmg-it01–>192.168.111.200–>apr-frmg-it01

    02-22, 10:33:22.321–>Start RPC test (APR-SFLD-DEV02–>apr-frmg-it01)

    RPC test failed

    Server: Windows 2003 R2

    Client: Windows Vista Ultimate

    Any idea where to turn to now?

  13. Ross says:

    Just wanted to put a note on this thread that the DTCPing tool that’s currently available to download will mis-report a failure on Server2008 (and presumably Vista/Win7).  Though the DTCPing tool would be aware of it’s peer, it would give an error that the servers had the same CID of 0000-000000-0000 (don’t have the correct amount of zeros).

    I contacted MS support and was able to get a copy of the v2 tool, and it worked just fine.

  14. cheng says:

    A customer would like to confirm if MSDTC relies on WINS?  Thanks!

  15. rposbo says:

    if you're just dev-ing on a Win 7 machine and are having "access denied" msdtc issues with the reverse bind to your own pc, you can always just disable local firewall or set up a new incoming firewall rule that is port: any, protocol:any.

  16. goro says:

    my DTCPing is always error when start first time, can you tell what could be wrong ?

    Platform:Windows 2003

    Windows 2003 environment is detected:

    Reading MSDTC settings from local registry:

    Security settings for DTC under Windows 2003

    NetworkDtcAccess            :true

    NetworkDtcAccessAdmin       :true

    NetworkDtcAccessClients     :true

    NetworkDtcAccessTransactions:true

    NetworkDtcAccessTip         :true

    XaTransactions              :false

    DomainControllerState       :true

    AccountName                 :NT AuthorityNetworkService

    TurnOffRpcSecurity          :true

    IP Configure Information

    Host Name . . . . . . . . . : acepuri04

    DNS Servers . . . . . . . . : 10.1.34.40

    Node Type . . . . . . . . . :

    NetBIOS Scope ID. . . . . . :

    IP Routing Enabled. . . . . : no

    WINS Proxy Enabled. . . . . : no

    NetBIOS Resolution Uses DNS : no

    Ethernet adapter {DFB2F226-8C82-4402-A92C-17264DB43108}:

    Description . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet – Teefer2 Miniport

    Physical Address. . . . . . : 00-1A-64-48-75-54

    DHCP Enabled. . . . . . . . : no

    IP Address. . . . . . . . . : 10.1.34.40

    Subnet Mask . . . . . . . . : 255.255.255.0

    Default Gateway . . . . . . : 10.1.34.1

    DHCP Server . . . . . . . . : 255.255.255.255

    Primary WINS Server . . . . : 0.0.0.0

    Secondary WINS Server . . . : 0.0.0.0

    Lease Obtained. . . . . . . : Thu Jan 01 00:00:00 1970

    Lease Expires . . . . . . . : Thu Jan 01 00:00:00 1970

    ++++++++++++lmhosts.sam++++++++++++

    ++++++++++++hosts      ++++++++++++

    127.0.0.1       localhost

    10.1.32.41 BI

    ++++++++++++++++++++++++++++++++++++++++++++++

        DTCping 1.9 Report for ACEPURI04  

    ++++++++++++++++++++++++++++++++++++++++++++++

    Firewall Port Settings:

    Port:5001-5500

    Test open port: 5018 –> success

    Error(0x6B9) at rpcUtil.cpp @54

    –>1721(Not enough resources are available to complete this operation.)

  17. puneetg says:

    @goro,

    You are getting the Error Message 6 as mentioned above in the blog post which means RPC is running out of available ports on the machine. I can see that you have opened up 5001-5100 DCOM ports but looks like something is consuming all these ports and thats why DTCPING is displaying an error.

    You can run NETSTAT -ano to find out the port usage and that way you will be able to determine which program is consuming too many DCOM ports.

    Hope this helps.

    @cheng

    MSDTC relies on NETBIOS name resolution. Under the hood that can happen either due to DNS or WINS. MSDTC doesnt have a direct dependency on WINS but if WINS is enabled, MSDTC may use it for NETBIOS name resolution

  18. foman says:

    Hi puneetg.

    I got this error. Could you tell me what this error mean ?

    07-03, 18:21:45.605–>Error(0x424) at clutil.cpp @256

    07-03, 18:21:45.606–>–>OpenCluster

    07-03, 18:21:45.607–>–>1060(The specified service does not exist as an installed service.)

  19. puneetg says:

    @forman,

    Hello and Sorry for a late reply on this. When you run DTCPING it makes a call to the OpenCluster API (msdn.microsoft.com/…/aa371774(VS.85).aspx) to detect if the machine is on a cluster or not. From the logs it looks like that call is failing with an error. Typicall if the machine is not on a cluster, we get a specific error code but you are getting an error which says "The specified service does not exist as an installed service".  Is this machine on a Cluster? Are you running the DTCPING tool as a Cluster Administrator. We may need more details about the machine (Operating System, whether it is a cluster) and the user under which you are running the tool. Also is the Cluster Service running on the machine?

  20. capricorn says:

    Hi!

    I am trying hard to solve this but it dont work for me. I am using window 2008 server 32 bit. I am getting error

    DTCping log file: C:Documents and SettingsusernameDesktopDTC_PINGTURTLES8618

    RPC server is ready Please Start Partner DTCping before pinging

    ++++++++++++Validating Remote Computer Name++++++++++++

    Please refer to following log file for details: C:Documents and SettingsusernameDesktopDTC_PINGTURTLES861896.log

    Invoking RPC method on turtle86

    Problem:fail to invoke remote RPC method Error(0x6BA) at dtcping.cpp @303

    –>RPC pinging exception –>1722(The RPC server is unavailable.)

    RPC test failed

    —-

    There is not firewall between two computers. I can telnet to 135 port. DTC traffic is allowed. I have remove antivirus but nothing is working for me.

    One funny thing i faced is that if run the DTCping and use the same computer name not the other i will get the message:

    RPC server is ready

    Does not make sense to test RPC on the same machine

    After that if i use dtcping and write other computer name it works fine. Really strange.

    Can you help in that ?

    Regards,

  21. Illya says:

    Cool. Steps that you described solved my problems with MSDTC. Thank you very much!

    My environment are two Win Server 2008 R2, log on as a domain admin, MSDTC were already configured. DTCping works fine.

    But how many ports should be opened in reasonable manner? By the way, on a screenshot you typed range 5000-51000 (is it too long?).

  22. puneetg says:

    @Illya,

    Glad to know that the steps helped in solving your problem.

    The screenshot shows 5000-5100 ports (100 ports) to be opened up and on a non-clustered machine 100 range of ports has so far proved to be a reasonable range to configure, however there is no solid data backing this theory.

    Thanks,

    Puneet

  23. Hi,

    My tests are success but my log indicate:

    ++++++++++++hosts      ++++++++++++

    12-14, 12:12:09.283–>Error(0x424) at clutil.cpp @256

    12-14, 12:12:09.283–>–>OpenCluster

    12-14, 12:12:09.283–>–>1060(Le service spécifié n’existe pas en tant que service installé.)

    ++++++++++++++++++++++++++++++++++++++++++++++

        DTCping 1.9 Report for MY_COMPUTER

    ++++++++++++++++++++++++++++++++++++++++++++++

    Firewall Port Settings:

    Port:5000-5100

    RPC server is ready

    ++++++++++++Validating Remote Computer Name++++++++++++

    12-14, 12:12:15.211–>Start DTC connection test

    Name Resolution:

    srv-test–>192.168.1.250–>test

    12-14, 12:12:17.395–>Start RPC test (MY_COMPUTER–>srv-test)

    RPC test is successful

    Partner's CID:F7F44B1E-9DA7-42F1-86C1-03ABA37D1E83

    ++++++++++++RPC test completed+++++++++++++++

    My computer is under Windows Seven 64 bits and the server Windows Server 2008

    I dont find any solution.

    Can you help me plz

    thx for advance

  24. sqlman says:

    Prod sql server that is hosted had similar/same issue – could not run distributed transactions – ended up having to open ports in firewall then specify port range as stated in blog. After reboot, DTCPing continue to fail but distributed transaction was successful. Thank you for the info!

  25. Ed says:

    Great article, helped us. Thank you.

  26. Jason says:

    Please help to reply the following question regarding troubleshooting MSDTC with DTCPing:

    social.msdn.microsoft.com/…/dtcping-tool-1717the-interface-is-unknown-error

    Thanks in Advance!

  27. gemison says:

    Hi,

    I received the following error during DTCPing.

    Problem:fail to invoke remote RPC method

    Error(0x6B5) at dtcping.cpp @303

    –>RPC pinging exception

    –>1717(The interface is unknown.)

    RPC test failed

    Any idea on the reason and ways to resolve it? Thanks in advance.

    Note: More information on the error at social.msdn.microsoft.com/…/dtcping-tool-1717the-interface-is-unknown-error

  28. Malik Adeel Imtiaz says:

    When we ping DTC it crached our application below is the Event Report

    Faulting application name: Dtcping.exe, version: 1.9.0.1, time stamp: 0x42e97e2c

    Faulting module name: CLUSAPI.dll, version: 6.1.7601.17514, time stamp: 0x4ce7b797

    Exception code: 0xc0000005

    Fault offset: 0x00005863

    Faulting process id: 0x353c

    Faulting application start time: 0x01d0105ad45185eb

    Faulting application path: E:SolDataPreDTC PingDtcping.exe

    Faulting module path: C:Windowssystem32CLUSAPI.dll

    Report Id: 14e72687-7c4e-11e4-a115-005056850cf1

    Your kind help is needed to resolve this issue.

  29. puneetg says:

    Hello Malik ,

    What OS are you trying DTCPING on ? There is a newer version of DTCPING that might work for you. If you can send us details of OS and complete EventViewer entry and DTCPING log file, we can share the tool with you. Please email me at puneetg@microsoft.com with the details

  30. Phil Dale says:

    Fantastic article – has been a life saver for me.  Thanks PuneetG for your help.

  31. JDMils says:

    When creating a DTC resource in SQL 2008, I would get this error:

    "An error occurred while creating the Distributed Transaction Coordinator (DTC).

    An error occurred while creating a client access point for '60DTF5SRSQ01'.

    Failed to save client access point changes.

    Failed to update the network name.

    An error occurred renaming resource '60DTF5SRSQ01 (2)' to '60DTF5SRSQ01'.

    Cannot create a file when that file already exists."

    We found that the problem was that the resource volume we chose to home the DTC resource on had the same name as the DTC resource. Once we renamed the disk volume, we were able to create the DTC object in SQL!

  32. Kevin says:

    I was getting the gethostbyname failure, but was still able to ping the server. It turned out that the server name was too long for DTCPing to handle. I had to add an entry to my hosts file that gave the IP address of the server a different (shorter) name.

    Unfortunately, DTCPing had a successful test but DTC still did not work (with the real name or the shorter dummy name).

  33. VijiKarthik says:

    Hi,

    when I try to open Sevices.msc of one Windows 2012 server from another Windows 2012 server using IP or Hostname without using FQDN. I have received Error code:1727: RPC failed did not execute .  

    Kindly help me to solve this issue.