MSDTC security model in windows 2003,sp1(same as windows xp,sp2)
//This content applies to windows 2003,sp1 and windows xp,sp2 machines
Microsoft introduced MSDTC security in windows xp and windows 2003,If you didn't configure MSDTC security settings properly,you may not able to run MSDTC transactions between two machines.
like i explained in my previous blog (https://blogs.msdn.com/madhuponduru/archive/2005/12/03/499781.aspx), MSDTC is RPC application,so MSDTC security model also some what looks like RPC security model.
If you want to run MSDTC transactions between two machines,make sure you enabled "Network dtc","Allow inbound" and "Allow outbound" settings on both machines(You can configure these settings from COM+ MMC)
MSDTC has three security options
No authentication:
==================
it won't authenticate incoming user,so if you have two machines in different domain or if these machines are in different work groups or one of this machine is windows 2000,you have to choose this option
Incoming caller security:
========================
This is default security model for cluster machines,If you are dealing with cluster machines,This is the maximum security you can set.
Mutual authentication:
======================
If you are running transactions between two windows 2003,sp1 machines or if you are running transactions between two windows xp,sp2 or between windows 2003,sp1 and windows xp,sp2 and If they are in same domain.You can use this option
we documented about all these options in this KB article
New functionality in the Distributed Transaction Coordinator service in Windows Server 2003 Service Pack 1 and in Windows XP Service Pack 2
https://support.microsoft.com/?kbid=899191
https://blogs.msdn.com/florinlazar/archive/2004/06/18/159127.aspx
RPC Security Essentials
https://msdn.microsoft.com/library/default.asp?url=/library/en-us/rpc/rpc/rpc_security_essentials.asp