Last time I wrote a post on MSDTC under Vista. Here I'd like to talk a general MSDTC issue, which is not Vista-specific.
Internally MSDTC uses NetBIOS names and passes them to RPC to talk to remote machines. This requires both server and client to be able to resolve each other's machine name.
Sometimes things can get messed up with DHCP or DNS caching. An easy workaround is to add entries with machine names and IP addresses in your hosts file. You probably need to do this on both the server and the client if neither could resolve the name.