Its all about security: BizTalk Server 2004 and Windows XP SP2

Just getting ahead of the game. When you install SP2 RC or later you will need to tweak a registry key for BizTalk Server to continue to work. The changes are because the SSO service uses RPC and the RPC service has been significantly locked down in XP SP2.  This stuff will be documented in the XP SP2 readme but never-the-less the settings to change is: 

A REG_DWORD value, EnableAuthEpResolution, under the \\HKLM\SOFTWARE\Policies\Microsoft\Windows NT\RPC key. Setting the value to 1 *and* a swift miracle cure (aka reboot) fixes the problem. The new registry setting is explained on the Microsoft web site as part of the Network Protection Technologies new in XPSP2.

Probably the easiest way to enter the missing value is to paste the few lines below into a file with a .reg extension and then double clicking the file into the registry:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\RPC]
"EnableAuthEpResolution"=dword:00000001

(Thanks to Christoph for pointing out my original post was a little more vague - I stole his more detailed explanation).

And no before you ask this doesn't reduce your security it enforces NTLM auth.

Here are the details: An RPC interface that is remotely and anonymously accessible and is registered by default on Windows XP presents a significant attack surface. RPC itself must register such an interface to provide endpoint resolution for calls using dynamic endpoints.

With the addition of the RestrictRemoteClients registry key, by default, the RPC Endpoint Mapper interface is not accessible anonymously. This is a significant security improvement, but it changes the task of resolving an endpoint. Currently, an RPC client that attempts to make a call using a dynamic endpoint will first query the RPC Endpoint Mapper on the server to determine what endpoint it should connect to. This query is performed anonymously, even if the RPC client call itself is performed using RPC security.

Anonymous calls to the RPC Endpoint Mapper interface will fail by default on Windows XP Service Pack 2 because of the default value for the new RestrictRemoteClients key. This makes it necessary to modify the RPC client runtime to perform an authenticated query to the Endpoint Mapper. If the EnableAuthEpResolution key is set on the client, the RPC client runtime will use NTLM to authenticate to the Endpoint Mapper. This authenticated query will only take place if the actual RPC client call uses authentication.