Check out Durga's blog post on setting up an agent in workgroup mode.
One thing I noticed is that the install guide is incorrect, as it says that the "Simple File Sharing" is turned off when installing on XP in workgroup mode. You have to clear that manually. I have corrected that in my post.
I also put together a visio diagram that shows which ports are used during setup and when the agent and controller run tests.
And here are the connections used during agent setup: