The XPS Viewer is a component of the .NET Framework 3.0, which in turn is a feature of the operating system. So you need to make sure you have it installed. Start up Server Manager, select Features and add the feature:
[Update: because I realised there was a problem just after I published the post, here's the "science bit"...]
Hopefully this will just work for you. On my Windows Server 2003 x64 machine it did not. Double clicking an XPS document brought up the "which program do you want to open this document with" dialog and XPS Viewer was not listed
So hit the "Browse" button and point it to "C:WindowsSysWOW64XPSViewerXPSViewer.exe".
Thereafter, the same dialog should offer you just XPS Viewer and you can then check the checkbox for "Always use this program"