Some time ago a customer of mine was developing a native application using Fax API, which worked fine on Windows Server 2003. On Windows Server 2008 though, when the application was being run as a service, they could see that FaxSendDocument API was not returning. But everything worked fine on that OS when a user run the application manually.
The RTF document they were sending was very simple and it didn't contain links, so they were not in this situation:
When you send a document from an application, links in the document may cause a dialog to appear, requesting information. If you do not handle the information request within several minutes, FaxSendDocument will fail and return an error.
The fax service only understands TIFF files naturally. So when you fax any other file format, it must first convert it to TIFF. Since the Fax Service doesn't know (for example) RTF, it leverages installed applications to do the rendering for it. The way it does this is by calling ShellExecute API on the document with the "printto" verb and the fax printer as the destination. The output of the fax printer is a TIFF file that the fax service can then fax.
About Sending a Fax
Printing from an application to the fax printer
In this process, the fax is sent from within the application. The printing application directly renders the image to be printed (faxed) and no further conversion is required.
The fax printer driver saves the rendered image as a temporary Tagged Image File Format (TIFF) file and sends it to the fax service for queuing and transmission. The fax send wizard is displayed so the user can provide specific fax transmission data.
In my customer's case, they were using MS Word 2010 to render the TIFF file, and Word is not supported in a service scenario: Considerations for server-side Automation of Office.
The only supported solutions I found that may work (note I didn't try them myself) are:
1) Change the application associated to RTF files.
2) Convert RTF files to TIFF in some other way (e.g. using third-party libraries or tools instead of Word), and then fax the TIFF file directly.
In the end, enabling the "Allow service to interact with desktop" flag on the service made FaxSendDocument API to stop hanging.
I hope this helps.
Alex (Alejandro Campos Magencio)