Troubleshooting TS Gateway connectivity on Windows 2008, IIS 7.0

Article
08/30/2008

Here is something which is not my domain but had to learn the hard way. I recently encountered an issue while enabling Terminal Services Gateway (TSG) on Windows 2008 server. TSG is coupled with IIS 7.0 hosted on Windows 2k8 server and that's how I came into picture. TSG in simple terms is a feature using which one can connect remotely to an internal network over secure HTTPS port 443 from the Internet. Earlier Remote Desktop Protocol (RDP) connections used TCP port 3389. In many corporate environment this port may be blocked by the firewall. However now with TSG connecting on port 443 (common SSL port for http traffic) user should not get into the common issues of port being blocked. In my case we had the TSG installed as one of the roles on the server. The setup was fine.
The only concern was that we already were using the Default Web site for some application. It can also happen otherwise, you have the TSG setup on an IIS 7 web site and if you go ahead and install let's say Exchange on top of it under the same site it may break the TSG functionality.

When you install TSG, it creates two virtual directories called Rpc and RpcWithCert under the web site as shown below.

Under the hood it appears a call is made for

https://<server-name>:443/rpc/rpcproxy.dll?localhost:3388 when you try to connect through TSG. So yes IIS is very much involved here.

Now what can you do to fix this, perhaps you can install your web application (say Exchange) on some other web site and a different SSL port like 444 and have TSG site listening on port 443. Or else just the opposite.

In my case we went with the 2nd option since we didn't want Exchange to be reinstalled again.
But even if you use either of the above options it may not go that smoothly as it looks to be.

You may see the error as shown below when you try to use terminal service through TSG.

Click on OK...

If you are seeing something like this, as a workaround create a new web site and copy the settings for the Virtual directories /Rpc and /RpcwithCert from the previous site to the the new web site. You can do this easily by copying the configuration in the applicationHost.config file.

Here are the steps:

1. Copy the following configuration (in the ApplicationHost.config file from C:\<Windows>\System32\inetsrv\config) from the previous site to the new site to add the virtual directories for your new web site.

 <site name="<new-web-site>" id=...>
...
<application path="/Rpc" applicationPool="SomeAppPool">
       <virtualDirectory path="/" physicalPath="C:\Windows\System32\RpcProxy" />
</application>
<application path="/RpcWithCert" applicationPool="SomeAppPool">
        <virtualDirectory path="/" physicalPath="C:\Windows\System32\RpcProxy" />
</application> 
...
</site>

So this will create two virtual directories in your new web site called Rpc and RpcWithCert.

Add an SSL binding for the new Web site on port 443 as well. Ensure no other site is listening on port 443.

2. Copy the following for the previous web site in the ApplicationHost.config file to the new web site.

This is the section contained in the Location tag for the Virtual directories /Rpc and /RpcWithCert. You need to copy this section from the location tag for the <previous-web-site> and add it to the location tag for the <new-web-site>.

 <location path="<previous-web-site>/Rpc">
        <system.webServer>
            <directoryBrowse enabled="false" showFlags="Date, Time, Size, Extension" />
            <handlers accessPolicy="Execute">
                <add name="RPCPROXY" path="*" verb="*" modules="IsapiModule" scriptProcessor="C:\Windows\system32\RpcProxy\RpcProxy.dll" requireAccess="Execute" />
            </handlers>
            <serverRuntime uploadReadAheadSize="0" />
            <defaultDocument enabled="true" />
            <modules>
                <add name="PasswordExpiryModule" />
            </modules>
            <security>
                <requestFiltering>
                    <requestLimits maxAllowedContentLength="2147483648" />
                </requestFiltering>
                <authentication>
                    <anonymousAuthentication enabled="false" />
                    <basicAuthentication enabled="false" />
                    <windowsAuthentication enabled="true" useKernelMode="false" />
                </authentication>
                <access sslFlags="Ssl, Ssl128" />
            </security>
            <httpErrors>
                <remove statusCode="401" />
                <error statusCode="401" path="C:\Windows\system32\RpcProxy\Error401.txt" responseMode="File" />
            </httpErrors>
        </system.webServer>
    </location>

 <location path="<previous-web-site>/RpcWithCert">
        <system.webServer>
            <directoryBrowse enabled="false" showFlags="Date, Time, Size, Extension" />
            <handlers accessPolicy="Execute">
                <add name="RPCPROXY" path="*" verb="*" modules="IsapiModule" scriptProcessor="C:\Windows\system32\RpcProxy\RpcProxy.dll" requireAccess="Execute" />
            </handlers>
            <defaultDocument enabled="true" />
            <security>
                <authentication>
                    <anonymousAuthentication enabled="false" />
                    <basicAuthentication enabled="false" />
                    <clientCertificateMappingAuthentication enabled="true" />
                    <digestAuthentication enabled="false" />
                    <windowsAuthentication enabled="false" useKernelMode="false" />
                    <iisClientCertificateMappingAuthentication enabled="true" />
                </authentication>
                <access sslFlags="Ssl, SslNegotiateCert, SslRequireCert, Ssl128" />
                <requestFiltering>
                    <requestLimits maxAllowedContentLength="2147483648" />
                </requestFiltering>
            </security>
            <serverRuntime uploadReadAheadSize="0" />
            <modules>
                <add name="PasswordExpiryModule" />
            </modules>
            <httpErrors>
                <remove statusCode="401" />
                <error statusCode="401" path="C:\Windows\system32\RpcProxy\Error401.txt" responseMode="File" />
            </httpErrors>
        </system.webServer>
    </location>

3. Ensure that we replace <previous-web-site> with the <new-web-site> in the following tags above:

<location path=" <previous-web-site> /RpcWithCert"> -----> <location path=" <new-web-site> /RpcWithCert">

4. Run iisreset from the cmd prompt. Or it may also work with just restarting W3SVC service (net stop w3svc, net start w3svc).

Go ahead and test RDP over TSG from the client. If it still doesn't work you may have to try the 5th step as below.

5. Add the following registry entry. Run this from the cmd prompt:
> reg add HKLM\Software\Microsoft\RPC\RpcProxy /v Website /t REG_SZ /d <new-web-site>

One last thing, ensure that the certificate issued to the TS server gateway is trusted on the client from where we are doing a terminal login.

Happy troubleshooting!

Cheers Martini Glass

Troubleshooting TS Gateway connectivity on Windows 2008, IIS 7.0

Additional resources