Happy New Year for 2016. I thought I'd start off the year with a blog on a topic I have been meaning to get to for 18 months now J, and that is on monitoring your Skype/Lync Edge servers with Operations Manager. Too often I go out and review Skype/Lync environments that include an Edge component and find that external role is not well monitored. In my opinion, in a Skype and Lync environment monitoring your external facing components is critical and in some ways more important than monitoring your internal servers. This is of course down to the fact that they are external facing and more likely to be attacked/compromised. In the same vein, Skype Edge Servers should also be well managed, patched, configured for Antivirus and updated with software updates as they are released. Please make sure you monitor your Edge Servers and your Reverse Proxies.
*End of rant* J
Now part of the reason I think Operations Manager doesn't get used enough to monitor the Edge Server role is it can be quite tricky monitoring a server that is 1) in the DMZ and 2) in a workgroup. So this blog is all about setting up monitoring of an Edge Server with Operations Manager 2012 R2.
To use a Gateway Server or not?
The first decision you need to make when embarking off on this monitoring adventure is to decide whether or not you are going to allow your Edge Servers to talk directly to the Operations Manager Management Servers directly, or whether or not you need to funnel all communication from the DMZ through an Operations Manager Gateway server. Gateway servers are useful for taking connections from lots of servers in a DMZ and funnelling back the communication to the internal Operations Manager Server, limiting of course the connections from untrusted (in the DMZ) to trusted in the internal network. This is likely to be the best option for you if you have a lot of servers in the DMZ that need to be monitored.
If you want to find out more about Gateway Servers in Operations Manager go and have a look at this article on Technet https://technet.microsoft.com/en-us/library/hh212823.aspx.
If you don't have a lot of servers in the DMZ, and maybe only have a small number then you might elect to allow direct connections from your Edge Servers to the Operations Manager Management Servers. For the purposes of this blog I am going to setup connections directly to the Management Server. I might do another blog later on using a Gateway Server to go through the process of deploying this role in Operations Manager. I am mainly focused on direct connections to the Operations Manager server as I tend mostly to work with customers who have only a small DMZ Windows fleet, and I just want to make sure Edge is being monitored.
1# - Creating a new certificate template in your Certificate Authority
An Edge Server sitting in a Workgroup out in the DMZ is unable to authenticate with an Operations Manager Management Server using the traditional authentication methods available to domain joined machines, and as such needs to authenticate to the OpsMgr server using certificate authentication. As both the OpsMgr server and Edge Server need certificates to mutually authenticate, the certificates they use must be configured for both "Client Authentication" and "Server Authentication" Application Policies. There is a good blog here that explains in simple terms the "Client Authentication" and "Server Authentication" http://blogs.msdn.com/b/kaushal/archive/2012/02/18/client-certificates-v-s-server-certificates.aspx. In short, stealing some of Kaushal's text from the blog, "Server Certificates are used to identify a server" and "serve the purpose of encrypting and decrypting the content". For client authentication certificates he says "Client certificates as the name indicates are used to identify a client or a user. They are meant for authenticating the client to the server".
So in the context of Edge and OpsMgr communication the certificates will be used for both authentication and sending and receiving of information, and hence will be used for both client and server purposes and hence we need a certificate that is setup to do both activities. Typically, a certificate is enabled for client authentication or server authentication not both. To setup a new certificate template (assuming you don't already have one setup for both Client and Server Auth) perform the following steps:
- Open up the Certificate Authority MMC console on your Certificate Authority;
- Go to the Certificate Templates node on the console and right click and select Manage;
- In the Certificate Templates Console, select the Web Server template and right click and select Duplicate Template;
- In the Properties of a New Template dialog, on the General tab change the Template display name and Template name to something more appropriate like Operations Manager template. NOTE: The Template Name field cannot include spaces, so I have named mine simply OpsMgrTemplate;
- On the Request Handling tab, select the Allow private key to be exported option (this is important as we need to export the cert);
- On the Extensions tab, click on the Application Policies section and select Edit;
- In the Edit Application Policies Extension dialog, click the Add button;
- Add Client Authentication and click OK;
- On the Security tab and Authenticated Users with Read and Enroll permissions (Allow). Close out of the Certificate Templates console;
- NOTE: Be careful with compatibility creating new templates, in the Compatibility Tab setup the right Operating Systems that will be used by the CA and the OpsMgr/Edge Servers;
- Now to add the new template to the Certificate Templates node, right click the Certificate Templates node and select New, Certificate Template to Issue;
- Select the certificate you setup in the previous steps and click OK.
You now have a certificate template ready to use for our OpsMgr mutual authentication (and sending/encrypting data).
2# - Create OpsMgr Management Server certificate using new template
The next step is to create a new certificate for your Operations Manager server that will communicate with your Edge Server. In my case the OpsMgr server's FQDN is opsmgr.contoso.org. This FQDN I will configure when installing the OpsMgr agent on the Edge Server. To create the certificate we will use certreq tool. So to create a new certificate the first step is to open Notepad on your OpsMgr server and create a OpsMgr.inf file. The .inf file will include the following text (I've saved to c:\certs\opsmgr.inf):
;Needs to be opsmgr server FQDN in friendly name and CN=
You use the certreq utility to create a request file using the .inf file. This is performed with the following command in a Command Prompt (run on the Opsmgr server itself):
certreq -new -machine c:\certs\opsmgr.inf c:\certs\opsmgr.req
Next we need to use the request file (.req) to get a certificate issued by your Certificate Authority (with the new template).
certreq -submit c:\certs\opsmgr.req c:\certs\opsmgr.cer
Now that the .cer has been created, we need to import the certificate into the Computer certificate store to finalise the process. This can again be done with the certreq utility.
certreq -accept -machine c:\certs\opsmgr.cer
The Operations Manager Management Server certificate is now ready to rock and roll, well almost J.
3# - Use MOMCertImport.exe to assign the certificate for use in monitoring
While we have at this point a certificate in the OpsMgr Computer store, we haven't assigned it for use in OpsMgr. This is done using the MOMCertImport.exe. You can find the MOMCertImport.exe on the ISO for Operations Manager in the SupportTools\AMD64 folder.
To register the cert with OpsMgr we perform the following command:
MOMCertImport.exe /subjectname:<FQDN of your Mgmt Server>
For the certificate to be used, we will need to restart the OpsMgr services, but we'll do that once we have the Edge Server's certificate as well. (which also needs to be imported using MOMCertImport.exe).
4# - Create Certificate for Edge Server
The next step is to create a certificate for the Edge Server that will be used in dealings with OpsMgr. We perform these steps on the Edge Server itself. In my case the Edge server's FQDN is edge03.contoso.org. This FQDN that will be used by the OpsMgr server when performing health checks etc.. To create the certificate we will again use the certreq tool (with the OpsMgr Template). So to create a new certificate the first step is to open Notepad on your Edge server and create a edge03.inf file. The .inf file will include the following text (I've saved to c:\certs\edge03.inf):
;Needs to be Edge server FQDN in friendly name and CN=
You use the certreq utility to create a request file using the .inf file. This is performed with the following command in a Command Prompt (IMPORTANT: Run on the Edge server itself):
certreq -new -machine c:\certs\edge03.inf c:\certs\edge03.req
Next, as the Edge Server is in the DMZ, we will use the .req file to request a certificate via the Certificate Authority's from a different machine. We will use the certreq utility again, but from a domain joined machine with access to the .req file. In this example I will copy the .req file to my Domain Controller running the CA service.
Run the following command on the CA:
certreq -submit c:\certs\edge03.req c:\certs\edge03.cer
The resulting .cer created needs to be copied back to the Edge Server so that it can be imported into the Computer Certificate store. Once copied to the Edge Server (c:\certs) the following command can be run to import the .cer into the Computer Certificate store (NOTE: Run on the Edge Server).
certreq -accept -machine c:\certs\edge03.cer
I now have a working certificate that can be used for the OpsMgr integration.
5# - Installing the OpsMgr agent on the Edge Server
The next step in the process is to install the OpsMgr agent on the Edge Server. You will configure the agent with the appropriate settings regarding the Management Group and Management Server the Edge Server will talk to. Importantly here the Management Server configured in the agent install will need to be the machine that we have setup certificates for Workgroup communication previously. Also, make sure that you patch the OpsMgr agent to the latest and greatest. At the time of writing of this blog the latest agent update was Update Rollup 8 for Operations Manager 2012 R2.
Also, when installing the agent make sure the Edge Server is able to resolve the IP address of the Operations Manager management server (as often Edge Servers are configured with an external DNS server and so may not be able to resolve all internal addresses).
I have made an assumption here too at this point that the certificate authority used to generate certificates is trusted by the Edge Server. If not, make sure the Certificate Authorities Trusted Root Certificate is added to the Trusted Root Certificate repository on the Edge Server.
6# - TCP 5723 from Edge OpsMgr Agent to Management Server
From a networking point of view, you will need to make sure that TCP 5723 is enabled from the Edge Server to the OpsMgr Management Server. That will of course mean that a DMZ box needs to talk back inside to an internal machine. This will of course require some review, but is fairly standard stuff and so shouldn't be too much of a drama in most environments. In my opinion, not monitoring a DMZ box is much more of a risk then adding an inbound TCP port from a DMZ box to your monitoring server. If you want to know more about OpsMgr ports have a look at this Technet article https://technet.microsoft.com/en-us/library/jj656649.aspx.
7# - Registering the Cert using the MOMCertImport utility
For the Edge Server's MOM Agent to use the certificate previously created, we need to again run the MOMCertImport utility. The utility is run quite simply on the Edge Server itself referencing the Subject Name of the certificate. The Subject Name of the certificate should be the FQDN of the Edge Server itself (the internal name that the OpsMgr Server will use to find the Edge machine, not any external facing name like sip.contoso.com). Copy the MOMCertImport.exe from the OpsMgr .iso to the Edge Server in order to run the utility. Then run the following command:
MOMCertImport.exe /subjectname "FQDN-Of-Your-Edge-Server"
Now at this point we are getting pretty close to having information being collected by the agent. Below shows my agent appearing in the Administration section of the console.
8# - Make sure Agent configured to allow Agent Proxy
For my SfB lab, my Edge03 server is part of a multi-server Edge Pool and hence will report back on information about the Pool itself as well as information on the edge03.contoso.org. The fact that it tracks info about the Edge Pool means that "Agent Proxy" needs to be enabled in OpsMgr. To do this, go to the Operations Manager console and Administration, Device Management, Agent Managed, and then select the Edge server you are monitoring. Right click the computer and select Properties. In the properties dialog navigate to the Security tab and enabled Agent Proxy.
9# - Grant Network Service access to RTC Local Administrators and RTC Component Local Group *Temporarily*
I had issues with the SfB discovery script on my Edge Server and it came down to permissions that the Network Service account has to the local SQL Express databases on the Edge Server. The extra access is typically enabled on a Front End Server, but removed on an Edge Server (no doubt due to more restrictive security requirements). The result is you will see these alerts in your Operations Manager Event Log section.
The SfB discovery script is basically the one that works out what the box is used for and are all the bits and pieces enabled and installed. Once run, I believe the discovery process is no longer run.
Anyway, to get the Management Pack discovery to work you need to add the "Network Service" account to both of the above security groups. You only need to allow this access for a small period of time until the discovery completes successfully and then you can remove the permissions. As you can see below, by default on an Edge Server not many accounts are added to those groups.
To correct the permissions issue run the following commands on the Edge Server using a Command Prompt (run as Administrator).
net localgroup "rtc local administrators" "network service" /add
net localgroup "rtc component local group" "network service" /add
You can then restart the OpsMgr agent service to speed up the discovery process. This can be done by running the following command from the Command Prompt:
Net stop HealthService && net start HealthService
If you see this message in your Event Log on the Edge Server discovery is working as expected.
WARNING: Once the discovery script is running and the Edge Server is reporting information to the Operations Manager, you can remove the Network Service from the two Edge Server local security groups in order to ensure the SfB installation is in the default configuration.
If you can keep an eye on the Operations Manager event log on the Edge to see if any more errors come up, as further corrective action may be required (i.e. if discovery needs to be run again in the future).
Well I hope you get some benefit from this blog and that you always make sure you monitor your Edge Server's.