SCOM can be quite complicated!
One of the biggest confusions is around why agents go gray, why you can't sort by their gray "state", and how to view only grayed out agents. The short answer is the health watcher service is the reason. An unreachable agent can't tell you it's unreachable; you need a "watcher" to tell you it's not reachable. The "Health Service Watcher" turns agents/servers in the console gray (which does NOT impact their state) when the SCOM Management Group can't reach/see them.
This impacts a lot of SCOM functionality!
- How do you get a list of gray servers? Create a view targeting to the Health Service Watcher and look for the unhealthy state.
- Why can't my scoped users see heartbeat failures? Why can't I sent them notifications on offline servers? Because you didn't include the Health Watcher Service in their group scope most likely. If you scoped them by group, modify that group to include the Health Service watcher. Scoping is great for security and segregation....when you do it right.
How To Create a Group of Servers that Includes their Health Service Watcher
- Create your groups as usual. For this example I'm going to assume you created them using the SCOM Console.
- Export the MP and open in Notepad (or your favorite editor). My sample MP is below.
- After the last </MembershipRule>, add in the text below. Make sure you update the variables as noted to match your environment!
<MonitoringClass>$MPElement[Name="[Alias for Microsoft.SystemCenter.Library]!Microsoft.SystemCenter.HealthServiceWatcher"]$</MonitoringClass>
<RelationshipClass>$MPElement[Name="[Alias for Microsoft.SystemCenter.InstanceGroup.Library]!Microsoft.SystemCenter.InstanceGroupContainsEntities"]$</RelationshipClass>
<MonitoringClass>$MPElement[Name="[Alias for Microsoft.SystemCenter.Library]!Microsoft.SystemCenter.HealthService"]$</MonitoringClass>
<MonitoringClass>$MPElement[Name="[Alias for Microsoft.Windows.Library]!Microsoft.Windows.Computer"]$</MonitoringClass>
Save the updated MP (keep the file name the same!) and reimport. Your group will now show the additional Health Service Watchers!
What we did was add another group population criteria (one not available in the console currently) and SCOM calculated the group membership using our updated rules. Users scoped to this new group will be able to see Heartbeat failure alerts, receive notifications when their servers are down, etc.