System Center Operations Manager

Tasty Treats for System Center Operations Manager (by Tyson Paul)

URLGenie Management Pack for SCOM – An Easy, Powerful Solution for Bulk Website Monitoring

Let me start by first saying that I was inspired to start this project after dissecting a very cool solution by Kristopher Bash. Without this excellent example, I would have never set out on this authoring journey.

A few years ago I was working for a large eCommerce company and I was responsible for monitoring a very large number of sites. I was tired of using the slow and cumbersome (although complex and powerful) built-in Web Application Transaction Monitoring wizard and wanted something faster and easier. My MCS buddy, Boklyn, suggested WebMon. Although that solution is definitely streamlined, I needed to be able to post SOAP messages to web services using authentication in addition to testing forms-based logins, many of which used Javascript for field validation and submission. I decided to build my own solution. This MP has been a work in progress for over 2 years. I have been slowly chipping away at my lengthy “to do” list for this pack only when my schedule would allow. The hardest part was finding the time to build a lab in Azure, followed by load testing, followed by extensive documentation with tutorials. After no small amount of testing, I finally feel like it’s ready for the community to use. See the MP guide for tutorials and load testing results. I hope this management pack makes your life easier. Enjoy.

Current version: 2017.3.16.1330
Management pack and Guide available for download here. (sorry, the guide has not been fully updated yet)

2018.02.07: I’m pretty close to releasing the newest build of this thing. Anyone interested in testing it for me? Does anyone actually read this stuff?

Overview

The URL Genie Management Pack provides a fast and easy way to implement monitoring for a large numbers of URL instances from only a few instances up to many thousands! In addition there are some special features which allow monitoring sites which require client certificates in addition to pages that use forms-based authentication. With URLGenie you can easily configure monitoring for thousands of standard URL instances in less than a minute.

The URL instances and their respective monitoring criteria get instantiated on any number of “watcher” nodes from one or more XML configuration files. Any managed Windows computer can be activated as a watcher node with a simple Console task during which the watcher nodes get configured with a path to where it should look for configuration files. There can exist any number of configuration files, each with any number of requests defined within. Typically the configuration files will be centrally located in a single shared network folder. A decent place for the shared configuration folder is on management server or the data warehouse server with all watcher nodes configured with the same shared folder path. This is the most simple and scalable configuration.

*Please make note of the supported limit of URLs here.

There are standard monitors which target the http and https class types. Each individual monitor will alert with plenty of alert specific context information. This is significantly different than the Operations Manager standard Web Availability or Synthetic Transaction monitoring which will only alert on the rollup and contain no specific or useful alert context information.

The standard monitors all support the various types of http authentication: None, Basic, NTLM, Digest, and Negotiate. In addition, there are special monitor types which can be enabled for URLs that require a client certificate or even for web sites that use forms-based authentication, a first for SCOM (to my knowledge)!

 

Setup Overview: (from the URLGenie Management Pack Guide)

  1. Import Management Pack
  2. Decide where you want to store your configuration files. URL instances get discovered from one or more configuration files. You can store files locally on each watcher node but my suggestion is to create a shared folder on your management server, data warehouse server, or other file server where you can read/write your configuration files. This way any URL instances that you define in your configuration files can be discovered by any/all watcher nodes depending on how you configure the <watchers> tags. See Parameters and Instance Properties section of the MP Guide.
  3. Activate 1 or more Watcher nodes. See “URLGenie EnableWatcherNode” task section of the MP Guide. Use the path from step 2 above.
  4. Create one or more configuration files. See Configuration File Examples section of the MP Guide. Once you activate 1 or more Watcher nodes, the instance discovery (http) will run on the activated Watcher node(s). The discovery will attempt to gather instance info from the config files (if the <watchers> tag matches the server name), then create the URL instances which will become automatically monitored within a few minutes after the group population workflows complete. Https discovery will occur once the initial http instances have been discovered.

 

Monitors

  • URL Request Dependency Monitor
  • URLGenie CA Untrusted Monitor
  • URLGenie Certificate Expired Monitor
  • URLGenie Content Monitor
  • URLGenie DNS Resolution Failure Monitor
  • URLGenie Error Code Monitor
  • URLGenie Status Code Monitor
  • URLGenie Reachable Monitor
  • URLGenie Response Time Monitor
  • URLGenie Client Certificate Scripted Monitor
  • URLGenie URLGrab Scripted Monitor
  • URLGenie URLGrab IE Login Scripted Monitor

Rules

  • URLGenie ResponseTime Collection PerfRule
  • URLGenie URLGrab IE Login ResponseTime Collection Scripted PerfRule
  • URLGenie URLGrab Client Certificate ResponseTime Collection Scripted PerfRule
  • URLGenie URLGrab ResponseTime Collection Scripted PerfRule
  • URLGenie WatcherNode ConfigFile Path Test Alert Rule

 

Tasks

  • URLGenie Test Path
  • URLGenie Generate Config From List
  • URLGenie PingServer
  • URLGenie Get Certificate Info
  • URLGenie DisableWatcherNode
  • URLGenie EnableWatcherNode

 

Screenshots

Load test with 6000 URL instances on one watcher node (a management server). Configured in minutes. (see MP guide for more detail)

 

 

Enable Watcher Node

1)  Execute EnableWatcherNode task

 

2)  Watcher node activation success

The output is verbose but we are looking for the success and verification of the config file path as shown below.

 

3) Watcher node is discovered. Health rollup for the watcher is disabled by default.

 

Generate Configuration File

1) Start with a basic text file of URLs/addresses

 

2)  Run the task to generate the config file from this basic list.

 

3)  Config file is created successfully

 

4)  The configuratoin file is created with default parameters. Feel free to modify the settings as needed.
Notice the <watchers> tag contains “MS02”. Any watcher nodes that contain “ms02” (not case sensitive) in their name (FQDN) will be able to discover these instances.

 

URL Instances

 

 Health Explorer

 

 

Critical Context Info

 

 

Alert View

 

 

Get Certificate Info Task (for https instances)

 

Site Login Test

To configure the IE Login Scripted Monitor you will need to find the HTML IDs of the elements designated below. You can find the element IDs by enabling “developer mode” ( Internet Explorer: F12, Chrome: CTRL+SHFT+I ) and selecting the objects to find the section of source code that defines the element properties. See the management pack guide for full tutorial.

 

 

Example of a forced critical condition by overriding content parameter

By setting the ContentMatch to something unlikely, I have created a false error condition for the monitor and can now examine the context data to validate that the login is occurring correctly. By looking at the InnerText data retrieved from the web page after logging in, I see text that appears on the expected account page after successfully logging into the target site. This is shown in the state change context data area in Health Explorer and also on the Alert Context tab and Description field of the resulting alert.

 

IE Login Test ContentMatch Failure Alert

The alert description indicates that the login test was successful but did not find the content match as intended.


 

IE Login Monitor Healthy

After the ContentMatch field is set to a value that is expected to appear on the target page, the monitor returns to healthy and you can see the expected site text in the context area of Health Explorer.

 

 

 

 

 Example Email Notification (See this blog post for more details)

 

Severity: Critical Error

Severity:

Critical Error

Alert Description:

Content Validation Error

Monitor Settings:
URL: https://www.qnap.com
ContentMatch String: h
GroupID: URLGenie_Default
DNSResolutionTime: 0
Interval: 300
RetryCount: 1
Wiki: No link provided
Description: URL address to monitor.

******* Request Headers *******
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; Windows NT 6.1; en-US)
Content-Type: text/xml;charset=utf-8
Accept-Language: en-US
Accept-Charset: utf-8
From: SCOM@yourdomain.com
Connection: Keep-Alive

******* End Request Headers *******

******* Response Headers *******
ResponseHeaders: HTTP/1.1 200 OK
Connection: keep-alive
Date: Sat, 09 May 2015 00:09:25 GMT
Content-Length: 1996
Content-Type: text/html; charset=UTF-8
Last-Modified: Tue, 21 Apr 2015 13:25:30 GMT
ETag: “4236f2-7cc-5143bfafc0680”
Server: Apache

******* End Response Headers *******

Command Channel:

MyDev

Source:

https://www.qnap.com

Path:

MS02.Contoso.com;\\Db01.contoso.com\scom_do_not_touch\URLGenie

Principal Name:

MS02.Contoso.com

Alert Name:

HTTP Request Error: Content Validation Error

Alert Resolution State:

New (0)

Alert Monitor Name:

URLGenie Content Monitor

Alert Monitor Description:

None

Time Raised:

5/8/2015 5:09:25 PM

Alert ID:

b1980341-a92d-453a-8fa7-75bc8e8fc003

SCOM Operations Console Info:

Operations Console Login Info

SCOM Web Console link:

Web Console

Research It:

Bing It!

ADMIN DIAGNOSTIC INFO

——————–

**Use this command to view the full details of this alert in SCOM Powershell console:

get-SCOMalert -Id “b1980341-a92d-453a-8fa7-75bc8e8fc003” | format-list *

This email sent from:

MS01.Contoso.com

CF1: Alert.NetBIOSName:

MS02

CF2: Alert.NetBIOSDomain Name:

CONTOSO

CF3: Alert.PrincipalName:

MS02.Contoso.com

CF4: SubscriberList:

Subscribers:TEST Subscriber;

CF5: Management Pack Name:

URLGenie Management Pack

CF6: Alert Class Name:

URLGenie.HttpRequest

CF7: Alert.Category

AvailabilityHealth

CF8: Workflow Type

MONITOR

CF9:

RESERVED

CF10: Helpful POSH Command

get-SCOMAlert -ID “b1980341-a92d-453a-8fa7-75bc8e8fc003” | format-list *

Context:
Note: This context data is only relevant to the moment/time at which this alert was sent.

type : Microsoft.SystemCenter.WebApplication.WebAp
plicationData
time : 2015-05-08T17:09:25.0005666-07:00
sourceHealthServiceId : 120875E8-C79E-91CA-E3AC-EA09A391F4BD
RequestResults : RequestResults
TransactionResponseTime : 0.0024875
TransactionResponseTimeEvalResult : 0
CollectPerformanceData : CollectPerformanceData

Knowledge Article:


Summary

The context information for these alerts is not always helpful. To see more detailed information about this alert log into the console for the management group.

Wiki


Note:

The context information provided in this notification is limited and is not always helpful. To see more detailed information about this alert, log into the appropriate SCOM console for the applicable datacenter (SCOM management group) and use the Health Explorer to find more details about the state change event for the object.

Please let me know if you have any suggestions for improvements.

Page History

2017.3.16.1330: Added updated MP version info.

2015.11.6: Fixed a typo.