URLGenie Management Pack for SCOM - An Easy, Powerful Solution for Bulk Website Monitoring

 

2018.12.10: Version 2.0.0.36 available. Fixed the clientcert group population.
2018.10.23 Update: Version 2 now available!  

Current version: 2.0.0.36 (this does require a rip and replace of previous versions before v2.x) Management pack and Guide available for download here. (the guide has been fully updated )

 

Let me start by first saying that I was inspired to start this project after dissecting a very cool solution by Kristopher Bash. Without this excellent example, I would have never set out on this authoring journey.

A few years ago I was working for a large eCommerce company and I was responsible for monitoring a very large number of sites. I was tired of using the slow and cumbersome (although complex and powerful) built-in Web Application Transaction Monitoring wizard and wanted something faster and easier. My MCS buddy, Boklyn, suggested WebMon. Although that solution is definitely streamlined, I needed to be able to post SOAP messages to web services using authentication in addition to testing forms-based logins. I decided to build my own solution. This MP has been a work in progress for over 2 years. I have been slowly chipping away at my lengthy "to do" list for this pack only when my schedule would allow. The hardest part was finding the time to build a lab in Azure, followed by load testing, followed by extensive documentation with tutorials. After no small amount of testing, I finally feel like it's ready for the community to use. See the MP guide for tutorials and load testing results. I hope this management pack makes your life easier. Enjoy.

Overview

The URL Genie Management Pack provides a fast and easy way to implement monitoring for a large numbers of URL instances from only a few instances up to many thousands! In addition there are some special features which allow monitoring sites which require client certificates in addition to pages that use forms-based authentication. With URLGenie you can easily configure monitoring for thousands of standard URL instances in less than a minute.

The URL instances and their respective monitoring criteria get instantiated on any number of “watcher” nodes from one or more XML configuration files. Any managed Windows computer can be activated as a watcher node with a simple Console task during which the watcher nodes get configured with a path to where it should look for configuration files. There can exist any number of configuration files, each with any number of requests defined within. Typically the configuration files will be centrally located in a single shared network folder. A decent place for the shared configuration folder is on management server or the data warehouse server with all watcher nodes configured with the same shared folder path. This is the most simple and scalable configuration.

*Please make note of the supported limit of URLs here.

There are standard monitors which target the http and https class types. Each individual monitor will alert with plenty of alert specific context information. This is significantly different than the Operations Manager standard Web Availability or Synthetic Transaction monitoring which will only alert on the rollup and contain no specific or useful alert context information.

The standard monitors all support the various types of http authentication: None, Basic, NTLM, Digest, and Negotiate. In addition, there are special monitor types which can be enabled for URLs that require a client certificate or even for web sites that use forms-based authentication, a first for SCOM (to my knowledge)!

 

Setup Overview: (from the URLGenie Management Pack Guide)

  1. Import Management Pack
  2. Decide where you want to store your configuration files. URL instances get discovered from one or more configuration files. You can store files locally on each watcher node but my suggestion is to create a shared folder on your management server, data warehouse server, or other file server where you can read/write your configuration files. This way any URL instances that you define in your configuration files can be discovered by any/all watcher nodes depending on how you configure the <watchers> tags. See Parameters and Instance Properties section of the MP Guide.
  3. Activate 1 or more Watcher nodes. See "URLGenie EnableWatcherNode" task section of the MP Guide. Use the path from step 2 above.
  4. Create one or more configuration files. See Configuration File Examples section of the MP Guide. Once you activate 1 or more Watcher nodes, the instance discovery (http) will run on the activated Watcher node(s). The discovery will attempt to gather instance info from the config files (if the <watchers> tag matches the server name), then create the URL instances which will become automatically monitored within a few minutes after the group population workflows complete. Https discovery will occur once the initial http instances have been discovered.
  5. Profits.

 

Monitors

URLGenie DNS Resolution Failure Monitor
URLGenie Reachable Monitor
URLGenie Certificate Expires Soon Monitor
URLGenie Error Code Monitor
URLGenie Certificate Expired Monitor
URLGenie Status Code Monitor
URLGenie IE Login Scripted Monitor
URLGenie Scripted Monitor
URLGenie Content Monitor
URLGenie Certificate Invalid Monitor
URLGenie CA Untrusted Monitor
URLGenie WatcherNode ConfigFile Path Test Monitor
URLGenie Response Time Monitor

URLGenie Watcher Request Dependency Monitor
URLGenie Aggregate Health Monitor

Rules

URLGenie Scripted Request ResponseTime Collection Scripted PerfRule
URLGenie HTTP ContentSize Collection PerfRule
URLGenie HTTP Scripted IE Login ResponseTime Collection Scripted PerfRule
URLGenie HTTP DNS Resolution Time Collection PerfRule
URLGenie HTTP Days to Certificate Expiration PerfRule
URLGenie HTTP Time To Last Byte Collection PerfRule
URLGenie HTTP Download Time Collection PerfRule
URLGenie HTTP Time To First Byte Collection PerfRule
URLGenie HTTP ResponseTime Collection PerfRule

 

Screenshots

Load test with 6000 URL instances on one watcher node; a management server. Configured in minutes. (see MP guide for more detail)

 

 

Enable Watcher Node

1)  Execute EnableWatcherNode task

 

2)  Watcher node activation success

The output is verbose but we are looking for the success and verification of the config file path as shown below.

 

3) Watcher node is discovered.

 

Generate Configuration File

1) Start with a basic text file of URLs/addresses

 

2)  Run the task to generate the config file from this basic list.

 

3)  Config file is created successfully

 

4)  The configuratoin file is created with default parameters. Feel free to modify the settings as needed.
Notice the <watchers> tag contains "MS02". Any watcher nodes that contain "ms02" (not case sensitive) in their name (FQDN) will be able to discover these instances.

 

URL Instances

 

 Health Explorer

 

 

Critical Context Info

 

 

Alert View

 

 

Get Certificate Info Task (for https instances)

 

Forms Login Test

The Scripted IE Login monitor is able to simulate a login, even a 2-page login sequence (1:username, 2:password) like the example below.

Overview: To configure the IE Login Scripted Monitor

You will need to find the HTML IDs of the elements designated below. You can find the element IDs by enabling “developer mode” (Internet Explorer: F12, Chrome: CTRL+SHFT+I ) and selecting the objects to find the section of source code that defines the element properties. See the management pack guide for full tutorial.

Developer Mode (Internet Explorer)

Once you identify the form fields and buttons, you can override the monitor with the necessary values.

 

Below is an example of a login test with a TWO URL sequence:
https://login.live.com/login.srf^https://account.microsoft.com/devices

The first URL will open the login page (the first of two login pages; 1 for username, 2 for password).
The second URL is the actual target page that will be monitored for a content match: "find my device".

 

Forced content match error:

 

IE Login Test ContentMatch Failure Alert

The alert description indicates that the login test was successful but there was a content match error.

 

IE Login Monitor Healthy

After the ContentMatch field is set to a value that is expected to appear on the target page, the monitor returns to healthy and you can see the expected site text in the context area of Health Explorer.

 

 

 

 

 

 Example Email Notification (See this blog post for more details)

 

Severity: Critical Error

Severity:

Critical Error

Alert Description:

Content Validation Error

Monitor Settings:URL: https://www.qnap.comContentMatch String: hGroupID: URLGenie_DefaultDNSResolutionTime: 0Interval: 300RetryCount: 1Wiki: No link providedDescription: URL address to monitor.******* Request Headers *******GET / HTTP/1.1User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; Windows NT 6.1; en-US)Content-Type: text/xml;charset=utf-8Accept-Language: en-USAccept-Charset: utf-8From: SCOM@yourdomain.comConnection: Keep-Alive******* End Request Headers ************** Response Headers *******ResponseHeaders: HTTP/1.1 200 OKConnection: keep-aliveDate: Sat, 09 May 2015 00:09:25 GMTContent-Length: 1996Content-Type: text/html; charset=UTF-8Last-Modified: Tue, 21 Apr 2015 13:25:30 GMTETag: "4236f2-7cc-5143bfafc0680"Server: Apache******* End Response Headers *******

Command Channel:

MyDev

Source:

https://www.qnap.com

Path:

MS02.Contoso.com;\\Db01.contoso.com\scom_do_not_touch\URLGenie

Principal Name:

MS02.Contoso.com

Alert Name:

HTTP Request Error: Content Validation Error

Alert Resolution State:

New (0)

Alert Monitor Name:

URLGenie Content Monitor

Alert Monitor Description:

None

Time Raised:

5/8/2015 5:09:25 PM

Alert ID:

b1980341-a92d-453a-8fa7-75bc8e8fc003

SCOM Operations Console Info:

Operations Console Login Info

Web Console

Research It:

Bing It!

ADMIN DIAGNOSTIC INFO

--------------------

**Use this command to view the full details of this alert in SCOM Powershell console:

get-SCOMalert -Id "b1980341-a92d-453a-8fa7-75bc8e8fc003" | format-list *

This email sent from:

MS01.Contoso.com

CF1: Alert.NetBIOSName:

MS02

CF2: Alert.NetBIOSDomain Name:

CONTOSO

CF3: Alert.PrincipalName:

MS02.Contoso.com

CF4: SubscriberList:

Subscribers:TEST Subscriber;

CF5: Management Pack Name:

URLGenie Management Pack

CF6: Alert Class Name:

URLGenie.HttpRequest

CF7: Alert.Category

AvailabilityHealth

CF8: Workflow Type

MONITOR

CF9:

RESERVED

CF10: Helpful POSH Command

get-SCOMAlert -ID "b1980341-a92d-453a-8fa7-75bc8e8fc003" | format-list *

Context: Note: This context data is only relevant to the moment/time at which this alert was sent.

type : Microsoft.SystemCenter.WebApplication.WebApplicationDatatime : 2015-05-08T17:09:25.0005666-07:00sourceHealthServiceId : 120875E8-C79E-91CA-E3AC-EA09A391F4BDRequestResults : RequestResultsTransactionResponseTime : 0.0024875TransactionResponseTimeEvalResult : 0CollectPerformanceData : CollectPerformanceData

Knowledge Article:


Summary

The context information for these alerts is not always helpful. To see more detailed information about this alert log into the console for the management group.

Wiki


Note:

The context information provided in this notification is limited and is not always helpful. To see more detailed information about this alert, log into the appropriate SCOM console for the applicable datacenter (SCOM management group) and use the Health Explorer to find more details about the state change event for the object.

 

 

 

Please let me know if you have any suggestions for improvements.

Page History:

2018.10.23: Updated screenshots and content to reflect the version 2 updates.

2017.3.16.1330: Added updated MP version info.

2015.11.6: Fixed a typo.