URLGenie Management Pack for SCOM – An Easy, Powerful Solution for Bulk Website Monitoring


Let me start by first saying that I was inspired to start this project after dissecting a very cool solution by Kristopher Bash. Without this excellent example, I would have never set out on this authoring journey.

A few years ago I was working for a large eCommerce company and I was responsible for monitoring a very large number of sites. I was tired of using the slow and cumbersome (although complex and powerful) built-in Web Application Transaction Monitoring wizard and wanted something faster and easier. My MCS buddy, Boklyn, suggested WebMon. Although that solution is definitely streamlined, I needed to be able to post SOAP messages to web services using authentication in addition to testing forms-based logins, many of which used Javascript for field validation and submission. I decided to build my own solution. This MP has been a work in progress for over 2 years. I have been slowly chipping away at my lengthy “to do” list for this pack only when my schedule would allow. The hardest part was finding the time to build a lab in Azure, followed by load testing, followed by extensive documentation with tutorials. After no small amount of testing, I finally feel like it’s ready for the community to use. See the MP guide for tutorials and load testing results. I hope this management pack makes your life easier. Enjoy.

 

NEW! Current version: 2017.3.16.1330
Management pack and Guide available for download here. (sorry, the guide has not been fully updated yet)

Current version: 2015.5.8.1706 here (OLD version )

Overview

The URL Genie Management Pack provides a fast and easy way to implement monitoring for a large numbers of URL instances from only a few instances up to many thousands! In addition there are some special features which allow monitoring sites which require client certificates in addition to pages that use forms-based authentication. With URLGenie you can easily configure monitoring for thousands of standard URL instances in less than a minute.

The URL instances and their respective monitoring criteria get instantiated on any number of “watcher” nodes from one or more XML configuration files. Any managed Windows computer can be activated as a watcher node with a simple Console task during which the watcher nodes get configured with a path to where it should look for configuration files. There can exist any number of configuration files, each with any number of requests defined within. Typically the configuration files will be centrally located in a single shared network folder. A decent place for the shared configuration folder is on management server or the data warehouse server with all watcher nodes configured with the same shared folder path. This is the most simple and scalable configuration.

There are standard monitors which target the http and https class types. Each individual monitor will alert with plenty of alert specific context information. This is significantly different than the Operations Manager standard Web Availability or Synthetic Transaction monitoring which will only alert on the rollup and contain no specific or useful alert context information.

The standard monitors all support the various types of http authentication: None, Basic, NTLM, Digest, and Negotiate. In addition, there are special monitor types which can be enabled for URLs that require a client certificate or even for web sites that use forms-based authentication, a first for SCOM (to my knowledge)!

 

Setup Overview: (from the URLGenie Management Pack Guide)

  1. Import Management Pack
  2. Decide where you want to store your configuration files. URL instances get discovered from one or more configuration files. You can store files locally on each watcher node but my suggestion is to create a shared folder on your management server, data warehouse server, or other file server where you can read/write your configuration files. This way any URL instances that you define in your configuration files can be discovered by any/all watcher nodes depending on how you configure the <watchers> tags. See Parameters and Instance Properties section of the MP Guide.
  3. Activate 1 or more Watcher nodes. See “URLGenie EnableWatcherNode” task section of the MP Guide. Use the path from step 2 above.
  4. Create one or more configuration files. See Configuration File Examples section of the MP Guide. Once you activate 1 or more Watcher nodes, the instance discovery (http) will run on the activated Watcher node(s). The discovery will attempt to gather instance info from the config files (if the <watchers> tag matches the server name), then create the URL instances which will become automatically monitored within a few minutes after the group population workflows complete. Https discovery will occur once the initial http instances have been discovered.

 

Monitors

  • URL Request Dependency Monitor
  • URLGenie CA Untrusted Monitor
  • URLGenie Certificate Expired Monitor
  • URLGenie Content Monitor
  • URLGenie DNS Resolution Failure Monitor
  • URLGenie Error Code Monitor
  • URLGenie Status Code Monitor
  • URLGenie Reachable Monitor
  • URLGenie Response Time Monitor
  • URLGenie Client Certificate Scripted Monitor
  • URLGenie URLGrab Scripted Monitor
  • URLGenie URLGrab IE Login Scripted Monitor

Rules

  • URLGenie ResponseTime Collection PerfRule
  • URLGenie URLGrab IE Login ResponseTime Collection Scripted PerfRule
  • URLGenie URLGrab Client Certificate ResponseTime Collection Scripted PerfRule
  • URLGenie URLGrab ResponseTime Collection Scripted PerfRule
  • URLGenie WatcherNode ConfigFile Path Test Alert Rule

 

Tasks

  • URLGenie Test Path
  • URLGenie Generate Config From List
  • URLGenie PingServer
  • URLGenie Get Certificate Info
  • URLGenie DisableWatcherNode
  • URLGenie EnableWatcherNode

 

Screenshots

Load test with 6000 URL instances on one watcher node (a management server). Configured in minutes. (see MP guide for more detail)

 

 

Enable Watcher Node

1)  Execute EnableWatcherNode task

 

2)  Watcher node activation success

The output is verbose but we are looking for the success and verification of the config file path as shown below.

 

3) Watcher node is discovered. Health rollup for the watcher is disabled by default.

 

Generate Configuration File

1) Start with a basic text file of URLs/addresses

 

2)  Run the task to generate the config file from this basic list.

 

3)  Config file is created successfully

 

4)  The configuratoin file is created with default parameters. Feel free to modify the settings as needed.
Notice the <watchers> tag contains “MS02”. Any watcher nodes that contain “ms02” (not case sensitive) in their name (FQDN) will be able to discover these instances.

 

URL Instances

 

 Health Explorer

 

 

Critical Context Info

 

 

Alert View

 

 

Get Certificate Info Task (for https instances)

 

Site Login Test

To configure the IE Login Scripted Monitor you will need to find the HTML IDs of the elements designated below. You can find the element IDs by enabling “developer mode” ( Internet Explorer: F12, Chrome: CTRL+SHFT+I ) and selecting the objects to find the section of source code that defines the element properties. See the management pack guide for full tutorial.

 

 

Example of a forced critical condition by overriding content parameter

By setting the ContentMatch to something unlikely, I have created a false error condition for the monitor and can now examine the context data to validate that the login is occurring correctly. By looking at the InnerText data retrieved from the web page after logging in, I see text that appears on the expected account page after successfully logging into the target site. This is shown in the state change context data area in Health Explorer and also on the Alert Context tab and Description field of the resulting alert.

 

IE Login Test ContentMatch Failure Alert

The alert description indicates that the login test was successful but did not find the content match as intended.


 

IE Login Monitor Healthy

After the ContentMatch field is set to a value that is expected to appear on the target page, the monitor returns to healthy and you can see the expected site text in the context area of Health Explorer.

 

 

 

 

 Example Email Notification (See this blog post for more details)

 

Severity: Critical Error

Severity:

Critical Error

Alert Description:

Content Validation Error

Monitor Settings:
URL: https://www.qnap.com
ContentMatch String: h
GroupID: URLGenie_Default
DNSResolutionTime: 0
Interval: 300
RetryCount: 1
Wiki: No link provided
Description: URL address to monitor.

******* Request Headers *******
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; Windows NT 6.1; en-US)
Content-Type: text/xml;charset=utf-8
Accept-Language: en-US
Accept-Charset: utf-8
From: SCOM@yourdomain.com
Connection: Keep-Alive

******* End Request Headers *******

******* Response Headers *******
ResponseHeaders: HTTP/1.1 200 OK
Connection: keep-alive
Date: Sat, 09 May 2015 00:09:25 GMT
Content-Length: 1996
Content-Type: text/html; charset=UTF-8
Last-Modified: Tue, 21 Apr 2015 13:25:30 GMT
ETag: “4236f2-7cc-5143bfafc0680”
Server: Apache

******* End Response Headers *******

Command Channel:

MyDev

Source:

https://www.qnap.com

Path:

MS02.Contoso.com;\\Db01.contoso.com\scom_do_not_touch\URLGenie

Principal Name:

MS02.Contoso.com

Alert Name:

HTTP Request Error: Content Validation Error

Alert Resolution State:

New (0)

Alert Monitor Name:

URLGenie Content Monitor

Alert Monitor Description:

None

Time Raised:

5/8/2015 5:09:25 PM

Alert ID:

b1980341-a92d-453a-8fa7-75bc8e8fc003

SCOM Operations Console Info:

Operations Console Login Info

SCOM Web Console link:

Web Console

Research It:

Bing It!

ADMIN DIAGNOSTIC INFO

——————–

**Use this command to view the full details of this alert in SCOM Powershell console:

get-SCOMalert -Id “b1980341-a92d-453a-8fa7-75bc8e8fc003” | format-list *

This email sent from:

MS01.Contoso.com

CF1: Alert.NetBIOSName:

MS02

CF2: Alert.NetBIOSDomain Name:

CONTOSO

CF3: Alert.PrincipalName:

MS02.Contoso.com

CF4: SubscriberList:

Subscribers:TEST Subscriber;

CF5: Management Pack Name:

URLGenie Management Pack

CF6: Alert Class Name:

URLGenie.HttpRequest

CF7: Alert.Category

AvailabilityHealth

CF8: Workflow Type

MONITOR

CF9:

RESERVED

CF10: Helpful POSH Command

get-SCOMAlert -ID “b1980341-a92d-453a-8fa7-75bc8e8fc003” | format-list *

Context:
Note: This context data is only relevant to the moment/time at which this alert was sent.

type : Microsoft.SystemCenter.WebApplication.WebAp
plicationData
time : 2015-05-08T17:09:25.0005666-07:00
sourceHealthServiceId : 120875E8-C79E-91CA-E3AC-EA09A391F4BD
RequestResults : RequestResults
TransactionResponseTime : 0.0024875
TransactionResponseTimeEvalResult : 0
CollectPerformanceData : CollectPerformanceData

Knowledge Article:


Summary

The context information for these alerts is not always helpful. To see more detailed information about this alert log into the console for the management group.

Wiki


Note:

The context information provided in this notification is limited and is not always helpful. To see more detailed information about this alert, log into the appropriate SCOM console for the applicable datacenter (SCOM management group) and use the Health Explorer to find more details about the state change event for the object.

Please let me know if you have any suggestions for improvements.

Page History

2017.3.16.1330: Added updated MP version info.

2015.11.6: Fixed a typo.


Comments (98)

  1. Anonymous says:

    Thanks, Tyson! Looks like a very useful solution. I'll be trying this out very soon!

  2. Anonymous says:

    Hi Paul,

    I'm using (or better want to use 😉 )  a Windows 7 machine as a watcher node, even after Updating PowerShell to V 3.0 I'm not able to add the windows 7 machine as a watcher (Still gives the error wrong PowerShell version)

    I know this is a know SCOM issue (you need to start PowerShell again from the SCOM script) , but as it works on the servers; I'm not sure if it is the same issue.

    Greetings Marc

  3. Tyson.Paul says:

    @Powershell Guy: I've run into this problem before with another client. They were able to update the Windows Management Framework to v4.0 on the Windows 7 machine, and this solved the problem.

    http://www.microsoft.com/…/details.aspx

    Please let me know your findings. If you still encounter problems, please email screenshots to me.

  4. Anonymous says:

    Thanks. Will be testing this soon.

  5. Anonymous says:

    Looks really good but i'm having trouble with a scripted IE monitor trying to log into a forms authenticated site  – the submit button doesn't have an ID just

    <input name="login" class="submit button-1" type="submit" value="Login">

    Is there any way to get this to work?

  6. Tyson.Paul says:

    PaulO,

      I've run into this before and I was able to get it to work correctly by simply using the input name which, in your case, is "login".  Let me know if you still have problems.

    -Tyson

  7. Anonymous says:

    Thanks Tyler,

    I gave that a shot, still no joy the page being returned is the login page (rather than the logn failed try again page). Just to rule out the login profile I overrode the username/password in the monitor to the actual strings  (still no good). Just to make things harder I'm trying to monitor a service that is hosted by a third party so I don't have access to access logs – I suspect i'll be firing up fiddler or similar on watcher node.

    Cheers

    PaulO

  8. Tyson.Paul says:

    PaulO,

      Can you try to see if you have Mshtml.dll registered correctly on your watcher node? Try running this command from an elevated Powershell console:

    Get-ChildItem C:Windows -include "Microsoft.mshtml.dll*" -Recurse

  9. Anonymous says:

    Thanks again Tyler,

    Output is below  – the probe is honouring the contnetmatch  – so I suspect this isn't the problem

    PS C:Windowssystem32> Get-ChildItem C:Windows -Include "Microsoft.mshtml.dll" -Recurse

       Directory: C:WindowsassemblyGACMicrosoft.mshtml7.0.3300.0__b03f5f7f11d50a3a

    Mode                LastWriteTime     Length Name

    —-                ————-     —— —-

    -a—        15/05/2015  10:32 AM    8013664 Microsoft.mshtml.dll

    Cheers,

    Paul O

  10. Geo says:

    Imported to my test box and when I go to enable the watcher node, I don't get prompted with the dialogue to override, so it's using the default of \YourFileserver.domain.comSCOM_DO_NOT_TOUCHURLGenie, what did I miss.

    Thanks

  11. steven says:

    if i need to monitor the url via proxy, how can i configure it

  12. Tyson.Paul says:

    Steven,
      See the management pack guide. I've copied/pasted some of the configuration file element definitions below. You must configure these within the configuration file(s).
    Proxy [string, URL]
    Default: <blank>
    Description: Optional. Address of your proxy server.
    ProxyAuthenticationScheme [string] <"None" | "Basic" | "Digest" | "NTLM" | "Negotiate">
    Default: "None"
    Description: If authentication method other than None is used, you will have to specify a username (see ProxyUser) and also configure a Basic account type in the Console, then assign that account to the object instance with the "URLGenie Proxy Account".
    ProxyPass [Basic Account Type, configured in the Console]
    Default:
    Description: THIS IS NOT CONFIGURED IN THE .XML CONFIG FILE. However, this parameter is overrideable for most monitors. If ProxyAuthenticationScheme authentication method other than None is used, you will have to specify a username (see ProxyUser) and also configure a Basic account credential type in the Console, then assign that account to the object instance with the URLGenie Proxy Account profile. Only the password from this credential object is used.
    ProxyUser [string]
    Default: <blank>
    Description: Username for use with Proxy authentication. Note: the password must be configured in the Console with a Basic Account type.
     
    Here is an example of what your elements might look like in the configuration file when using a proxy with NTLM:
    <url><![CDATA[http://www.somewebsite.com]]></url>
    ……
    <proxy><![CDATA[http://proxy.contoso.com:8080]]></proxy>
    <proxyauthenticationscheme>NTLM</proxyauthenticationscheme>
    <proxyuser>contosoksmith</proxyuser>    
     
    You would create a Basic Account:
    name: KSmith Proxy Access Account
    username: <doesn't matter, use anything>
    password: <password for domain user account: contosoksmith >
    The domain user account "KSmith" must exist in Active Directory.
    Add the "KSmith Proxy Access Account" to the URLGenie Proxy Account profile and apply it to the Object:
    http://proxy.contoso.com:8080 http://www.somewebsite.com
     
    Here's an example of what your elements might look like in the configuration file when using a proxy with no authentication (None).
    <url><![CDATA[http://www.somewebsite.com]]></url>
    ……
    <proxy><![CDATA[http://proxy.contoso.com:8080]]></proxy>
    <proxyauthenticationscheme>None</proxyauthenticationscheme>
    <proxyuser></proxyuser>    

  13. steven says:

    it is a must to include <![CDATA? . my proxy is not in internet url format.

  14. Tyson.Paul says:

    @Steven:  CData is not required but is helpful when your URL contains special characters.

  15. Steven says:

    i insert xx.xx.xx:8080 proxy ip into xml but it doesn't seem work. authenticatiion is None. How long will it take effect?

  16. steven says:

    i can't get it working with or withput CData. How long would it take effect after edited the .xml. The proxy is with None authentication.

  17. steven says:

    @tyson, i still not able to get it working for proxy and web which require password by follow the guide

  18. Tyson.Paul says:

    @Steven:  Use the "Email Blog Author" link above and I will do my best to assist you directly via email.

  19. Keegs says:

    First off, this is GREAT!  Being able to whip up monitoring on a list of URLs almost instantly is a huge time saver.  I have a question regarding monitoring the same URL multiple times though.  I have multiple websites that are hosted on multiple servers for redundancy.  So I'd like to check Website A and Website B both on Server 1 and Server 2.  How I currently have it setup is to check "http://Server1&quot; and send the HOST header of "www.WebsiteA.com" on 1 request and "www.WebsiteB.com" on another.  It looks like the URL Genie views this as 2 checks for the same URL so it only sends 1.  Example being I see the monitor for Website B on Server 1, but not for Website A.

    Is there a better way of doing this, or a way of doing this at all?  Any help would be great.  Thanks so much!

    1. Tyson.Paul says:

      Keegs,
      Is this still an issue for you? If so, let me know.

      1. Keegs says:

        @Tyson.Paul yes this is still a thing. Any help would be much appreciated.

        1. Tyson.Paul says:

          @Keegs,
          Will you please send me your email address by using the “contact author” link that should be somewhere on this page? I have a solution for you.

          1. Tyson.Paul says:

            Huh. Just got home and tried to find the “contact author” link. It no longer seems to be available after the recent blog migration. You can find my contact info at the top of the management pack guide. Send me an email. I can help.

  20. Marcel says:

    Great tool thx, it could save us a lot of time 🙂

    The only challenge we have is, that it seems that the "retry count" doesn't seem to work.

    We expected with the setting in the xml, for a url :

    – Interval: 120

    – RetryCount: 3

    that a alert will not be fired until the watcher node measures 3x unavailability (with a interval of 120 seconds)

    In our environment, with this setting, a alert is generated directly (after first sample), so it seems it ignores the "RetryCount" setting…

    As a result our dashboard has to much alerts, without having a serious problem, because it happens often that it misses a sample (reboot etc.)

    We only would like a alert if there is a real problem.

    Are you aware of this problem, or do we miss something.

    Thanks for your help and keep up the good work!

  21. Tyson.Paul says:

    @Marcel,

      The retry count problem is a known bug in the underlying module.

  22. Marcel says:

    @Paul

    Thanks for your quick answer, unfortunately it's not the answer we hoped for :o)

    Is there any development going on for the urlgenie pack? Can it be fixed within the pack, or can we better focus on another solution? thx again!

  23. Hales says:

    Trying to find if I can monitor for statuses >399.  I need to monitor a subset of sites that use IP filtering, and I need to see anything more than 399 as a success, rather than a failure, is this possible?

    Thanks

  24. Tyson.Paul says:

    @Marcel,

      Unfortunately it is the product group that must fix the underlying code of the module. When I get a chance I'll revisit that issue and see if I can get update on the bug status.

  25. Tyson.Paul says:

    @Hales,

     I can look into other possibilities but as of now the comparison logic will only use > for an error detection. Ping me next month and I may have good news for you.

    1. Tyson.Paul says:

      @Hales,
      I’ve been working on porting over the MP into VSAE. I’m about halfway done. So far I’ve added some improvements: scripted monitor, unique instance names, and working on the “retry count” issue.

  26. Hales says:

    Also, using the IE scripted login – is there a way to handle ADFS logins?

    i.e: Go to a website, click on a button, authentication pops up redirecting through ADFS before reaching page

  27. Rene says:

    @Paul.

    great MP. We now have 98 URL's to monitor. All "simple" URL-checks.

    So far so good.

    Now we need to insert 15 URL's that need a client-side certificate.

    I extracted the thumbprint and entered it into the xml-entry.

    unfortunately there's an error. No certificate found on server…

    Provider of the certificate is 100% sure it's the right certificate…

    Any idea???

    Grtz.  Rene

  28. Hales says:

    @Paul – And one final thing, is there any way for changing the display name so it's not just the URL?

    Thanks

  29. Tyson.Paul says:

    @Hales:   Nope. However, you can use the "Description" field for this purpose.

  30. Tyson.Paul says:

    @Rene: Override the workflow, set WriteToEventLog parameter to 2 then check the event log on the watcher node for more detail.

  31. Dave Bastin says:

    Why is my watcher node trying to connect to 93.184.221.200?

    Aliases:  download.gigabyte.eu

             gigabytestorage.azureedge.net

             gigabytestorage.ec.azureedge.net

    1. Tyson.Paul says:

      @Dave,
      I have no idea. It’s not related to URLGenie unless you’re watching some site at that address.

  32. Rene says:

    Paul,

    sorry for the late respone..Due to other work, I wasn't able to test htis earlier… Again Sorry.

    Message I get in event-log:

    The term 'Invoke-WebRequest' is not recognized as the name of a cmdlet

    Check on the watcher-node says: Powershel version 4 -1 -1

    invoke-webrequest in Powershell ISE works localy on watchernode.

    Certificate is found when I logon localy on watchernode…

    Any other Idea what I'm doing wrong??

    Thanx…….  Rene

  33. Rene says:

    Paul, thnx for the advise.

    Error I see in eventlog is: The term 'Invoke-WebRequest' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

    When I enter the same invoke-webrequest on the watcher-node localy it works fine. PSversion = 4..

  34. Jörg says:

    Hi,

    we made our first tests with URLGenie and it looks great.

    But we have one Problem to solve, so that we can use ist for all our checks.

    We send a XML to a secific URL and get a Response back. We check the returning XML by ContentMatch. The Problem now ist, that there are two or more allowed strings, that may come back. We only can check one string, so the other positive strings get an negative result.

    Is there a possibility to have more strings or regular expressions allowed?

    Thanks,

    Jörg

  35. X from Canada says:

    Hi Paul,

    How can I remove monitored URLs? I have some test URLs going on with URLGenies but not be able to remove them.

    Thanks,

    X

  36. Tyson.Paul says:

    @X,  rename the config file(s) to Requests_whatever.xml.BAK.

    Or you could delete them but I think renaming is better. As long as the discovery workflow can still access the configured shared folder path, it should undiscover the URLs when it finds zero config files.

  37. Tyson.Paul says:

    @Keegs,

      I don't have an easy workaround for you at this time. Perhaps add a trailing slash to make the second URL unique.

    1). http://www.something.com

    2). http://www.something.com/

  38. X from Canada says:

    @Paul, I have removed them in configuration file but it won't go away, I will follow your instruction to rename or delete Request file.

    It is awesome MP with a lot of manual steps, haha. We are going to save x,000 dollars from it.

    Thank you,

    X

  39. X from Canada says:

    @Paul, yes, it worked. Removed all URLs.

    Thanks.

  40. Tyson.Paul says:

    @Jörg:

      You can override the ContentMatch monitor.

    Override this parameter: ContentMatchRegExOperator

    with this value: DoesNotMatchRegularEspression

    Use a regular expression for the ContentMatch value; something like this:  s*(firststring|second|third)

    For more content matching operator options see  "ContentMatchRegExOperator" in the "Parameters and Instance Properties" section of the management pack guide.

  41. Tyson.Paul says:

    @Marcel,

       I think I might have a way to solve the RetryCount issue but I won't have time for this side project until a few months down the road.

    1. Oz says:

      Any idea when the RetryCount bug will be fixed? Only asking since we get a significant number of errors per day that aren’t really an issue if it were just to perform some retries.

      1. Tyson.Paul says:

        Hi Oz,
        In order to fix this (I do think it is possible) I have to port the MP over to Visual Studio (from the 2007R2 Authoring Console). This will take a significant amount of effort to do it cleanly and correctly. I have actually begun this endeavor though I have much left to do. As for an estimate…. I can only promise to chip away at it as my free time permits and free time is in very short supply. Though I have been known to move much faster if those of you who ask for things are willing to make donations to charity. (just some suggestions: https://chivecharities.org/ or http://www.alsa.org/ ).

  42. Juan says:

    I am sorry for asking this, but How can I remove the URLGenie MP?

    I couldn't make it work, and I would like to remove it properly  from SCOM

  43. Tyson.Paul says:

    @Juan,

      I'm happy to help you get it configured. However, if you want to remove it, simply delete the management pack.

  44. Christanya says:

    Paul, am new to SCOM but I really like this MP but this requirement of the URL being unique may prevent me from using this professionally as we have a need multiple requests to post various parms to the same site to then evaluate the results via the simple contentmatch. What can we do to make changing this requirement a high(er?) priority for you and the teams involved?

    1. Tyson.Paul says:

      Hi Christanya,
      I can see how this change could prove to be helpful. This is a project that I’ve been slowly developing over a few years as time permits. I travel a LOT and free time doesn’t exist. At Microsoft there is an endless (yes endless) list of things that I need to do, have to do, could do, and want to do. There is no team, just a man on a mission to make the world a little bit better for nice folks like you. That being said, if you would agree to do something nice for someone else (like a child or veteran) then I would be inclined to make this code modification a priority for you.
      I’ll make a suggestion: https://chivecharities.org/

      Your move.

      -T

      1. Christanya says:

        Paul, I love your challenge but this solution is for my employer, not me personally, and to cut through the red tape with procurement / HR to allow me to do this without a firm commitment when representing them will be a daunting effort. If you want to take this conversation into private email, I can prove that I, personally, have been a continual donor to 3 children’s hospitals in Texas for over 10 years and my employer has a large giving presence that is publically verifiable. Again, I’m not familiar with SCOM and management packs so if you can let me know if your sealed MP can be edited by anyone to achieve the desired results (allowing non-unique URLs), then we can go that route and I would be open to sharing our results with you with no quid pro quo involved.

        1. Tyson.Paul says:

          Ok Christanya, sounds legit. I’ll do it. Give me a few days to find some time. Maybe this Friday. Glad to hear you are helping to make the world a better place. 😉

          1. Christanya says:

            Thank you, Paul!

  45. Ken says:

    Tyson first off thanks for creating this wonderful MP this will be truly helpful (if I can get it to work ; ) which is why I’m reaching out to you. I’m very very new to SCOM but I have installed and configured a new 2012 R2 environment and I have deployed your MP per your thorough instructions. However I can’t seem to produce any url monitoring traffic which tells me something is wrong somewhere I just don’t know where. Watcher node is present, xml file created, permissions are all set. Haven’t noticed any errors in Ops log reference this MP. Any suggestions would be greatly appreciated.

    1. Tyson.Paul says:

      Hi Ken,
      Check your hotmail inbox.

  46. Mark Derveloy says:

    I was able to successfully configure URL monitor for website answering on port 80, but getting errorcode violation “2147500037” for other URLs which include port non-standard port numbers within the URL.

    Example: http://server.domain.com/test – Success
    Example: http://server.domain.com:7777/test – Errorcode “2147500037”

  47. Yousef says:

    I am trying to monitor a website using URLGenie MP, I followed the documentation to monitor website with login “URLGenie URLGrab IE Login Scripted Monitor” but I am getting the below error.

    Please help

    Property Value

    Failure

    FAILURE retrieving the URL: [ https://XXXX.XXx.com ] ! $InnerText is empty: [] ..while attempting to fetch: [https://citrixkw.agility.com] ____ InnerText: ____ ErrorData below: You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. You cannot call a method on a null-valued expression. The property ‘StatusBar’ cannot be found on this object. Verify that the property exists and can be set. The property ‘visible’ cannot be found on this object. Verify that the property exists and can be set. Retrieving the COM class factory for component with CLSID {0002DF01-0000-0000-C000-000000000046} failed due to the following error: 80080005 Server execution failed (Exception from HRESULT: 0x80080005 (CO_E_SERVER_EXEC_FAILURE)).

    31.9325

    1. Tyson.Paul says:

      @Yousef,
      I have seen this problem before. I believe it is related to this library: Microsoft.mshtml.dll . Let me know if you are still having this problem and I’ll do my best to help you find a solution.

      1. Hi Paul –

        Any news about this library Microsoft.mshtml.dll? I am still having an error while testing the URL Login monitoring.

        Thanks!
        Marlon

    2. Tyson.Paul says:

      In regards to the “Microsoft.mshtml.dll” problem. Here’s how I solved the problem in my lab. I suggest you test this in a lab environment first.
      Overview:

      Locate “Microsoft.mshtml.dll ” on a server where the script is working as expected.
      Use PowerShell to copy the DLL; create a copy on your problem server.
      Use Gacutil.exe to reinstall the assembly on the problem server.

      Finding the DLL was a little weird because you can’t use File Explorer to navigate directly to the folder for some reason. I finally located the Microsoft.mshtml.dll on my “good” Win2012R2 server (where the script was working as expected). I had to use PowerShell to find the DLL and then to copy it to another location (C:\temp). Once I could see the DLL file, I copied it to my test server (also Win2012R2). I also grabbed a copy of Gacutil.exe from my workstation located at this path on my Win8.1 workstation at the time:
      C:\Program Files (x86)\Microsoft SDKs\Windows\v8.1A\bin\NETFX 4.5.1 Tools\x64 .
      I then copied Gacutil.exe to my test server and ran this command in PowerShell to reinstall the DLL:
      .\gacutil.exe /i .\Microsoft.mshtml.dll

      This solved the problem for me. Test this in your lab before production.

  48. Someone says:

    Scripted IE login portion does not work. The first URL will work but all others fail. Deleting MP.

    1. Tyson.Paul says:

      @Someone,
      The IE Login monitor DOES work for most sites when configured correctly. If you are trying to monitor more than one URL with a single monitor (a multi-URL sequence like: http://www.web1.com^www.web2.com ) this is not supported at this time so ya, I would expect the others to fail. The other “scripted” monitor type will monitor a multi-URL sequence. I’ve recently made significant improvements to the “scripted” type to handle N number of URLs. I plan to make this available sometime in the next few months after further enhancements and testing. But as for the IE Login type, it’s not perfect as noted in the MP guide which I recommend reading:

      “This type of monitor is experimental. I’ve had great success in my lab environment and in a handful of production environments but this monitor type is not perfect. It simulates user interaction with the web browser. Any interactive prompts resulting from unpredictable configurations will produce unreliable results.”

      PS. I’ll be sure to issue a full refund. :-/

  49. Hi there Tyson,

    Thanx for this MP. I am trying to get form login working for the same webpage from your tutorial: https://login.live.com. The UsernameElementID, PasswordElementID and ClickButtonID is still the same. However I get this message:

    There is a problem with the site/login process. See message below.

    Monitor Status: Success
    Response Time: 12944

    Monitor Message:

    Response time [ 12944 ] exceeded threshold: [120 seconds.]

    ..while attempting to fetch: [https://login.live.com/]
    ____
    InnerText:

    Navigation to the webpage was canceled

    What you can try:

    Refresh the page.Refresh the page.
    ____
    ErrorData below:

    Any ideas? I do not get any popups etc when logging in manually from the concerning watcher.

    Thanx in advance.

    Regards,
    Chris

    1. Tyson.Paul says:

      @Chris20052005,
      12944 is an unusually long response time. Did you override the ProbeActionTimeoutSeconds (Default:180)? Regardless, the scripted IE object is getting hung up on something. In order to troubleshoot this you will have to run the PowerShell script manually, providing hard-coded values in the script, including setting the visible property to $true so you can observe the IE window actions. You can export the MP and extract the script. ScriptedIELoginContentMatch.ps1

      $objIE.visible = $true;

      # set these values to something appropriate
      [int]$CaseSensitive=0
      [string]$ContentMatch=”EMPTY”
      [double]$ErrorTime=60
      [string]$ClickButtonID=”EMPTY”
      [string]$Password=”EMPTY”
      [string]$PasswordElementID=”EMPTY”
      [string]$URL=”EMPTY”
      [string]$Username=”EMPTY”
      [string]$UsernameElementID=”EMPTY”
      [int]$VerbosityLimit=8192
      [double]$WarningTime=30
      [int]$WriteToEventLog=2 #will enable logging to OpsMan event log

      Also, you may want to comment out this line shown below so that IE will remain open for you to observe. You will have to close it manually after each test.
      $objIE.QUIT()

      Also, if testing from a watcher that is not a management server (and you are not using the Ops Man Shell), you will probably encounter PowerShell errors related to the SCOM modules/objects not being present. Ignore them or comment those lines out.

      1. It appears the response time problem was caused by a language issue. The SCOM MS server and SQL server were using other number formats than US number formats (comma instead of dot for decimals and dots instead of comma for digit grouping). I just changed every language setting to US language and now it works:

        There is a problem with the site/login process. See message below.

        Monitor Status: Failure
        Response Time: 3.4905

        Monitor Message:

        Content match failure for [fgjfgijfg]!

        ..while attempting to fetch: [https://login.live.com/]
        ____
        InnerText:

        Sign in

        Use your Microsoft account.
        What’s this?

        Please enter your phone number or your email address in the format someone@example.com.

        Email or phone

        Please enter the password for your Microsoft account.

        Password

        Keep me signed in

        No account? Create one!

        Forgot my password

        Sign in with a single-use code

        Sign in with a different Microsoft account

        Terms of UsePrivacy & Cookies
        ____
        ErrorData below:

        However still no login takes place as you can see. I also executed the script manually as you described, but no errors were detected:

        PS C:\temp> .\urlgenieloginPSscript.ps1
        ResponseTimeSeconds: 3.9107
        FAILURE: Content match failure for [fgdddd] !

        Content match failure for [fgdddd]!

        ..while attempting to fetch: [https://login.live.com/]
        ____
        InnerText:

        Sign in

        Use your Microsoft account.
        What’s this?

        Please enter your phone number or your email address in the format someone@example.com.

        Email or phone

        Please enter the password for your Microsoft account.

        Password

        Keep me signed in

        No account? Create one!

        Forgot my password

        Sign in with a single-use code

        Sign in with a different Microsoft account

        Terms of UsePrivacy & Cookies
        ____
        ErrorData below:

        System.__ComObject
        Performing cleanup…
        Exiting…

        PS C:\temp>

        1. Any ideas left Tyson?

          Regards,
          Chris

        2. By the way I used these values:
          [int]$CaseSensitive=0,
          [string]$ContentMatch=”fgdddd”,
          [double]$ErrorTime=60,
          [string]$ClickButtonID=”idSIButton9″,
          [string]$OptionalElement,
          [string]$OptionalElementID,
          [string]$Password=”xxxxxxxx”,
          [string]$PasswordElementID=”i0118″,
          [string]$URL=”https://login.live.com/”,
          [string]$Username=”xxxxxx@outlook.com”,
          [string]$UsernameElementID=”i0116″,
          [int]$VerbosityLimit=8192,
          [double]$WarningTime=30,
          [int]$WriteToEventLog=0

  50. Laurent says:

    Hi Paul,

    we have a proxy problem when implementing your MP.
    we must create a user for proxy account (OK), associate this account to profile (OK) and associate this profile to proxy object (NOK)

    in step :

    Add the “KSmith Proxy Access Account” to the URLGenie Proxy Account profile and apply it to the Object:

    http://proxy.contoso.com:8080

    We haven’t proxy object defined in scom. so, how can we do?
    How to create this object??? (sorry, we are newbee in scom 🙂 ).

    Laurent

    1. Tyson.Paul says:

      @Laurent,
      Forgive me but the object is ‘http://www.somewebsite.com’ and the proxy address is just a property of the object. I have edited the comment to reflect that change. I’m sorry for the confusion. If you are still having problems with the proxy authentication then please notify me again. I will do my best to help you.

      1. Laurent says:

        Thanks for this quick reply 🙂

  51. Hi Tyson, I still have a problem with the site/login process. There is no logon happening while I am using the exact same site as you. Something might have changed with the logon mechanism, for login.live.com.

    I used the following config:
    [int]$CaseSensitive=0,
    [string]$ContentMatch=”fgdddd”,
    [double]$ErrorTime=60,
    [string]$ClickButtonID=”idSIButton9″,
    [string]$OptionalElement,
    [string]$OptionalElementID,
    [string]$Password=”xxxxxxxx”,
    [string]$PasswordElementID=”i0118″,
    [string]$URL=”https://login.live.com/”,
    [string]$Username=”xxxxxx@outlook.com”,
    [string]$UsernameElementID=”i0116″,
    [int]$VerbosityLimit=8192,
    [double]$WarningTime=30,
    [int]$WriteToEventLog=0

    However no login is happening. Check result below:

    Monitor Status: Failure
    Response Time: 3.4905

    Monitor Message:

    Content match failure for [fgjfgijfg]!

    ..while attempting to fetch: [https://login.live.com/]
    ____
    InnerText:

    Sign in

    Use your Microsoft account.
    What’s this?

    Please enter your phone number or your email address in the format someone@example.com.

    Email or phone

    Please enter the password for your Microsoft account.

    Password

    Keep me signed in

    No account? Create one!

    Forgot my password

    Sign in with a single-use code

    Sign in with a different Microsoft account

    Terms of UsePrivacy & Cookies
    ____
    ErrorData below:

    I also executed the script manually as you described, but no errors were detected:

    PS C:\temp> .\urlgenieloginPSscript.ps1
    ResponseTimeSeconds: 3.9107
    FAILURE: Content match failure for [fgdddd] !

    Content match failure for [fgdddd]!

    ..while attempting to fetch: [https://login.live.com/]
    ____
    InnerText:

    Sign in

    Use your Microsoft account.
    What’s this?

    Please enter your phone number or your email address in the format someone@example.com.

    Email or phone

    Please enter the password for your Microsoft account.

    Password

    Keep me signed in

    No account? Create one!

    Forgot my password

    Sign in with a single-use code

    Sign in with a different Microsoft account

    Terms of UsePrivacy & Cookies
    ____
    ErrorData below:

    System.__ComObject
    Performing cleanup…
    Exiting…

    Any ideas?

    Regards,
    Chris

    1. Tyson.Paul says:

      @Chris20052005,
      There could be a number of reasons why this is happening. There are many steps and prereqs required to enable the workflow to run successfully. These steps are described in the MP guide. Please make sure that you have followed the guidelines carefully. Next I would suggest logging into the watcher node (either Console or RDP) as the designated RunAs account (the account context which will run the workflow on the watcher node) and attempt to open the website with Internet Explorer. Make sure you are not presented with ANY popups that might hinder the workflow. Finally, I would cut/extract the PowerShell script from the MP and try to run it manually (again, as the RunAs account) with hardcoded values on the watcher node to see what the script is doing. This last option is a bit advanced and might not be for everyone. But do what you can.
      Please keep in mind that, as the MP guide states, this monitor type is experimental and I have had good success with it in my lab and so have many other users. It’s not perfect and you shouldn’t expect it to be perfect in every environment on every watcher node. There are bound to be some unexpected roadblocks that I can’t predict.

  52. JB says:

    Hi Paul,

    Any way to set up different recovery tasks for each URL?

    Thanks!

    1. Tyson.Paul says:

      Hi JB,
      Not easily. The recovery is for the monitor type, not per URL instance. I suppose you could write a script with a bunch of conditional statements; If (thisURL) {Then do stuff}; Else {Do other stuff}. That would be a bit of a pain. I’m curious what type of recovery you would attempt to perform on a watched URL.

      1. JB says:

        Hey Paul,

        Specifically we want to restart web services if the URL is not responding.

  53. Manoj Mathew says:

    Paul, Kudos for sharing this MP.It’s very well designed. Is there a way to customize the Alert description. Ex: Contact XXX team or Application Cluster Web Login is Down” ?

    1. Tyson.Paul says:

      @Manoj,
      Not really.

  54. Martin says:

    Hi Paul,
    Great work! Thank you!
    Testing https://login.live.com/ – what do you use for AuthenticationScheme?
    I read every page of your MP guide. I am trying your test on https://login.live.com/; however, I get the same result as Chris.
    Many thanks,
    Martin

    1. Tyson.Paul says:

      @Martin,
      You wouldn’t use “AuthenticationScheme” for a site that requires input into a username and password fields. You would have to configure the IE Scripted monitor. AuthenticationScheme is used for HTTP authentication methods (None, Basic, Digest, Negotiate, NTLM)
      Can you send me an email (address should be in the MP guide) and I can assist you directly.

      1. daniel Hobbs says:

        Hi

        great Mp

        we have a proxy in the form of XXX.XXX.XXX.XXX:80

        we seem not to be able to get out using the basic authentication setting the following

        XXX.XXX.XXX.X:80
        NTLM
        WEBMON

        WEBMON is an AD (just an example ) user and we have tested using the scom transaction monitor which was ok
        we distributed the accout to the webwatcher node Health service

        and aimed the proxy profile at the urlgenie web instance http://www.bbc.co.uk/news

        it remains in error
        do we need a domain prefix before the NTLM account

        thanks

        1. Tyson.Paul says:

          @Daniel Hobbs,
          I think your authuser value in the config file should be NetbiosDomainName\WEBMON.

          So if my domain were Contoso.com with Netbios domain name of “Contoso” this would be my authuser value:
          Contoso\WEBMON
          The RunAs account should be configured as a “Basic Authentication” account type, not Windows. The account should be added to the URLGenie Proxy Profile for the URL instance/object.
          If you continue to have problems, send me email. My address can be found in the MP guide.

          -T

  55. Martin Ehrnst says:

    Hi Tyson. Thank you for this great addition to SCOM.
    After using it for a while, were moving to SCOM 2016, can you confirm that URLGenie will work without any issues, or should we do some extra testing on it?

    1. Tyson.Paul says:

      @Martin,
      I just tested with my latest build (successfully) however it should work with the version that you have.

      1. Martin Ehrnst says:

        Thanks, i will try 🙂

  56. Wazik.Radim says:

    Hi Paul
    I need to monitor the availability SharePoint from internet. We have a two-tier authentication – with certificate and user login. The first step after entering the url is checked user certificate, then the user login using MS TMG and finally the welcome page of the SharePoint Portal. This can be monitored by your MP?

    1. Tyson.Paul says:

      @ Wazik: That’s a tall order. I’ve never tested a scenario as you have described. I think it is unlikely.

  57. Wazik.Radim says:

    Thanks for the reply, Paul!

  58. Alex.McNabb says:

    Hi Paul,

    I’ve set up this MP in our environment because of the client certificate capabilities, but I’m running into some issues. I’ve followed the guide, but when I create an XML file with the correct certthumbprint and place it in our URLGenie folder, it doesn’t create a URLGenie.HttpsRequest monitor. Instead it just makes a regular URLGenie.HttpRequest monitor even though the URL is https://server.domain.net:1234/index.html. I haven’t changed any of the other default settings so I’m wondering if there’s a step I’m missing along the way.

    Any advice would be appreciated.

    1. Tyson.Paul says:

      @Alex.McNabb:
      Hi McNabb, the httpS discovery relies on the httP discovery to run first. Therefore it takes a little longer to eventually discover the httpS instance. If you have populated the CertThumbprint element in the config file then the instance should eventually get added to the URLGenie.URL.ClientCert.Group group after group calc runs on the mgmt server(s). Once a member of that group, all normal http monitoring will become disabled/inactive. You will then have to manually override the client cert monitor (URLGenie.URLGrabScripted.ClientCert.Monitor) for the instance to enable it. Please see the tutorial, “Enable Monitoring a URL Instance with a Client Certificate” in the MP guide. If you continue to have trouble after reading the guide, please contact me directly. My email address is contained in the MP guide.

  59. Anonymous says:

    Hi Paul,

    Do you have a release date for the next version of this MP?
    I’m looking forward to the retry count feature that you mentioned a couple of months ago.

    Kind regards,
    J.

    1. Tyson.Paul says:

      @Anonymous,
      The MP is basically finished however I am still waiting for a few kind people who are currently testing it. Then there is the task of documenting it which I have yet to do. Honestly, documenting it is the worst part; soooo tedious and time consuming. Every single line has to be true, correct, and accurate to ensure that every user gets the most out of the product without wasting precious time. I can, however, provide you with a few of the things that have been changed:

      Retry now works correctly.

      The Scripted Clientcert monitor type has been removed and the client cert functionality has been integrated into the basic Scripted monitor type. I didn’t see any reason to have two separate monitors when I could bake all of the goodness into one. This “one” scripted type now supports multiple (unlimited) URL (and criteria) instances for testing sequences. You simply configure multiple criteria with the carrot. Here is a snip from the documentation:

      This scripted datasource is designed to parse multiple values from the criteria parameters
      based on this caret delimiter [^] in order to test a sequence of URLs (1 or more) with individual test criteria.
      If Multiple URLs are submitted (Example: http://www.abc.com^www.xyz.com^www.123.com )
      If only one value is submitted for a particular test criteria parameter (Example: ContentMatch [website,online] ),
      then “website” and “online” will be used as the ContentMatch test keywords for ALL URLs and is considered to be a single criteria value even though it consists of two keywords (comma separated)
      If more than one value is submitted for a criteria (but less than the total number of URLs)
      (Example: ContentMatch [fishing^bowling] ), this is considered two values which is more than 1 but less than the total number of URLs and will result in a monitor failure and script exit.
      This is so you can submit one parameter to act as a global setting for all URLs OR be explicit with each URL by submitting an equal number of values for the number of URLs.
      In the example (for 3 URLs) above you would have to submit either one single value like the following: [contact us,login] or [login] or [contact us]
      or you would have to submit 3 separate values separated by the caret: [fishing,pole^finance^shirt,pants] or [pickle^plum^bread].
      You separate multiple string keywords with a comma. Example: [fishing,pole] will test a page for both “fishing” and “pole” but is considered to be a single parameter for the content match.

      Example:
      StatusThreshold=200
      If number of URLs = 4, number of StatusThreshold values = 1, The StatusThreshold test will use that one value for all 4 URLs.

      Example2:
      StatusThreshold=200^200^504^403
      If number of URLs = 4, number of StatusThreshold values = 4, The StatusThreshold test will use the individual values for each individual corresponding URL.
      If number of URLs = 4, number of StatusThreshold values = 2 or 3, This will result in failure and script exit.

      Some benefits of using this monitor type are: ignore/trust SSL cert (server side), use of client cert, uses Invoke-WebRequest cmdlet instead of the OotB “Microsoft.SystemCenter.WebApplication.UrlProbe” that the other workflows would normally use, Retry works, basic and verbose event logging. Supports these criteria: request body, case-sensitivity, client cert, server cert ingore, checking content match of http response error data, content match, response time, retry, response status code, agent, and verb.

      Significant improvement to the IEScriptedLogin monitor type. Now will identify and populate user/pass elements by ID or Name (instead of just ID). Better IE process termination. Configurable flow wait (useful after some slow form submissions). Improved error handling. Support for logout URL navigation. Improved error handling and logging. This monitor type is painfully difficult to orchestrate. It’s not perfect but it’s significantly better than the previous version.

      I’ll continue to chip away at the documentation as time permits. Thanks for your patience.

  60. Antonio says:

    Great job Tyson: congrats and thanks for sharing!

  61. Brad McKenna says:

    Tyson, I have a question that I may have missed being answered in this post/comments. However how can I prevent duplicate notifications, or rather only send (or raise) an alert if one of the watcher nodes fail? The goal is simply to ensure on a failure that the ‘end user’ for the alert only receives one notification, instead of one for each watcher node that reported the failure.

    For our setup, if we have the Requests xml on a server that two nodes access, and the configuration for the websites is via the (.) period, to ensure all watcher nodes in the event we add/remove watcher nodes that this is easily picked up.

    I have a feel there is something simple that I am missing, but help from you or anyone else would be greatly appreciated.

    Thank you!

    1. Tyson.Paul says:

      In this situation the URLs, although they look similar, are actually two different instances (one owned/hosted by each watcher). Therefore there is not a good way to avoid receiving duplicate alerts. If the MP were redesigned to use unhosted classes, then the mgmt servers would share them and this would solve your specific problem. However, that would introduce different drawbacks.

Skip to main content