SCOM Enhanced Email Notification Script Version 2.1

In September of 2010 Tao Yang wrote a fantastic Powershell script (v2.0 here) for command channel notifications in System Center Operations Manager. Here is my adaptation of the script.

NOTE: I've tested this thoroughly in my lab and I suggest you do the same before using in any other environments. This script is meant to be an example and therefore use it at your own risk. Tested with SCOM 2012, 2012 SP1, 2012 R2, 2016.

Download current version from Technet here: Download

Version History

  • Update!   2017.10.10: v2.3.62  Fixed resend loop on "Closed" alert resolution state.
  • 2017.9.28:v2.3.61Added support for execution policy bypass from command line. This is an alternative to modifying the global execution policy.  
  • 2017.8.30: v2.3.6 Improved network device details for CustomField9.
  • 2016.5.17:  v2.3.5  Fixed muli-subscriber failure problem. Delayed alert customfield updates until the very end of the script due to stale data issue (event ID 33333: "Request to update alert ignored due to invalid TimeModified"). Cleaned up a few unnecessary lines of code.
  • 2015.11.5: v2.3  Added configuration file per Tao's suggestion. Thanks, Tao!
  • 2015.7.12: v2.2.5  Added mail send logging.
  • 2015.5.14: v2.2.4  Added LastModified field. The script adds the "LastModified" time to the email body. However, the script still takes a few more seconds to finish and right before it finishes, it updates the alert status to "Notified" thereby modifying the alert. If you look in the Console at the LastModified time, it won’t precisely match what is shown in the email notification; the Console will show a timestamp that is slightly more recent. 
  • 2015.5.14: v2.2.3 Removed a few unnecessary lines/commands that could potentially cause script errors and/or failure
  • 2015.5.3: v2.2.2 Removed older style SCOM cmdlets, only 2012R2+ supported now.
  • 2015.5.1: v2.2.1 Added support to include Parent name (IP address) and 'sysName' properties for network devices into CustomField9 of alert, also included in notification. Added Powershell code snippet to documentation for setting registry value to increase asynchronous process limit.
  • 2015.3.12: Changed how the PSModulePath gets set.
  • 2015.1.30: Added additional detail to instructions section.
  • 2015.1.8: Added Subscriber schedule validation AND Subscriber Address schedule validation logic. Cleaned up a few minor typos.
  • 2014.12.2: Fixed a typo in the error logging output message.
  • 2014.10.17: Updated so that email body would contain customer field 4 (CF4) before sending the email for "New" alerts.
  • 2014.9.16: Update the subject line format.
  • 2014.8.19: Updated documentation with notes on email subscriptions overrides.  

Email Critical Alert Format Example:

From:  SCOM-Alert [mailto:NOREPLY@contoso.com]
Sent:  Thursday, October 8, 2015 10:49 AM
To:  Tyson Paul <Tyson@contoso.com>
Subject:  New, TEST:MS02\https://gunbot.net/ammo/rimfire/22short/, HTTP Request Error: Content Validation Error

Severity: Critical Error

Severity: Critical Error
Alert Description: Content Validation Error Monitor Settings: URL: https://gunbot.net/ammo/rimfire/22short/ ContentMatch String: Pickle Rick! GroupID: URLGenie_Default DNSResolutionTime: 0.1516305 Interval: 300 RetryCount: 1 Wiki: https://blogs.msdn.com/b/tysonpaul/archive/2015/05/04/urlgenie-management-pack-for-scom-an-easy-solution-for-bulk-website-monitoring.aspx Description: Gunbot.net, Near realtime tracking of who has ammo, mags and reloading supplies in stock.

******* Request Headers ******* GET /ammo/rimfire/22short/ HTTP/1.1 User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; Windows NT 6.1; en-US) Content-Type: text/xml;charset=utf-8 Accept-Language: en-US Accept-Charset: utf-8 From: SCOM@yourdomain.com Connection: Keep-Alive

******* End Request Headers *******

******* Response Headers ******* ResponseHeaders: HTTP/1.1 200 OK Connection: close Date: Thu, 08 Oct 2015 17:46:12 GMT Content-Length: 4842 Content-Type: text/html; charset=UTF-8 Server: Apache/2.2.15 (CentOS) X-Powered-By: PHP/5.3.3

******* End Response Headers *******

Command Channel: MyDev
Source: https://gunbot.net/ammo/rimfire/22short/
Path: MS02.Contoso.com;\\Db01.contoso.com\scom_do_not_touch\URLGenie
Principal Name: MS02.Contoso.com
Alert Name: HTTP Request Error: Content Validation Error
Alert Resolution State: New (0)
Alert Monitor Name: URLGenie Content Monitor
Alert Monitor Description: None
Time Raised: 10/08/2015 10:45:42
Last Modified: 10/08/2015 10:48:05
Alert ID: b88f30ec-b008-4fc5-b785-a19cc3646238

SCOM Operations Console Info:

Operations Console Login Info
Web Console

Research It:

Bing It!
ADMIN DIAGNOSTIC INFO --------------------
**Use this command to view the full details of this alert in SCOM Powershell console: Get-SCOMalert -Id "b88f30ec-b008-4fc5-b785-a19cc3646238" | format-list *
This email sent from: MS01.Contoso.com
CF1: Alert.NetBIOSName: MS02
CF2: Alert.NetBIOSDomain Name: CONTOSO
CF3: Alert.PrincipalName: MS02.Contoso.com
CF4: SubscriberList: Subscribers:Subscriber1;
CF5: Management Pack Name: URLGenie Management Pack
CF6: Alert Class Name: URLGenie.HttpRequest
CF7: Alert.Category AvailabilityHealth
CF8: Workflow Type MONITOR
CF9: RESERVED
CF10: This Email Generated From C:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification_v2.2.4.ps1
Context:  Note: This context data is only relevant to the moment/time at which this alert was sent. type : Microsoft.SystemCenter.WebApplication.WebApplicationDatatime : 2015-10-08T10:45:42.0960698-07:00sourceHealthServiceId : 120875E8-C79E-91CA-E3AC-EA09A391F4BDRequestResults : RequestResultsTransactionResponseTime : 0.270948TransactionResponseTimeEvalResult : 0CollectPerformanceData : CollectPerformanceData

 

Knowledge Article:


 

Summary

The context information for these alerts is not always helpful. To see more detailed information about this alert log into the console for the management group.

Wiki


Note:

The context information provided in this notification is limited and is not always helpful. To see more detailed information about this alert, log into the appropriate SCOM console for the applicable SCOM management group and use the Health Explorer to find more details about the state change event(s) for the object.
 

 

Closed Alert Example

From:  SCOM-Alert [mailto:NOREPLY@contoso.com]
Sent:  Thursday, October 8, 2015 3:32 PM
To:  Tyson Paul <Tyson@contoso.com>
Subject:  Closed, TEST:MS02\https://gunbot.net/ammo/rimfire/22short/, HTTP Request Error: Content Validation Error

Severity: Critical Error

Severity: Critical Error
Alert Description: Content Validation Error Monitor Settings: URL: https://gunbot.net/ammo/rimfire/22short/ ContentMatch String: Pickle Rick! GroupID: URLGenie_Default DNSResolutionTime: 0.1516305 Interval: 300 RetryCount: 1 Wiki: https://blogs.msdn.com/b/tysonpaul/archive/2015/05/04/urlgenie-management-pack-for-scom-an-easy-solution-for-bulk-website-monitoring.aspx Description: Gunbot.net, Near realtime tracking of who has ammo, mags and reloading supplies in stock.

******* Request Headers ******* GET /ammo/rimfire/22short/ HTTP/1.1 User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; Windows NT 6.1; en-US) Content-Type: text/xml;charset=utf-8 Accept-Language: en-US Accept-Charset: utf-8 From: SCOM@yourdomain.com Connection: Keep-Alive

******* End Request Headers *******

******* Response Headers ******* ResponseHeaders: HTTP/1.1 200 OK Connection: close Date: Thu, 08 Oct 2015 17:46:12 GMT Content-Length: 4842 Content-Type: text/html; charset=UTF-8 Server: Apache/2.2.15 (CentOS) X-Powered-By: PHP/5.3.3

******* End Response Headers *******

Command Channel: MyDev
Source: https://gunbot.net/ammo/rimfire/22short/
Path: MS02.Contoso.com;\\Db01.contoso.com\scom_do_not_touch\URLGenie
Principal Name: MS02.Contoso.com
Alert Name: HTTP Request Error: Content Validation Error
Alert Resolution State: Closed (255)
Alert Monitor Name: URLGenie Content Monitor
Alert Monitor Description: None
Time Raised: 10/08/2015 10:45:42
Last Modified: 10/08/2015 15:30:32
Alert ID: b88f30ec-b008-4fc5-b785-a19cc3646238

SCOM Operations Console Info:

Operations Console Login Info
Web Console

Research It:

Bing It!
ADMIN DIAGNOSTIC INFO --------------------
**Use this command to view the full details of this alert in SCOM Powershell console: Get-SCOMalert -Id "b88f30ec-b008-4fc5-b785-a19cc3646238" | format-list *
This email sent from: MS01.Contoso.com
CF1: Alert.NetBIOSName: MS02
CF2: Alert.NetBIOSDomain Name: CONTOSO
CF3: Alert.PrincipalName: MS02.Contoso.com
CF4: SubscriberList: Subscribers:Subscriber1;
CF5: Management Pack Name: URLGenie Management Pack
CF6: Alert Class Name: URLGenie.HttpRequest
CF7: Alert.Category AvailabilityHealth
CF8: Workflow Type MONITOR
CF9: RESERVED
CF10: This Email Generated From C:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification_v2.2.4.ps1
Context:  Note: This context data is only relevant to the moment/time at which this alert was sent. type : Microsoft.SystemCenter.WebApplication.WebAp plicationData time : 2015-10-08T15:30:32.3385560-07:00 sourceHealthServiceId : 120875E8-C79E-91CA-E3AC-EA09A391F4BD RequestResults : RequestResults TransactionResponseTime : 0.1240154 TransactionResponseTimeEvalResult : 0 CollectPerformanceData : CollectPerformanceData

 

Knowledge Article:


 

Summary

The context information for these alerts is not always helpful. To see more detailed information about this alert log into the console for the management group.

Wiki


Note:

The context information provided in this notification is limited and is not always helpful. To see more detailed information about this alert, log into the appropriate SCOM console for the applicable SCOM management group and use the Health Explorer to find more details about the state change event(s) for the object.

 

Directions:

In the script there are instructions at the top that describe a few of the settings (lines of code) that must be configured before use. Simply edit the "Settings" area within the script. Use Notepad or your favorite text editor. I like Powershell ISE or Notepad++ .
Configuration is very simple and there are examples for each setting within the script comments at the top.

 

Here's what I recommend:

PERFORM THESE TEST PROCEDURES IN A LAB ENVIRONMENT (NON PRODUCTION):

Remove all management servers from your Notifications Resource Pool except for one single server. This will help you troubleshoot during testing if needed.

Create this folder on your test mgmt server: "C:\SCOM_SCRIPTS\Notifications"

Put the script into that new folder.

Open the Config.xml file and customize the necessary fields: smtp server, port, smtp sender, web console url, etc.  See the example files also in the folder. This Config.xml file must be located in the same directory as the .SCOMEnhancedEmailNotification.ps1 script. I suggest you put these files in this path on your management servers:   C:\SCOM_SCRIPTS\Notifications

Make sure that your default action account is a local administrator on your test mgmt server.

Make sure that your Powershell execution policy is not blocking the command channel execution.

  • View policy: Get-ExecutionPolicy 
  • Or just run the following to be sure the policy is "unrestricted". In an elevated Powershell console on the mgmt server, run this command:
    Set-ExecutionPolicy Unrestricted
  • Note: a reboot may be necessary after modifying the execution policy.

 

Create the command channel:

Field  Parameter Notes
Full path of the command file: %SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe Standard path to Powershell.exe (all versions)
Command line parameters: "C:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification.ps1" -Description 'MyDev' -alertID '$Data/Context/DataItem/AlertId$' -SubscriptionID '$MPElement$'   * IF additional Ops Manager event log details are desired,  use the  -Verbose switch in the command line.
Startup folder for the command line: C:\SCOM_SCRIPTS\Notifications

NOTE: If you don't want to set the global execution policy to Unrestricted then consider adding the following to the beginning of your Command line parameters:
-ExecutionPolicy Bypass -File

Example:   -ExecutionPolicy Bypass -File " C:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification.ps1" -Description 'MyDev' -alertID '$Data/Context/DataItem/AlertId$' -SubscriptionID '$MPElement$'  

Note: When using -ExecutionPolicy parameter notice the double quotes in the command above!  If you use single quotes around the file name instead of double quotes your command channel won't trigger and you won't get any logging or events in the event log and you will lose hours troubleshooting this (like I did).  CommandChannel1

 

 

Subscribers:

  • Create a Test Subscriber with your email address. You can configure multiple subscribers, each with one or more recipients. The script will send to To, CC, and BCC. Notice the "TO" and "CC" preceding the addresses. If "TO", "CC", or "BCC" are omitted, the default becomes "TO".
    Note: the email address is in the "Name" field. With other channels the "Name" field is used for a friendly/display name. With the Command channel you use the actual email address here.

Subscriptions:

 

 

  • Pick one or more "Subscribers"

 

 

New! Added support for schedules on Subscriber and individual Addresses.

Note: ANY Exclusions will take precedence over any Inclusions.

Testing:

  • Generate a test alert with this command from regular command prompt (not Powershell console), paste the following into command window:EventCreate /T ERROR /ID 101 /L APPLICATION /SO TEST /D "This is a synthetic transaction test only. Disregard this event. Note: This alert was generated by entering the below command into a command pompt on the server in question:> EventCreate /T ERROR /ID 101 /L APPLICATION /SO TEST /D

 

  • Watch Task Manager on the test mgmt server, look for a Powershell.exe process to spin up. Usually the process will take 10 - 20 seconds to send the mail.
  • You may want to adjust your max asynchronous process limit for Powershell as described here:
    https://www.scomnifties.com/powershell-script-to-set-asynchronous-process-limit-on-management-servers/
  • You should receive the email quickly if you configured everything correctly.
  • Copy the script folder to your other management servers that will be in the Notifications Resource Pool. It must be in the same path on each mgmt. server.
  • Put each management server alone in the Notifications Resource Pool, one at a time,  and test each with the Synthetic Transaction procedure described above.

Troubleshooting:

If the script doesn't appear to be running, open Task Manager, sort by Name, and watch for a new Powershell.exe process to spin up under the Management Server Action Account context/user . The script takes from 10-20 seconds to complete. If the new Powershell.exe process doesn't open within 60 seconds or opens briefly and then disappears, then you have a problem.

Make sure the script itself is not being blocked from running. Right-click the script, select Properties. If you see a "Unblock" button, click it to make sure the script is not being blocked from execution.

The Powershell execution policy may be Restricted. Change this to Unrestricted. Set-ExecutionPolicy Unrestricted

There may be an error with the Channel configuration or path(s). Double check the path to the .ps1 script. Make sure the full file path is precise. I suggest navigating to the actual path where the script lives and copying the full path right from the File Explorer address/path field. This way there's no chance of misspelling the path to the script. See command line parameter field example above.

Check the Notifications folder for any errors that might appear in the error log. Use the -Verbose and -WriteToEventLog command line parameter switches for additional Ops Manager event logging detail.

If needed, you can manually run the script. You must first set the required variables near the top of the script.

 ############### UNCOMMENT FOR MANUAL TESTING ############
 # $Description = "ManualTest"
 # $alertID = (Get-SCOMAlert)[0].ID.Guid

# Create a test subscription with "Synthetic" in the name
# $SubscriptionID = (Get-SCOMNotificationSubscription -DisplayName *Synthetic* ).ID.Guid

# $Verbose = $true
################## FOR TESTING ##########################

An additional method to gain visibility into any errors that might occur (before the script has a chance to write error output to the event log) is to append the line shown directly below at the end of the command line parameters:

  ; Write-Output "$($Error)" |  Out-File -Filepath C:\SCOM_SCRIPTS\Notification_Errors.txt

Example:

"C:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification.ps1" -Description "MyDev" -alertID '$Data/Context/DataItem/AlertId$' -SubscriptionID '$MPElement$'  -ErrorLogFile "C:\SCOM_SCRIPTS\Errors.log" -Verbose -WriteToEventLog  ; Write-Output "$($Error)" |  Out-File -Filepath C:\SCOM_SCRIPTS\Notification_Errors.txt

This will output any errors that occur when the workflow attempts to launch the command channel.

Also, I've seen some weird things happen when copying/pasting from online sources into Powershell scripts. Make sure you are not copying some weirdly encoded characters. I'm pretty sure this blog doesn't use any strange encoding but to keep your sanity, copy the code portions into Notepad++ and force the encoding to be UTF-8, then copy from there into the Console wizard.

*Please let me know if you have any suggestions to improve this article.

 

 

Page History:

  • 2015/11/5: Updated the email examples. Updated the configuration instructions.
  • 2015/5/14: Cleaned up some of the text. Added script version history near top.
  • 2014/9/16: Updated script. Improved notification subject line format.
  • 2014/10/15: Updated command channel arguments examples, removed some quotes
  • 2014/10/17: Updated troubleshooting section. Also updated the script to add the Subscribers to the CF4 field on "New" alerts. Previously subscribers would be empty for new alerts.
  • 2014/10/27: Updated troubleshooting section.
  • 2014/10/29: Updated some sloppy grammar.

 

Disclaimer: Sample scripts in this guide are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.