[PowerShell Script] Displaying the Top 5 Exceptions and Up to 5 Different Call Stacks For Each Exception

 

During a lab I had about a month ago, one of my colleagues, Tag, whom I had opportunity to mention in a previous post, asked me if I could create a PowerShell script that displays statistics of exceptions, with call stacks included.

For me it sounded like a neat idea because sometimes !DumpAllExceptions is not enough. Let me explain, when using this command you get all exceptions in the managed heap and the number of times they appeared. You also may get one call stack for each exception. That’s fine, but sometimes you may want to get statistics of exceptions after a specific period of time. Or you may want to track a few specific exceptions and see all the call stacks they have.

This is what this script does: it shows you all the exceptions sorted by number of times and up to 5 different call stacks for the top 5 exceptions!

To use it you may use ADPlus or WinDbg to create a log file and log all exceptions or only the exceptions you want, for instance:

 .logopen <fileName>

sxe -c "!PrintException;!clrstack;gc" dz

Or:

.logopen <fileName>

sxe -c "!PrintException;!clrstack;gc" clr

Or:

.logopen <fileName>

sxe -c "!PrintException;!clrstack;gc" av

Or any other similar expression.

You can even use something like:

.logopen <fileName>

sxe -c ".echo ===================;!PrintException;!clrstack;gc" av

You’ve got the idea. You have to save the exceptions ( !PrintException) and the CLR stack ( !clrstack) into a log file.

After doing this, you’ll have a log file that may be very large.

Then the fun starts! The log file should be used as an argument for the script below, called ExceptionStats.PS1. Run the script below, get the statistics, stacks, top exceptions, and then send an e-mail to the development team based on this information. You’ll probably spend more time writing the e-mail than getting the exception statistics with this script. J

Note: This is a standalone script; it doesn’t require PowerDbg because it doesn’t interact with WinDbg.

This is a sample from the log file. Yours should be similar, although not identical, to mine:

Opened log file 'c:\dumps\CLRExceptionsDuringLaunch.log'

0:000> sxe ld:mscorwks.dll

0:000> lm

start end module name

11000000 1100a000 Startup (deferred)

79000000 79046000 mscoree (deferred)

7c800000 7c8f4000 KERNEL32 (deferred)

7c900000 7c9b0000 ntdll (private pdb symbols) C:\Symbols\SYMBOLS-LAB\ntdll.pdb\36515FB5D04345E491F672FA2E2878C02\ntdll.pdb

0:000> sxe clr

0:000> g

ModLoad: 79e70000 7a3ff000 C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll

eax=00000000 ebx=00000000 ecx=008a0000 edx=7c90eb94 esi=00000000 edi=00000000

eip=7c90eb94 esp=0012f1bc ebp=0012f2b0 iopl=0 nv up ei ng nz ac pe nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000296

ntdll!KiFastSystemCallRet:

7c90eb94 c3 ret

0:000> .load sos

0:000> sxe -c ".echo;.echo;.echo ========================================;.echo ACCESS VIOLATION;!clrstack" AV

0:000> sxe -c ".echo;.echo;.echo ========================================;!pe;!clrstack" CLR

0:000> g

(cd0.440): C++ EH exception - code e06d7363 (first chance)

(cd0.440): C++ EH exception - code e06d7363 (first chance)

(cd0.440): C++ EH exception - code e06d7363 (first chance)

(cd0.440): CLR exception - code e0434f4d (first chance)

sxe -c ".echo ========================================;!PrintException;!clrstack" CLR

################################################################################

sxe -c ".echo ===============;!PrintException;!clrstack;gc" clr

sxe -c ".echo ===============;!PrintException;!clrstack;gc" dz

################################################################################

========================================

Exception object: 013402a0

Exception type: System.IO.FileNotFoundException

Message: Could not load file or assembly 'Config.XmlSerializers, Version=0.0.5.1, Culture=neutral, PublicKeyToken=777ff38dbd9452b8' or one of its dependencies. The system cannot find the file specified.

InnerException: System.IO.FileNotFoundException, use !PrintException 01340d34 to see more

StackTrace (generated):

<none>

StackTraceString: <none>

HResult: 80070002

OS Thread Id: 0x440 (0)

ESP EIP

0012d030 7c81eb33 [HelperMethodFrame_PROTECTOBJ: 0012d030] System.Reflection.Assembly._nLoad(System.Reflection.AssemblyName, System.String, System.Security.Policy.Evidence, System.Reflection.Assembly, System.Threading.StackCrawlMark ByRef, Boolean, Boolean)

0012d0c8 7937dd77 System.Reflection.Assembly.nLoad(System.Reflection.AssemblyName, System.String, System.Security.Policy.Evidence, System.Reflection.Assembly, System.Threading.StackCrawlMark ByRef, Boolean, Boolean)

0012d0f0 7937dbe8 System.Reflection.Assembly.InternalLoad(System.Reflection.AssemblyName, System.Security.Policy.Evidence, System.Threading.StackCrawlMark ByRef, Boolean)

0012d118 793af4f6 System.Reflection.Assembly.Load(System.Reflection.AssemblyName)

0012d120 638fa948 System.Xml.Serialization.TempAssembly.LoadGeneratedAssembly(System.Type, System.String, System.Xml.Serialization.XmlSerializerImplementation ByRef)

0012d160 63960ae1 System.Xml.Serialization.XmlSerializer..ctor(System.Type, System.String)

0012d1a4 639609b7 System.Xml.Serialization.XmlSerializer..ctor(System.Type)

0012d1a8 00f6fa5f PSWGS.Common.Client.Framework.Configuration.Config.RetrieveConfigData()

0012d1e4 00f6f6ba PSWGS.Common.Client.Framework.Configuration.Config.GetConcreteConfigObject(System.Type)

0012d24c 00f6f529 PSWGS.Common.Client.Framework.Configuration.ConfigManager.RetrieveConfigObject(System.Type)

0012d25c 00f6f473 PSWGS.Common.Client.Framework.Configuration.ConfigManager.get_NetworkConnectivityConfig()

0012d264 00f6e0d7 PSWGS.Common.Client.Framework.SalFramework.EndPortStrategy.Initialize(System.Xml.XPath.XPathNavigator)

0012d294 00f6dae9 Microsoft.ApplicationBlocks.Common.ProviderManagerBase.GetProvider(System.String, System.String)

0012d2e8 00f6d89d Microsoft.ApplicationBlocks.SmartClient.Offline.ConnectionManagerConfigurator.CreateDetectionStrategy()

0012d2fc 00f6d83c Microsoft.ApplicationBlocks.SmartClient.Offline.ConnectionManagerConfigurator.get_DetectionStrategy()

0012d308 00f6d763 Microsoft.ApplicationBlocks.SmartClient.Offline.ConnectionManagerBuilder..ctor()

0012d324 00f6d60a PSWGS.Common.Client.Framework.SalFramework.SalFrameworkBuilder..ctor()

0012d338 00f6d598 PSWGS.Common.Client.Framework.SalFramework.SalFrameworkBuilder.get_Instance()

0012d364 00f6d4c6 PSWGS.Common.Client.Framework.SalFramework.CommonServiceAgent..cctor()

0012d6ec 79e7c74b [GCFrame: 0012d6ec]

0012dc54 79e7c74b [PrestubMethodFrame: 0012dc54] PSWGS.Common.Client.Framework.SalFramework.CommonServiceAgent..ctor()

0012dc64 00f6d47d PSWGS.Mobile.Mdt.Provisioning.ProvisioningManager..ctor()

0012dc74 00f6d399 PSWGS.Mobile.Mdt.IncidentManagement.IncidentManager..ctor()

0012dc84 00f6d2dc PSWGS.Mobile.Mdt.IncidentManagement.WorkingThread..ctor()

0012dc90 00f6d29d PSWGS.Mobile.Mdt.IncidentManagement.WorkingThread..cctor()

0012e014 79e7c74b [GCFrame: 0012e014]

0012e544 79e7c74b [GCFrame: 0012e544]

0012f478 79e7c74b [PrestubMethodFrame: 0012f478] PSWGS.Mobile.Mdt.Startup.Main()

0012f69c 79e7c74b [GCFrame: 0012f69c]

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

eax=0012ce8c ebx=e0434f4d ecx=00000000 edx=00000029 esi=0012cf18 edi=00180d80

eip=7c81eb33 esp=0012ce88 ebp=0012cedc iopl=0 nv up ei pl nz na po nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202

KERNEL32!RaiseException+0x53:

7c81eb33 5e pop esi

0:000> sxe -c ".echo;.echo;.echo ========================================;!pe;!clrstack;g" CLR

0:000> sxe -c ".echo;.echo;.echo ========================================;.echo ACCESS VIOLATION;!clrstack;g" AV

0:000> g

(cd0.440): C++ EH exception - code e06d7363 (first chance)

(cd0.440): C++ EH exception - code e06d7363 (first chance)

(cd0.440): C++ EH exception - code e06d7363 (first chance)

(cd0.440): CLR exception - code e0434f4d (first chance)

========================================

Exception object: 013a8188

Exception type: System.IO.FileNotFoundException

Message: Could not load file or assembly 'Config.XmlSerializers, Version=0.0.5.1, Culture=neutral, PublicKeyToken=777ff38dbd9452b8' or one of its dependencies. The system cannot find the file specified.

InnerException: System.IO.FileNotFoundException, use !PrintException 013a8c1c to see more

StackTrace (generated):

<none>

StackTraceString: <none>

HResult: 80070002

OS Thread Id: 0x440 (0)

ESP EIP

0012d028 7c81eb33 [HelperMethodFrame_PROTECTOBJ: 0012d028] System.Reflection.Assembly._nLoad(System.Reflection.AssemblyName, System.String, System.Security.Policy.Evidence, System.Reflection.Assembly, System.Threading.StackCrawlMark ByRef, Boolean, Boolean)

0012d0c0 7937dd77 System.Reflection.Assembly.nLoad(System.Reflection.AssemblyName, System.String, System.Security.Policy.Evidence, System.Reflection.Assembly, System.Threading.StackCrawlMark ByRef, Boolean, Boolean)

0012d0e8 7937dbe8 System.Reflection.Assembly.InternalLoad(System.Reflection.AssemblyName, System.Security.Policy.Evidence, System.Threading.StackCrawlMark ByRef, Boolean)

0012d110 793af4f6 System.Reflection.Assembly.Load(System.Reflection.AssemblyName)

0012d118 638fa948 System.Xml.Serialization.TempAssembly.LoadGeneratedAssembly(System.Type, System.String, System.Xml.Serialization.XmlSerializerImplementation ByRef)

0012d158 63960ae1 System.Xml.Serialization.XmlSerializer..ctor(System.Type, System.String)

0012d19c 639609b7 System.Xml.Serialization.XmlSerializer..ctor(System.Type)

0012d1a0 00f6fa5f PSWGS.Common.Client.Framework.Configuration.Config.RetrieveConfigData()

0012d1dc 044109f6 PSWGS.Common.Client.Framework.Configuration.SystemConfig.RetrieveConfigData()

0012d1e4 00f6f6ba PSWGS.Common.Client.Framework.Configuration.Config.GetConcreteConfigObject(System.Type)

0012d24c 00f6f529 PSWGS.Common.Client.Framework.Configuration.ConfigManager.RetrieveConfigObject(System.Type)

0012d25c 044106c2 PSWGS.Common.Client.Framework.Configuration.ConfigManager.get_SystemConfig()

0012d264 00f6e0e2 PSWGS.Common.Client.Framework.SalFramework.EndPortStrategy.Initialize(System.Xml.XPath.XPathNavigator)

0012d294 00f6dae9 Microsoft.ApplicationBlocks.Common.ProviderManagerBase.GetProvider(System.String, System.String)

0012d2e8 00f6d89d Microsoft.ApplicationBlocks.SmartClient.Offline.ConnectionManagerConfigurator.CreateDetectionStrategy()

0012d2fc 00f6d83c Microsoft.ApplicationBlocks.SmartClient.Offline.ConnectionManagerConfigurator.get_DetectionStrategy()

0012d308 00f6d763 Microsoft.ApplicationBlocks.SmartClient.Offline.ConnectionManagerBuilder..ctor()

0012d324 00f6d60a PSWGS.Common.Client.Framework.SalFramework.SalFrameworkBuilder..ctor()

0012d338 00f6d598 PSWGS.Common.Client.Framework.SalFramework.SalFrameworkBuilder.get_Instance()

0012d364 00f6d4c6 PSWGS.Common.Client.Framework.SalFramework.CommonServiceAgent..cctor()

0012d6ec 79e7c74b [GCFrame: 0012d6ec]

0012dc54 79e7c74b [PrestubMethodFrame: 0012dc54] PSWGS.Common.Client.Framework.SalFramework.CommonServiceAgent..ctor()

0012dc64 00f6d47d PSWGS.Mobile.Mdt.Provisioning.ProvisioningManager..ctor()

0012dc74 00f6d399 PSWGS.Mobile.Mdt.IncidentManagement.IncidentManager..ctor()

0012dc84 00f6d2dc PSWGS.Mobile.Mdt.IncidentManagement.WorkingThread..ctor()

0012dc90 00f6d29d PSWGS.Mobile.Mdt.IncidentManagement.WorkingThread..cctor()

0012e014 79e7c74b [GCFrame: 0012e014]

0012e544 79e7c74b [GCFrame: 0012e544]

0012f478 79e7c74b [PrestubMethodFrame: 0012f478] PSWGS.Mobile.Mdt.Startup.Main()

0012f69c 79e7c74b [GCFrame: 0012f69c]

(cd0.440): C++ EH exception - code e06d7363 (first chance)

(cd0.440): C++ EH exception - code e06d7363 (first chance)

(cd0.440): C++ EH exception - code e06d7363 (first chance)

(cd0.440): CLR exception - code e0434f4d (first chance)

Screenshots:

 

 

 

 

 

 

Source code for ExceptionStats.PS1:

########################################################################################################

# Script: ExceptionsStat

#

# Parameters: <fileName> - The path and filename that has the logged exceptions. To get them you should run commands like:

# .logopen <fileName>

# sxe -c "!PrintException;!clrstack;gc" dz

# Or:

# sxe -c "!PrintException;!clrstack;gc" clr

# Or:

# sxe -c "!PrintException;!clrstack;gc" av

# Or any other similar expression.

#

# You can even use something like:

# sxe -c ".echo ===================;!PrintException;!clrstack;gc" av

#

# Purpose: Reads the file that has the output for !PrintException and !ClrStack and displays statistics related to the most often exceptions and their call stack.

#

# Usage: Before running the script you need to create a file that has exceptions and stacks. To do that use:

# .logopen <fileName>

# sxe -c "!PrintException;!clrstack;gc" dz

# Or:

# sxe -c "!PrintException;!clrstack;gc" clr

# Or:

# sxe -c "!PrintException;!clrstack;gc" av

# Or any other similar expression.

#

# You can even use something like:

# sxe -c ".echo ===================;!PrintException;!clrstack;gc" av

#

# Note! This script doesn't require the PowerDbg library because it doesn't interact with WinDbg.

#

# Changes History:

# Ayax Vargas - Added progress bar (presentation)

#

# Roberto Alexis Farah

# All my functions are provided "AS IS" with no warranties, and confer no rights.

########################################################################################################

param(

[string] $fileName = $(throw "Error! You must provide the path and file name.")

)

set-psdebug -strict

$ErrorActionPreference = "stop"

trap {"Error message: $_"}

[string] $exceptionsOutput = "exceptionstats.csv"

[string] $exceptionsSorted = "sorted.csv"

[string] $delimiter = ";"

########################################################################################################

# Function: Extract-Exception

#

# Parameters: <fileName> - The path and filename that has the logged exceptions.

#

# Return: Nothing.

#

# Purpose: Create two csv files. One has the exception name as key and the exception details and call stack as value.

# The other file has one type of each exception and the number of times it appears. It's sorted by number of times.

#

# Changes History:

#

# Roberto Alexis Farah

# All my functions are provided "AS IS" with no warranties, and confer no rights.

########################################################################################################

function Extract-Exception([string] $fileName = $(throw "Error! You must provide the path and file name."))

{

write-Host "Scanning file $fileName...`n" -foreground Green -background Black

# Extract output removing commands.

$builder = New-Object System.Text.StringBuilder

# Title for the CSV fields.

$builder = $builder.AppendLine("key,value")

$wasException = $false

$isClrstack = $false

$exceptionStatistics = @{}

# Script presentation variables:

$ExceptionsFile = Get-Content $fileName # Additional Variable to hold entire file

$exlen = $ExceptionsFile.Length # Variable to hold file length (in lines)

$i = 0 # Counter variable for % Complete Calculation

foreach($line in $ExceptionsFile)

{

$i++

# First we exctract information from !PrintException, identifiying the related patterns.

# If the exception was read we don't need to spend CPU cycles trying to identify a pattern that is not there.

if($wasException -eq $false)

{

if($line -match "((?<key>(^Exception type:)|(^Message:)|(^InnerException:)|(^HResult:))\s+(?<value>.+))")

{

#Removed period to indicate status; goal: Unclutter output

#write-Host "." -foreground Green -background Black -nonewline

# We use the Exception Type as our key for the csv file.

if($matches["key"] -eq "Exception type:")

{

$builder = $builder.Append($matches["value"] + ",")

# Add it as a key to our hash table. First check if the key was already added and retrieve its accumulator.

$accumulator = $exceptionStatistics[$matches["value"]]

# If the value is null, let's insert the key for the first time.

if($accumulator -eq $null)

{

# First time this exception appears.

$exceptionStatistics[$matches["value"]] = 1

}

else # The key is already there, so increase the accumulator.

{

# The accumulator counts the number of times the exception appeared.

$exceptionStatistics[$matches["value"]] = $accumulator + 1

}

}

else # The other fields and values from !PrintException, are going to be values in our csv file.

{

# Removed period to indicate status; goal: Unclutter output

# This is super important! We use "," as delimiters for the CSV file and ";" to separate fields. It means we cannot have

# these characters appearing in other places, so we replace them for space if they appear in the call stack.

$temp = $matches["value"].Replace(",", " ")

$temp = $temp.Replace($delimiter, " ");

$builder = $builder.Append($matches["key"] + " " + $temp + $delimiter)

}

# Indicates we were in !PrintException context. We use HResult: because it's the last field from !PrintException.

if($matches["key"] -eq "HResult:")

{

$wasException = $true

# When processing it we will replace ";" for new line, so two ";" side by side will be two new lines.

$builder = $builder.Append($delimiter)

}

}

}

else

{

# Now we identify the !clrstack. We just need the stack frames, we can ignore all other information including ESP and EIP.

if($line -match "(^\w+\s\w+\s(?<value>.+))") # Gets the stack.

{

# Removed period to indicate status; goal: Unclutter output

# write-Host "." -foreground Green -background Black -nonewline

Write-Progress -Activity "Parsing file" -status "Progress:" -PercentComplete ($i/ $exlen * 100)

# This is super important! We use "," as delimiters for the CSV file and ";" to separate fields. It means we cannot have

# these characters appearing in other places, so we replace them for space if they appear in the call stack.

$temp = $matches["value"].Replace(",", " ")

$temp = $temp.Replace($delimiter, " ")

# Just the stack from !clrstack.

$builder = $builder.Append($temp + $delimiter)

}

elseif($line -match "((^OS Thread Id:)|(^ESP EIP))" )

{

# We don't use the header from !clrstack in if() clausule because it appears less times then frames.

$isClrstack = $true

}

else # If the patterns doesn't match it means we read the !clrstack or it didn't find it yet.

{

# If the flag is true it means we read all information from !clrstack. Otherwise it means we read garbage.

if($isClrstack -eq $true)

{

$wasException = $false # Okay, now we can read !PrintException again.

$isClrstack = $false # It's not !clrstack anymore.

# Inserts new line.

$builder = $builder.AppendLine("")

}

}

}

}

# Save a csv file with all exceptions and the number of times they appear, like: exception,number of times.

$exceptionStatistics.GetEnumerator() | select-object key, value | Sort-Object Value -descending | export-Csv -path $exceptionsSorted -noTypeInformation

# Send output to a file.

out-file -filepath $exceptionsOutput -inputobject "$builder"

$builder = $null

write-Host "`nAll exceptions were extracted from the input file!`n" -foreground Green -background Black

}

########################################################################################################

# Function: Display-ExceptionStatisticsSummary

#

# Parameters: None.

#

# Return: Nothing.

#

# Purpose: Display the exceptions by number of times, sorted by number of times.

#

# Changes History:

#

# Roberto Alexis Farah

# All my functions are provided "AS IS" with no warranties, and confer no rights.

########################################################################################################

function Display-ExceptionStatisticsSummary()

{

# Scans the csv file tha has the sorted exceptions. Then, for each sorted exception we go to the other csv file, that has all

# details for each exception, and display those that matches the exceptions from the first csv file.

foreach($line in $(get-Content $exceptionsSorted))

{

# Get the key and value from the csv file.

[string[]] $values = @($line.ToString().Split(","))

if($values[0] -ne "key")

{

write-Host "Exception " -foreground Green -background Black -nonewline

write-Host $values[0] -foreground Red -background Black -nonewline

write-Host " appeared " -foreground Green -background Black -nonewline

write-Host $values[1] -foreground Red -background Black -nonewline

write-Host " times.`n" -foreground Green -background Black

}

}

}

########################################################################################################

# Function: Display-ExceptionStatisticsDetails

#

# Parameters: None.

#

# Return: Nothing.

#

# Purpose: Display the top 5 exceptions by number of times and their corresponding 5 different call stacks.

# If an exceptions has different call stacks, they all will be displayed.

#

# Changes History:

#

# Roberto Alexis Farah

# All my functions are provided "AS IS" with no warranties, and confer no rights.

########################################################################################################

function Display-ExceptionStatisticsDetails()

{

# Here we create a hash table with the exceptions and the call stack.

# Whenever we find a different call stack for a same exception, we add it to the "value" part. However,

# we add just 3 different call stacks.

$hashTable = @{}

[System.Int32] $callStackCounter = 0

import-csv -path $exceptionsOutput | `

sort-object key, value -unique | `

foreach-Object{

if($hashTable[$_.key] -eq $null)

{

$hashTable[$_.key] = "### STACK 1" + $delimiter + $delimiter + $_.value

[System.Int32] $callStackCounter = 1

}

elseif($callStackCounter -lt 5) # No more than 5 stacks for the same exception.

{

$callStackCounter += 1

$temp = $hashTable[$_.key]

$hashTable[$_.key] = $temp + $delimiter + $delimiter + "### STACK $callStackCounter" + $delimiter + $delimiter + $_.value

}

}

# Scans the csv file tha has the sorted exceptions. Then, for each sorted exception we go to the other csv file, that has all

# details for each exception, and display those that matches the exceptions from the first csv file.

# Remember, the first line is the header "key,value".

foreach($line in $(get-Content $exceptionsSorted -totalCount 6))

{

# Get the key and value from the csv file.

[string[]] $values = @($line.ToString().Split(","))

if($values[0] -ne "key")

{

write-Host "Exception " -foreground Green -background Black -nonewline

write-Host $values[0] -foreground Red -background Black -nonewline

write-Host " appeared " -foreground Green -background Black -nonewline

write-Host $values[1] -foreground Red -background Black -nonewline

write-Host " times.`n" -foreground Green -background Black

# Displays call stack.

write-Host $hashTable[$values[0]].Replace($delimiter, "`n") -foreground Green -background Black

write-Host "`n=======================================================================`n" -foreground Green -background Black

}

}

}

# Do the basic work. Extract the exceptions from the file and put it into a csv file. Put the exceptions and number of time they appear in

# another csv file.

Extract-Exception $fileName

write-Host "`n`n=============================== SUMMARY ===============================`n" -foreground Green -background Black

# Display the statistics.

Display-ExceptionStatisticsSummary

write-Host "`n`n============= TOP 5 EXCEPTIONS AND 5 DIFFERENT CALL STACKS =============`n" -foreground Green -background Black

# Display the statistic details.

Display-ExceptionStatisticsDetails