Troubleshooting WMI Repository Bloat


I was recently asked to assist in troubleshooting large WMI repositories, sometimes in upwards of 1GB. This was causing the slow RDP logon issue described in KB2020286. We found that RSOP logging was not enabled on these machines. My first step was to identify which servers had large WMI repositories. I created a SCOM Management Pack (MP) for this which I have attached. The MP also does some basic health checks of WMI using the script below.

Update: I removed the /verifyrepository check from the MP. We found that it can sometimes conflict with WMI on reboots. WMI also does a /verifyrepository on startup. If SCOM also runs this check at the same time then WMI may appear to be hung for several minutes, but it eventually recovers. Connections to WMI can still be made but queries won’t return anything until after both /verifyrepository commands are done.

1 #functions 2 function ConnectToWMINamespace 3 { 4 param ([string]$nameSpace, [ref]$ErrorsFound) 5 $wmiProvider = Get-WmiObject -Namespace $nameSpace -Class '__Namespace' 6 if ($?) 7 { 8 $ret = $false 9 } 10 else 11 { 12 $ErrorsFound.value += "Failed to connect to WMI Namespace {0} with error: {1}`n" -f $nameSpace, $error[0] 13 $ret = $true 14 } 15 return $ret 16 } 17 18 function VerifyRepository 19 { 20 $winmgmt = winmgmt /verifyrepository 21 if ($winmgmt -eq "WMI repository is consistent") {$ret = $false} else {$ret = $true} 22 return $ret 23 } 24 25 function GetFileSize 26 { 27 param ([string]$path) 28 $windir = 'windir' 29 $windir = Get-Item env:$windir 30 $fullpath = $windir.Value + $path 31 $file = Get-Item $fullpath 32 $filesize = $file.Length/1MB 33 return "{0:N0}" -f $filesize 34 } 35 36 function SendToSCOM 37 { 38 param ( 39 [string]$Failure, [string]$ConnectToCimv2, [string]$WMIRepository, 40 [int]$WMIRepositorySize, [string]$WMIRepositorySizeOverThreshold, 41 [int]$WMIRepositorySizeThreshold, [string]$Errors) 42 $api = New-Object -comObject MOM.ScriptAPI 43 $bag = $api.CreatePropertyBag() 44 $bag.AddValue('Failure', $Failure) 45 $bag.AddValue('ConnectToCimv2', $ConnectToCimv2) 46 $bag.AddValue('WMIRepository', $WMIRepository) 47 $bag.AddValue('WMIRepositorySize', $WMIRepositorySize) 48 $bag.AddValue('WMIRepositorySizeOverThreshold', $WMIRepositorySizeOverThreshold) 49 $bag.AddValue('WMIRepositorySizeThreshold', $WMIRepositorySizeThreshold) 50 $bag.AddValue('Errors', $Errors) 51 $bag 52 } 53 54 #Main script 55 [string]$ErrorsFound = "" 56 [bool]$bFailed = $false 57 [bool]$bOverSizeThreshold = $false 58 [int]$wmiRepositorySizeThreshold = '$Config/WMISizeThreshold$' 59 if ((!($wmiRepositorySizeThreshold -is [int])) -or ($wmiRepositorySizeThreshold -eq 0)) {$wmiRepositorySizeThreshold = 500} 60 61 #Run tests 62 $cimv2 = ConnectToWMINamespace -nameSpace("root\cimv2") -ErrorsFound([ref]$ErrorsFound) 63 $wmiRepository = VerifyRepository 64 [int]$wmiRepositorySize = GetFileSize -path("\system32\wbem\repository\objects.data") 65 66 #Review results 67 if ($cimv2) {$bFailed = $cimv2} 68 if ($wmiRepository) {$bFailed = $wmiRepository} 69 if ($wmiRepositorySize -gt $wmiRepositorySizeThreshold) 70 { 71 Write-Host $wmiRepositorySize 72 Write-Host $wmiRepositorySizeThreshold 73 $bFailed = $true 74 $bOverSizeThreshold = $true 75 } 76 77 #Print results 78 Write-Host "Overall Failure: "$bFailed.ToString() 79 Write-Host "cimv2 Failure: "$cimv2.ToString() 80 Write-Host "winmgmt /verifyrepository Failure: "$wmiRepository.ToString() 81 Write-Host "WMI Size: "$wmiRepositorySize"MB" 82 Write-Host "WMI Size Threshold: " $wmiRepositorySizeThreshold 83 Write-Host "Over Size Threshold Failure: "$bOverSizeThreshold.ToString() 84 Write-Host "WMI Errors Reported: "$ErrorsFound 85 86 #Send to SCOM 87 SendToSCOM $bFailed.ToString() $cimv2.ToString() $wmiRepository.ToString() $wmiRepositorySize $bOverSizeThreshold.ToString() $wmiRepositorySizeThreshold $ErrorsFound

Once we identified the machines with WMI repositories over 300MB we worked to identify what in the repository was taking up all the space. I researched various ways to determine this including recursively going through each namespace and class in WMI and counting them but since many providers populate instances that aren’t actually stored in the repository I figured this wouldn’t be an accurate method of determining what is taking up space in WMI. I ended up going with the method below instead.

  1. Copy the objects.data (WMI repository) file from several of the machines identified to your desktop. The file is located in the %windir%\system32\wbem\repository folder.
  2. Download strings from sysinternals
  3. Run the following command: strings objects.data > strings.txt
  4. Run the following PowerShell script after modifying the two variables at the top of the script.
1 $fileIn = 'c:\temp\strings.txt' 2 $fileOut = 'c:\temp\output.csv' 3 4 function PopulateHash ([string]$s, [hashtable]$h) 5 { 6 if (!$h.ContainsKey($s)) 7 { 8 $h.Add($s, 1) 9 } 10 else 11 { 12 $i = $h.Get_Item($s) 13 $i++ 14 $h.Set_Item($s, $i) 15 } 16 } 17 $strings = @{} 18 19 Write-Host "Counting strings" 20 21 foreach ($line in [System.IO.File]::ReadLines($fileIn)) 22 { 23 PopulateHash $line $strings 24 } 25 26 $strings.GetEnumerator() | Sort-Object Value -Descending | Export-Csv $fileOut 27

We found that SCCM data seemed to be using up the most space. Knowing that I figured that now would be a good time to recursively go through the root\ccm namespace (used by SCCM) to determine which class has the most instances. The script below shows you how many instances are under root\ccm (and any namespaces under this) and lists out each class along with the top namespaces.

1 function CountWMIObjects 2 { 3 param ([string]$nameSpace) 4 Write-Host "." -NoNewline 5 if ($nameSpace -like "root\ccm*") 6 { 7 $objects = Get-WmiObject -Namespace $nameSpace -List 8 foreach ($object in $objects) 9 { 10 $instances = Get-WmiObject -Namespace $nameSpace -Class $object.Name -ErrorAction SilentlyContinue 11 $instanceLine = "{0}({1})" -f $object.Name, $instances.count 12 $entryName = "{0}:{1}" -f $nameSpace, $object.Name 13 $global:AllInstances.Add($entryName, $instances.Count) 14 $count += $instances.Count 15 } 16 $namespaceLine = "{0} ({1} classes, {2} instances)" -f $nameSpace, $objects.count, $count 17 } 18 else 19 { 20 $objects = @(0) 21 } 22 23 $global:TotalObjects += $objects.Count 24 $subs = Get-WmiObject -Namespace $nameSpace __NAMESPACE | Sort-Object 25 26 if($subs -ne $null) 27 {$subs | ForEach-Object { CountWMIObjects "$nameSpace\$($_.Name)"}} 28 } 29 30 function Get-ScriptDirectory 31 { 32 $Invocation = (Get-Variable MyInvocation -Scope 1).Value 33 Split-Path $Invocation.MyCommand.Path 34 } 35 36 #Main script 37 [int]$global:TotalObjects = 0 38 [hashtable]$global:AllInstances = @{} 39 $OutputFile = "SCCMClasses.csv" 40 41 #Run test 42 CountWMIObjects "root" 43 44 #export to csv 45 $path = Join-Path (Get-ScriptDirectory) $OutputFile 46 $global:AllInstances.GetEnumerator() | sort value -Descending | Export-Csv $path -NoTypeInformation 47 Write-Host "" 48 Write-Host "SCCM Instances Found: "$global:TotalObjects 49 Write-Host "Output File: $path" 50

Eventually we found that one of the causes was SCCM Software Metering data, particularly on terminal servers, due to the amount of people logging on and launching applications. I continue troubleshooting WMI repository bloat in my Dumping WMI Instance Property Values for all Classes in a Namespace post.

WMIRepositoryBloatTroubleshooting.zip


Comments (6)

  1. Hi Russ,

    Thanks for the detailed explanation and info.

    I have one question, what did you do to fix the problem?

    Does an updated client setting where the Software Metering is disabled flush the already collected data?

    With kind regards,

    Sven Gertsen

  2. rslaten says:

    Hi Sven,

    Unfortunately I don't have the complete story in this case. I was simply helping diagnose the problem (using some code I wrote and Support Escalation Engineer Michael Smith's idea to use strings.exe to parse WMI repositories) and once it was diagnosed a different team implemented the solution. They definitely disabled metering for the affected terminal servers but I'm not sure if they had to uninstall/reinstall the SCCM client, delete the repository, etc…

    Russ

  3. GeekySeb says:

    Hello and thanks for all this info. It helped a lot. Here is the first lines of the result of your counting-line script. Any idea what's that and why is my OBJECTS.DATA file is 560MB?

    "New","New","556535"

    "C00000000_0000_0000_0000_000000000002","C00000000_0000_0000_0000_000000000002","542672"

    "6,)6","6,)6","541048"

    "AA8FB2BFAB5C4CFF12EF86C3BCD580008AFB8F7E4F06AC32B2981B188EFF229F","AA8FB2BFAB5C4CFF12EF86C3BCD580008AFB8F7E4F06AC32B2981B188EFF229F","509524"

    "FileSystemFile>","FileSystemFile>","95901"

  4. rslaten says:

    Interpreting the output can be challenging when the culprit isn't obvious. In your case I know that the C00… and FileSystemFile refer to SCCM software inventory. You can go on to part 2 of the post which might give you more details on what namespaces and classes are using up the most space. Once you have that information you might be able to tweak some of your application settings so less data is stored in WMI.

  5. GeekySeb says:

    Thanks for the reply. I don't know what SCCM is supposed to be doing on my machine, but this is certainly not correct. I backed-up the repository and reset it. It is now 13MB and I hope SCCM will not do that again. Otherwise I'll follow part 2. I'm assuming something got corrupted during a crash.

    Thank you very much for your help, it has been very useful.

  6. Gregory Hindman says:

    In the last script, I was running into an issue with the following:

    30 function Get-ScriptDirectory

    31 {

    32   $Invocation = (Get-Variable MyInvocation -Scope 1).Value

    33   Split-Path $Invocation.MyCommand.Path

    34 }

    After researching, it looks like this was by design for PowerShell 1.0 and 2.0. I am on Windows 10 running PowerShell 5.0

    PS C:WINDOWSsystem32> $PSVersionTable

    Name                           Value                                                                    

    —-                           —–                                                                    

    PSVersion                      5.0.10240.16384                                                          

    WSManStackVersion              3.0                                                                      

    SerializationVersion           1.1.0.1                                                                  

    CLRVersion                     4.0.30319.42000                                                          

    BuildVersion                   10.0.10240.16384                                                        

    PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}                                                  

    PSRemotingProtocolVersion      2.3

    I deleted the function and changed the path to the following:

    $OutputFile = "D:SCCMClasses.csv"

    $path = $OutputFile

    Thanks