Duplicate Files

PowerShell Team

Need a way to check if two files are the same?  Calculate a hash of the files.  Here is one way to do it:

## Calculates the hash of a file and returns it as a string.

function Get-MD5([System.IO.FileInfo] $file = $(throw ‘Usage: Get-MD5 [System.IO.FileInfo]’))

{

  $stream = $null;

  $cryptoServiceProvider = [System.Security.Cryptography.MD5CryptoServiceProvider];

  $hashAlgorithm = new-object $cryptoServiceProvider

  $stream = $file.OpenRead();

  $hashByteArray = $hashAlgorithm.ComputeHash($stream);

  $stream.Close();

  ## We have to be sure that we close the file stream if any exceptions are thrown.

  trap

  {

    if ($stream -ne $null)

    {

      $stream.Close();

    }

    break;

  }

  return [string]$hashByteArray;

}

I think about the only new thing here is the trap statement.  It’ll get called if any exception is thrown, otherwise its just ignored.  Hopefully nothing will go wrong with the function but if anything does I want to be sure to close any open streams.  Anyway, keep this function around, we’ll use it along with AddNotes and group-object to write a simple script that can search directories and tell us all the files that are duplicates.  Now… an example of this function in use:

MSH>”foo” > foo.txt

MSH>”bar” > bar.txt

MSH>”foo” > AlternateFoo.txt

MSH>dir *.txt | foreach { get-md5 $_ }

33 69 151 28 248 32 88 177 8 34 154 58 46 59 255 53

54 122 136 147 125 209 249 229 12 105 236 19 140 5 107 169

33 69 151 28 248 32 88 177 8 34 154 58 46 59 255 53

MSH>

Note that two of the files have the same hash, as expected since they have the same content.  Of course, it is possible for two files to have the same hash and not the same content so if you are really paranoid you might want to check something else in addition to the MD5 hash.

– Marcel

[Edit: Monad has now been renamed to Windows PowerShell. This script or discussion may require slight adjustments before it applies directly to newer builds.]

PowerShell Team PowerShell Team, PowerShell

Follow