Using a little Ruby to automate mstest code coverage runs


I hate repetitive tasks – especially ones that are easily automated.  Generating code coverage runs using mstest is an example of that.


The goal was to create a script that could do that following:


1)     Validate that the input binaries were available


2)     Validate that the necessary runtime components were available


3)     Instrument the input binaries


4)     Build up the test sandbox


5)     Start the coverage service


6)     Perform the tests using the instrumented binaries


7)     Stop the coverage service


Being able to start the process at any point in there was bonus points.


I choose to use ruby, and specifically Rake, to automate this process.  Rake is perfect for this type of task.  I can easily break the tasks down into their core parts and build up the automation one bit at a time.  Creating dependencies between the tasks is trivial.  It reads well (even with my poor ruby skills) and has a very nice command line experience.


What I ended up with was a process where I could do this:


C:\ruby\coverage>rake –tasks


(in C:/ruby/coverage)


coverage coverage:all       # Perform a complete code coverage pass…


coverage coverage:clean     # Clean the coverage temp directory and sandbox


coverage coverage:cover     # Instrument the assemblies for code coverage


coverage coverage:merge     # Merge the test results


coverage coverage:populate  # Populate the sandbox


coverage coverage:restore   # Restores the backed up files if they exist


coverage coverage:run       # Run the test suite


coverage coverage:start     # Start the coverage service


coverage coverage:stop      # Stop the coverage service



I can do all the things I set out to do (using “coverage:all”) or onesy-twosy as I need them (e.g. if rake aborts midway through the “run” step the coverage service will need to be manually stopped before the next run – I could modify a task to handle this more gracefully but I haven’t yet since the payoff isn’t there yet).


But you came for code …


Rakefile


 


require ‘rake’


require “win32/dir”


require ‘fileutils’


require ‘yaml’


 


def assert_config(config)


  [‘vsperfcmd’,


   ‘prodbin’,


   ‘root’,


   ‘sandbox’,


   ‘covtemp’,


   ‘covorig’,


   ‘vsinstr’,


   ‘suitebin’]. each { |setting|


      assert_config_setting(config, setting)


  } 


end


 


def assert_config_setting(config, setting)


  fail “Setting \”#{setting}\” not found in coverage.yml” if config[setting].nil?


end


 


Rake.application.init(‘coverage’)


 


config = YAML::load(File.open(‘coverage.yml’))


assert_config(config)


 


covfiles = FileList.new(


‘Microsoft.TeamFoundation.Admin.dll’,


‘Microsoft.TeamFoundation.Management.Core.dll’,


‘Microsoft.TeamFoundation.Management.SnapIn.dll’,


‘Microsoft.TeamFoundation.Management.Controls.dll’,


‘tfsconfig.exe’,


‘tfsmgmt.exe’


)


 


namespace :coverage do


 


  desc “Instrument the assemblies for code coverage”


  task :cover => [:clean_coverage, :clean_orig ] do


    Dir.chdir(config[‘prodbin’]) do


      mkdir config[‘covtemp’]


      covfiles.each do |f|


        fail “Input file not found: #{f} if not File.exists?(f)


        puts “Instrumenting #{f}


        cmd = “\”#{config[‘vsinstr’]}\” \”#{f}\” /COVERAGE”


        sh cmd


        assert_file_exists(File.join(config[‘covtemp’], f), cmd)


        copy_coverage(f, config[‘covtemp’])


      end


    end


  end


 


  desc “Restores the backed up files if they exist”


  task :restore do


    Dir.chdir(config[‘prodbin’]) do


      covfiles.each do |f|


        origfile = #{f}.orig”


        if File.exists?(origfile)


          rm f if File.exists?(f)


          mv origfile, f


         


          instrpdb = f.gsub(/(.*)\.(dll|exe)$/, ‘\1.instr.pdb’)


          rm instrpdb if File.exists?(instrpdb)


        end


      end


    end


  end


 


  task :clean_backup do


    Dir.chdir(config[‘prodbin’]) do


      covfiles.each do |f|


        origfile = #{f}.orig”


        rm origfile if File.exists?(origfile)


 


        instrpdb = f.gsub(/(.*)\.(dll|exe)$/, ‘\1.instr.pdb’)


        rm instrpdb if File.exists?(instrpdb)


      end


    end


  end


 


  task :clean_coverage do


    FileUtils.rm_rf(config[‘covtemp’])


  end


 


  task :clean_orig do


    FileUtils.rm_rf(config[‘covorig’])


  end


 


  task :clean_sandbox do


    FileUtils.rm_rf(config[‘sandbox’])


  end


 


  desc “Start the coverage service”


  task :start do


    mkdir config[‘sandbox’] unless File.exists?(config[‘sandbox’])


    sh “\”#{config[‘vsperfcmd’]}\” /start:coverage \”/output:#{File.join(config[‘sandbox’], ‘adminops.coverage’)}\” /user:redmond\\vseqa1 /user:redmond\\tfssvc /user:#{ENV[‘USERDOMAIN’]}\\#{ENV[‘USERNAME’]} /waitstart”


  end


 


  desc “Stop the coverage service”


  task :stop do


    sh #{config[‘vsperfcmd’]} /shutdown”


  end


 


  desc “Run the test suite”


  task :run do


        Dir.chdir(config[‘sandbox’]) do


          puts “Running Unit Tests”


          sh “aotest.exe /assembly:AdminOps.UnitTests.dll /test * /out:logs”


 


    end


  end


 


  desc “Merge the test results”


  task :merge do


  end


 


  desc “Populate the sandbox”


  task :populate=> [:clean_sandbox] do


    mkdir config[‘sandbox’]


   


    puts “Copying Product binaries…”


    FileUtils.cp_r(Dir.glob(File.join(config[‘prodbin’], ‘Microsoft.TeamFoundation.*’)), config[‘sandbox’])


    cp File.join(config[‘prodbin’], ‘NetFwTypeLib.dll’), config[‘sandbox’]


 


    puts “Copying Test binaries…”


    FileUtils.cp_r(File.join(config[‘suitebin’], ‘.’), config[‘sandbox’])


 


    puts “Copying Instrumented binaries…”


    FileUtils.cp_r(Dir.glob(File.join(config[‘covtemp’], ‘.’)), config[‘sandbox’])


  end


 


  desc “Perform a complete code coverage pass from scratch”


  task :all => [:clean, :cover, :populate, :start, :run, :stop, :merge]


 


  desc “Clean the coverage temp directory and sandbox”


  task :clean => [ :clean_coverage, :clean_sandbox, :restore, :clean_backup ]


end


 


def assert_file_exists(file, cmd)


  if not File.exists?(file)


    puts “COVERAGE ERROR: file not found: #{file}


    puts “Executed: #{cmd}


  end


end


 


def copy_coverage(file, dir) 


  if File.exists?(file)


    cp file, dir


  else


    die “Assembly not found for assembly #{file}


  end


 


  pdbfile = file.gsub(/(.*)\.(dll|exe)$/, ‘\1.pdb’)


  if File.exists?(pdbfile)


    cp pdbfile, dir


  else


    puts “WARNING: PDB file not found for assembly #{pdbfile}


  end


 


  instrpdbfile = file.gsub(/(.*)\.(dll|exe)$/, ‘\1.instr.pdb’)


  if File.exists?(instrpdbfile)


    cp instrpdbfile, dir


  else


    puts “WARNING: INSTR PDB file not found for assembly #{instrpdbfile}


  end


end


 


coverage.yml


vsperfcmd: E:/Program Files/Microsoft Visual Studio 10.0/Team Tools/Performance Tools/vsperfcmd.exe


prodbin: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386


root: C:/VMDEV/TfsArch1/binaries/x86chk


sandbox: C:/SANDBOX


covtemp: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386/coverage


covorig: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386/coverage/orig


vsinstr: E:/Program Files/Microsoft Visual Studio 10.0/Team Tools/Performance Tools/vsinstr.exe


suitebin: C:/VMDEV/TfsArch1/binaries/x86chk/SuiteBin/i386/tfs/AdminOps/bin


 


So there you have it…


 


I go to my ruby command line (which is just a normal command line with the ruby bin directory in the path  – e.g. “set PATH=%PATH%;c:\ruby\bin”) and perform:


C:\ruby\coverage>rake coverage:all


When the tests are done the coverage file I created is in c:\sandbox ready to be imported into VS or merged with other files or dumped to excel or whatever.


And one note about the rakefile – you may notice that I call “aotest.exe” not “mstest.exe” – aotest is just a little wrapper around mstest that one of our test devs wrote.  I could have just as easily used mstest.


 

Comments (0)