Determining Whether Tests Passed in Team Build


In a forum post a while back, I laid out a method for determining whether tests had passed during a build.  More recently, I have linked to this forum post in advising others on similar problems.  Unfortunately, as a sharp user pointed out in this same thread, my solution doesn’t actually work, since it relies on a property that is not accessible in Team Build v1!


So – to remedy the situation I have written a custom task which can be used to determine whether tests have succeeded or not.  This task takes advantage of the GetTestResultsForBuild method of the Microsoft.TeamFoundation.Build.Proxy.BuildStore class. 


As always, this task is provided as a sample only; its awesomeness cannot be guaranteed, etc.  The task inherits from the TeamBuildTask class I presented in an earlier post.

using System;
using System.Web.Services;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;
using Microsoft.TeamFoundation.Client;
using Microsoft.TeamFoundation.Build.Common;
using Microsoft.TeamFoundation.Build.Proxy;

namespace MyNamespace
{
public class CheckForTestSuccess : TeamBuildTask
{
/// <summary>
/// The ConfigurationToBuild Item Group for the Build.
/// </summary>
[Required]
public ITaskItem[] ConfigurationToBuild
{
get
{
return m_configurationToBuild;
}
set
{
m_configurationToBuild = value;
}
}

/// <summary>
/// An output property which will be true if all test runs succeeded and false otherwise.
/// </summary>
[Output]
public bool TestSuccess
{
get
{
return m_testSuccess;
}
set
{
m_testSuccess = value;
}
}

/// <summary>
/// Returns the name of the build step to be added for this task.
/// </summary>
/// <returns>Name of the build step to be added.</returns>
protected override string GetBuildStepName()
{
returnCheckForTestSuccess“;
}

/// <summary>
/// Returns the message of the build step to be added for this task – this is the
/// string displayed in the Team Build GUI.
/// </summary>
/// <returns>Message of the build step to be added.</returns>
protected override string GetBuildStepMessage()
{
returnChecking for test success.“;
}

/// <summary>
/// Put real task logic in this method.
/// </summary>
/// <returns>True if task is successful, otherwise false.</returns>
protected override bool ExecuteInternal()
{
string platform;
string flavor;
TestResultData[] testResults;

m_testSuccess = true;

foreach (ITaskItem configuration in ConfigurationToBuild)
{
platform = configuration.GetMetadata(“PlatformToBuild“);
flavor = configuration.GetMetadata(“FlavorToBuild“);

testResults = BuildStore.GetTestResultsForBuild(BuildUri, platform, flavor);

foreach (TestResultData testResult in testResults)
{
if (!testResult.RunPassed)
{
m_testSuccess = false;
break;
}
}
if (!m_testSuccess)
{
break;
}
}

return true;
}

private ITaskItem[] m_configurationToBuild;
private bool m_testSuccess;
}
}


To use this task, you’ll need to do something like the following in TfsBuild.proj:

<UsingTask TaskName=“CheckForTestSuccess” AssemblyFile=“CustomTasks.dll” />

<Target Name=“AfterTest”>
<CheckForTestSuccess TeamFoundationServerUrl=“$(TeamFoundationServerUrl)”
BuildUri=“$(BuildUri)”
ConfigurationToBuild=“@(ConfigurationToBuild)”>
<Output TaskParameter=“TestSuccess” PropertyName=“TestSuccess” />
</CheckForTestSuccess>
</Target>


The property TestSuccess would then be ‘true’ if and only if all test runs succeeded for the build.

Comments (11)

  1. Buck Hodges says:

    Aaron Hallberg has written a couple of posts about Team Build that started with questions from users.&amp;nbsp;…

  2. Leon Mayne says:

    I can’t seem to get TFS to run this. I get this error in the build log:

    Target AfterTest:

     C:TFSBuildAlchemyCI Test BuildBuildTypeTFSBuild.proj(153,3): error MSB4062: The "CheckForTestSuccess" task could not be loaded from the assembly C:TFSBuildTasksCheckBuildSuccess.dll. Could not load file or assembly ‘file:///C:TFSBuildTasksCheckBuildSuccess.dll’ or one of its dependencies. The system cannot find the file specified. Confirm that the <UsingTask> declaration is correct, and that the assembly and all its dependencies are available.

    Done building target "AfterTest" in project "TFSBuild.proj" — FAILED.

    I have all the related TFS DLLs in the same folder. I did get a problem with dependencies, which I managed to resolve but may have done incorrectly. Which DLLs are you referencing in your project?

  3. Buck Hodges says:

    Does the service account being used by the Team Build Windows service have access to the c:tfsbuildtasks directory?

    Buck

  4. I’m speaking at TechEd 2007 in Orlando this coming June on the whys and hows of customizing TFS. As part

  5. Sharing this information from Jeff Beehler’s Blog Aaron’s posts on how to extend team build through custom

  6. In a blog post back in September of last year I described how to use a custom task to determine whether

  7. In a blog post back in September of last year I described how to use a custom task to determine whether

  8. ramesh.m says:

    I followed the steps and created a Task in that I am sending email with Test results.

    But I am not getting any results.

    here is mycode

    protected override bool ExecuteInternal()

           {

               StringBuilder builder = new StringBuilder();

               TeamFoundationServer tfs = TeamFoundationServerFactory.GetServer(TeamFoundationServerUrl.ToString());

               BuildStore buildStore = (BuildStore)tfs.GetService(typeof(BuildStore));

               string buildUri = buildStore.GetBuildUri(m_teamProject, m_buildNumber);

               BuildData buildData = buildStore.GetBuildDetails(buildUri);

               string platform;

               string flavor;

               TestResultData[] testResults;

               m_testSuccess = true;

               //int testsPassed;

               //int testsFailed;

               int totalTests=0;

               foreach (ITaskItem configuration in ConfigurationToBuild)

               {

                   platform = configuration.GetMetadata("PlatformToBuild");

                   flavor = configuration.GetMetadata("FlavorToBuild");

                   builder.Append("No Of Test Cases:   " + BuildStore.GetTestResultsForBuild(BuildUri, platform, flavor).Length);

                   builder.Append(Environment.NewLine);

                   testResults = BuildStore.GetTestResultsForBuild(BuildUri, platform, flavor);

                   //TestResultData test = testResults;

                   //testsFailed=test.TestsFailed;

                   //testsPassed = test.TestsPassed;

                   totalTests = testResults.Length;

                   foreach (TestResultData testResult in testResults)

                   {

                       builder.Append("Test Result:   " + testResult.TestsPassed);

                       builder.Append(Environment.NewLine);

                       builder.Append("Test Result:   " + testResult.TestsFailed);

                       builder.Append(Environment.NewLine);

                       if (!testResult.RunPassed)

                       {

                           FailedTests = FailedTests + 1;

                           //m_testSuccess = false;

                           //break;

                       }

                       else

                       { SuccessTests = SuccessTests + 1; }

                   }

                   //if (!m_testSuccess)

                   //{

                   //    break;

                   //}

               }

               MailMessage message = new MailMessage("Someone@example.com", "Someone@example.com");

               message.Subject = "CI-Build Result";

               //message.IsBodyHtml = true;

               // Construct the alternate body as HTML.

               string body = "<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">";

               body += "<HTML><HEAD><META http-equiv=Content-Type content="text/html; charset=iso-8859-1">";

               body += "</HEAD><BODY><DIV><FONT face=Verdana color=#ff0000 size=2>this is some HTML text";

               body += "</FONT></DIV></BODY></HTML>";

               // Add the alternate body to the message.

               AlternateView alternate = AlternateView.CreateAlternateViewFromString(body);

               message.AlternateViews.Add(alternate);

               //"This is the <b>HTML text</b> for the body of the message and this is <font color=red>red</font>.";

               string strHeader = "<b><font color=blue; size=12px>" + buildData.TeamProject + "  Build " + buildData.BuildNumber + " " + buildData.BuildStatus + "</font></b>";  

               builder.Append(strHeader.ToUpper(CultureInfo.CurrentCulture));

               builder.Append(Environment.NewLine);

               builder.Append(Environment.NewLine);

               builder.Append("BuildMachine :      " + buildData.BuildMachine);

               builder.Append(Environment.NewLine);

               builder.Append("BuildNumber  :      " + buildData.BuildNumber);

               builder.Append(Environment.NewLine);

               builder.Append("BuildType    :      " + buildData.BuildType);

               builder.Append(Environment.NewLine);

               builder.Append("DropLocation :      " + buildData.DropLocation);

               builder.Append(Environment.NewLine);

               builder.Append("RequestedBy  :      " + UserFullName.GetCurrentUserFullName(buildData.RequestedBy));

               builder.Append(Environment.NewLine);

               builder.Append("StartTime    :      " + buildData.StartTime);

               builder.Append(Environment.NewLine);

               builder.Append("TeamProject  :      " + buildData.TeamProject);

               builder.Append(Environment.NewLine);

               builder.Append("BuildUri     :      " + buildData.BuildUri);

               builder.Append(Environment.NewLine);

               builder.Append("BuildStatus  :   " + buildData.BuildStatus);

               builder.Append(Environment.NewLine);

               builder.Append("BuildQuality :  " + buildData.BuildQuality);

               builder.Append(Environment.NewLine);

               builder.Append("FinishTime   :    " + buildData.FinishTime);

               builder.Append(Environment.NewLine);

               //if (SuccessTests + FailedTests != 0)

               //{

                   builder.Append("TestsPassed  :    " + SuccessTests);

                   builder.Append(Environment.NewLine);

                   builder.Append("TestsFailed  :   " + FailedTests);

                   builder.Append(Environment.NewLine);

               //}

               builder.Append("Build Log Location :    " + buildData.LogLocation);

               builder.Append(Environment.NewLine);

               message.Body = builder.ToString();

               //builder.Append("No Of Test Passed:   " + testResult.TestsPassed);

               //message.Body = builder.ToString();

               //builder.Append("No Of Test Failed:   " + testResult.TestsFailed);

               //message.Body = builder.ToString();

               message.IsBodyHtml = true;

               SmtpClient client = new SmtpClient("MySMTPSERVER");

               client.Send(message);

               return true;

           }

    I am getting the mail with 0 Test cases.

    but in the build log I have 900 tests Passed and 600 Failed.

    What may be the problem

  9. Garry Trinder says:

    One issue is that you seem to be misinterpreting the TestResultData class – this class encapsulates a test run, not an individual test.  See the docs for this class here: http://msdn2.microsoft.com/en-us/library/microsoft.teamfoundation.build.proxy.testresultdata_members(VS.80).aspx.  The TestsTotal property should give you the number of tests executed for the test run.

    It would appear, however, that you are getting 0 TestResultData objects back from the server.  Are you using the task as in the example (i.e. passing in @(ConfigurationToBuild) as the ConfigurationToBuild parameter value)?  If so, what platforms and flavors are specified in that item group?  What target are you overriding (AfterTest, or what)?

  10. Hi Aaron,

    Thank you for the post.  This was exactly what I was looking for.  One simple question but it appears your ExecuteInternal() method will always return true.

    I think you meant the return statement to be

    return m_testSuccess;

    instead of:

    return true;

    Thanks again for the help.

  11. My apology.  There’s nothing wrong with your code.  My mistake.