Helpful Internals of TRX (and VSMDI files)


One of the questions that comes up internally, and externally is “What are these magic GUID attributes in the file? How can I generate them?”. This is often from people who are creating tools to export TRX files from some sort of private test hardness, or generate VSMDI files in an automated fashion.

Note, that the XSD schemas are available with all visual studio installs in the:

%VSINSTALLDIR%\xml\Schemas\vstst.xsd file

directory, along with many other schemas.

TRX Files

Lets take a look at TRX files first. I’m not going to go through the whole file line by line — we put a lot of info into the file, and some of it can be easily guessed. Things in the TRX file other than the obvious “results”:

  • Runconfig – we just dump in the whole run config that was use as part of that run
  • Test List info – The test list info etc that was associated with the tests as they were run. If you just run the test from test view, or use the keyboard shortcuts (Ctrl-R, T), you won’t see the test list info.

Neither of these pieces of information are required per-se, we just dump them in there for reference reasons (mostly for re-running without having all the original information).

Interesting lines:

<TestRun id=”774eb9dc-b311-4cdd-9987-48dd12f1e5f4” />

The GUID here is just a randomly generated GUID — Guid.New().ToString() is good enough here, it’s just used to allows us to identify the run compared to any other arbitrary run.

<TestRunConfiguration name=”Local Test Run” id=”3b7e71cb-2506-4724-9b77-b3f5d5c4dd70“/>

This GUID is ID of the Test Run Config that was copied from the one specified at the time the run was created. This should match, but there’s no specific need — we don’t go and look up that GUID later.

<UnitTest id=”59863143-3238-122e-4d8a-637b491cc755“/>

This is where it gets interesting. This element, which is found under the TestDefinitions element enumerates/declares all the tests that are part of this run. Tests have a name, which is used to provide a friendly way for users to identify test cases from each other. However, the tests themselves need to be uniquely identified within the test system. Most test types (all but unit, for the ones that are shipped in-box) are identified using a random GUID, which is stored in the file the test is stored in. Example: Open up a word based manual test, and look on the properties of it — you’ll see an item which has a GUID in it, and is called “ID”. The ID here on these child elements (children of TestDefinitions) is that same GUID as found the test. However, this is not the case for unit tests. Unit tests are generated using a hash fully qualified class name, and the method name [1] (i.e. Company.Product.TestClass.TestMethod).

<Execution id=”958d14e2-2f82-4f2c-8e91-84543ca85428” />

This is another “random” GUID, however it is referred to later in the file, specifically as part of the results and tracking which tests were executed, so make sure that if you’re generating here, your not just throwing the GUID out of the window after you generate it.

<TestLists>
<TestList name=”All Loaded Results” id=”19431567-8539-422a-85d7-44ee4e166bda” />
</TestLists>

This is similar to the test run configuration section, where it’s a dupe of the actual test list file. This contains all the test lists that are in this run, with the name (as we can see), and the GUID — again, this really is just a random GUID. But as per <Execution />, it’s used later in the file.

<TestEntry testId=”59863143-3238-122e-4d8a-637b491cc755” executionId=”958d14e2-2f82-4f2c-8e91-84543ca85428” testListId=”8c84fa94-04c1-424b-9868-57a2d4851a1d” />

Test Entry provides a match between test lists, the test case, and the test result. Note that these GUIDs here refer to the IDs previously seen throughout this post. Make sure these GUIDs match up, and your results will all be peachy.

<UnitTestResult executionId=”958d14e2-2f82-4f2c-8e91-84543ca85428” testId=”59863143-3238-122e-4d8a-637b491cc755” />

These are the results (whose elements are determined by the test type that was executed, hence the ‘UnitTestResult’), and as one can see, there are some guids here, which tally back to elements & ID’s that were declared earlier in the file — specifically the execution ID, and the test ID (since this is a unit test, the computed ID).

VSMDI Files

The VSMDI file format is much simpler, although equally filled with GUIDs :). The basic format here is the runconfig (don’t ask why it’s in there, it’s got long historical reasons, and does allow for an… interesting scenario [2]) attached to the of the test lists (which is the same GUID no matter what), and then all the other test lists in the file, with their own GUID, a guid linking to a parent test list, and the contents of that list itself. Lets look at an example.

<TestList name=”Lists of Tests” id=”8c43106b-9dc1-4907-a29f-aa66a61bf5b6“>

This is the “Root” test list, the one you see at the root of test list editor window – an immovable, unchangeable test list. This GUID is constant and the same in all VSMDI files, and all machines. It’s been the same GUID since we shipped the first beta of VS 2005 (amusing to think that was over 4 years ago). If you don’t have this, then things are going to go a bit pete tong.

Other test lists take the same form, but with there own GUID, which is merely a randomly generated GUID:

<TestList name=”SampleList” id=”e3d327f8-49dd-4ee0-b8c0-57e9b47d6cff” parentListId=”8c43106b-9dc1-4907-a29f-aa66a61bf5b6“>

Here, we can see that this node is a child of the parent node (Lists of Tests) by noting the parentListId GUID is the same as the ID guid of the “Lists of Tests” test list. This allows you to establish many levels of nesting of test lists to create the right grouping and categorization you need. However, for tests to appear in these test lists, you need to add child elements of this TestList element:

<TestLink id=”59863143-3238-122e-4d8a-637b491cc755” />

Here, the ID refers to the test id of the test being linked in. If you look back through this article you can see the ID of this test showing in the TRX file section — these IDs really do identify the test across multiple stores (even into the TFS warehouse if you push your results there).

Conclusion

There you have it, a whirl wind tour of the TRX and VSMDI file formats — which should provide some insight for understanding the rest of our file formats. Note that while some of the information here is still relevant to VS 2005, the file format in that version is radically different, and requires some what unnatural acts to be able to generate it from outside of VS — I would recommend upgrading to 2008 if you wish to do that 🙂

[1] We hash the fully qualified test method with code like this:

using System;
using System.Diagnostics;
using System.Security.Cryptography; 

namespace TeamTestSample
{
    internal static class UnitTestIdHash
    {
        private static HashAlgorithm s_provider = new SHA1CryptoServiceProvider(); 

        internal static HashAlgorithm Provider
        {
            get { return s_provider; }
        } 

        /// 
        /// Calculates a hash of the string and copies the first 128 bits of the hash
        /// to a new Guid.
        /// 
        internal static Guid GuidFromString(string data)
        {
            Debug.Assert(!String.IsNullOrEmpty(data);
            byte[] hash = Provider.ComputeHash(System.Text.Encoding.Unicode.GetBytes(data)); 

            // Guid is always 16 bytes
            Debug.Assert(Guid.Empty.ToByteArray().Length == 16, "Expected Guid to be 16 bytes"); 

            byte[] toGuid = new byte[16];
            Array.Copy(hash, toGuid, 16); 

            return new Guid(toGuid);
        }
    }
}

[2] One little know feature of the testing tools inside visual studio, is that you can re-run a test run without sources, just by picking up the test results directory, and opening the TRX file, and clicking the run button. What is even less known is that if you have the Test storages (the dlls, webtest, etc), and the VSMDI, you can open the VSMDI directly (no project, or solution), and select the tests to run. The test run configuration that will be applied will the last test run configuration that was active when the vsmdi was last edited.


Comments (7)

  1. Maor David on Configure Team Foundation Build For An Incremental Build Mike Azocar on Want to be famous?…Enter…

  2. starbuck_ms says:

    Great article!

    What about the case where there really is no Test List or Test List is not within a VSMDI file? If the test list is set to a Guid.NewGuid(), then when the .trx file is opened within VS2008, it complains that the TestListID is invalid. A dialog of “Test List with Id {guid} not found.” Is there a way to get around this?

    Thanks in advance…

  3. dhopton says:

    So, to clarify the question:

    a) trx file with results in it

    b) You put the test list details in the test list section of the trx file

    c) You have bound the result result to the test list section?

    Do you have a specific example of the TRX in this case?

  4. starbuck_ms says:

    To answer…

    a) Yes.

    b) No. In the .xsd, the TestListType only says Description, RunConfiguration and TestLinks.

       I have none of these. Should I add them?

    c) I think so… The UnitTestResult node contains an attribute testListId which matches…

    It looks like the TestLists node is optional, not required. Then, if the TestLists Node is left out, then the TestEntries node can also be left out since it has a required attribute ‘testListId’. However, the UnitTestResult node has a required testListId.

    I can generate a file which contains a specific example. Can I email it to you?

    Thanks…

  5. dhopton says:

    Well, if you look at a TRX with test results in it that were bound to test lists, you’ll see a test list section in there.

    The key here is, the test id property is required; but it default to a "Well known" ID, as mentioned in myposts. However to refere to your custom lists, you need to dump them in the TRX file so we can load them. The vsmdi file isn’t required to view the tests by lists in the results window, just the trx. Thus all the goop is in there.

  6. DmytroL says:

    Hi Dominic,

    Would you please comment on several related questions?

    1. Is it possible to publish several TRX files under the same test run (of course provided all the TRX files in question specify the same correct test run GUID in the TestRun tag)?

    2. Is it permitted to overwrite the outcome of a certain test within the same test run?

    3. Are there any potential problems if these TRX files are published by different people?

    The scenario behind all this is: a test run is a collection of manual tests put together to verify certain functionality (I know there’s a notion of test list for this, but one can’t slice the TFS warehouse cube by test lists). Now, subsets of the test run are assigned to different QA engineers for execution. So, each QA engineer goes through her “share” of tests and publishes the results – but the results from all QA Engineers involved should go under the same test run.

    Thanks in advance!

  7. There was a interesting question posted in my comments on the TRX file format , about being able to republish