SysTest part V.: Test execution (results, runners and listeners)

In the series of previous posts I've covered TestClasses and TestSuites. You should already know how to create a TestClass and how to group your tests in TestSuites. In this post I'll try to cover the test execution environment.

Results and Runners

You probably remember the following code:

  static void RunSimpleTests(Args _args)
 {
    SysTestSuite suite = new SysTestSuite(classstr(SimpleTests));
    SysTestResult result = new SysTestResult();
    ;
    result.addListener(new SysTestListenerPrint());
    suite.run(result);
    print result.getSummary();

    pause;
 }

This code creates a runtime suite for all test methods from SimpleTests class. The following diagram displays all objects that are interacting during test execution.

SysTestRunner is a new component that is used to glue your test and result object. It takes care of the suite creation as well as the result object creation. The following sample is doing the same thing as the previous one.

  static void RunSimpleTestsUsingRunner(Args _args)
 {
    SysTestRunner runner = SysTestRunner::createTestRunner(classstr(SimpleTests));
    ;
    runner.getResult().addListener(new SysTestListenerPrint());
    runner.run();
    print runner.getSummary();

    pause;
 }

Actually it does more than that. Let's do a little experiment. Go to Tools > Development Tools > Unit Test > Parameters. Navigate to Listeners tab and select Infolog listener (move it from right to left using the [<] button). Now rerun the previous job and after pressing Yes on the pause dialog notice that an Infolog is displayed. This means that SysTestRunner creates a result object for you and by default configures the result object based on the settings from Parameters form (listeners, code coverage, etc.). 

Listeners

There is one more component in the diagram above, SysTestListener. Result object uses Observer pattern for reporting results. SysTestResult object acts as the subject and observers are "listeners". Listeners are implementing SysTestListener interface. To be more flexible it is possible to implement either "push model" or "pull model". All out of the box listeners in DynamicsAX use the push model. To support the push model the framework contains SysTestListenerData interface that is used to pass change value object to all registered observers.

DynamicsAX contains out of the box a wide range of listeners.

To use different listeners, just call addListener on your result object. You can have multiple listeners registered at the same time.

  static void UsingListeners(Args _args)
 {
    SysTestSuite suite = new SysTestSuite(classstr(SimpleTests));
    SysTestResult result = null;
    ;
    print 'Registered listeners: NO';
    result = new SysTestResult();
    print suite.run(result).getSummary();
    print '';
    pause;

    print 'Registered listeners: print';
    result = new SysTestResult();
    result.addListener(new SysTestListenerPrint());
    print suite.run(result).getSummary();
    print '';
    pause;

    print 'Registered listeners: infolog';
    result = new SysTestResult();
    result.addListener(new SysTestListenerInfolog());
    print suite.run(result).getSummary();
    // Give infolog chance to be displayed
    infolog.setTimeOut('notify', 200);
    infolog.wait();
    print '';
    pause;
    infolog.closeInfolog();

    print 'Registered listeners: print, database';
    result = new SysTestResult();
    result.addListener(new SysTestListenerPrint());
    result.addListener(new SysTestListenerDB());
    print suite.run(result).getSummary();
    print 'Because we used database listener we can load results using unit test form.';
    pause;
 }

The last part of the previous sample uses SysTestListenerDB listener. This listener persists all the test results in the database. You can browse the results with the Test jobs form by going to Tools > Development Tools > Unit Test > Test jobs. Navigate to the last test job (I know there is a bug with sorting of the jobs based on the ID). You should see our SimpleTests in there. Select this record and click on Tests button. A form is displayed with details about all test classes and test methods that were run.

Implementing a custom listener

If you need to implement your own listener (logging to a different database, to a file in a certain file format, using a COM or .NET object, etc.) then all you need to do is to implement the SysTestListener interface. The following example demonstrates how to create a “failure” memory listener. The listener will contain only information about failures. To get messages from the memory it will publish getMessages method.

  public class FailureListener implements SysTestListener
 {
    List failures;
     
    public void new()
    {
        ;
        failures = new List(Types::String);
    }
    public List getMessages()
    {
        return failures;
    }
 
    public void addFailure(SysTestListenerData _data)
    {
        str message = strfmt('%1: %2', _data.parmTest().getName(), _data.parmMessage())
        ;
        failures.addEnd(message);
    }
    public void addInformation(SysTestListenerData _data)
    {
    }
    public void close()
    {
    }
    public void endSuite(SysTestListenerData _data)
    {
    }
    public void endTest(SysTestListenerData _data)
    {
    }
    public void open()
    {
    }
    public void startSuite(SysTestListenerData _data)
    {
    }
    public void startTest(SysTestListenerData _data)
    {
    }
 }

The following sample demonstrates the use of the new listener:

  static void UsingFailureListener(Args _args)
 {
    SysTestSuite suite = new SysTestSuite(classstr(SimpleTests));
    SysTestResult result = new SysTestResult();
    FailureListener listener = new FailureListener();
    ListIterator iterator = null;
    ;
    result.addListener(listener);
    suite.run(result);

    iterator = new ListIterator(listener.getMessages());
    while (iterator.more())
    {
        print iterator.value();
        iterator.next();
    }
    print '--------------------------------------------------------------';
    print result.getSummary();

    pause;
 }

You can also use any of the available listeners and just extend its behavior. This way you can easily change their formatting for example because most of them have protected write method.

You can download these samples: Part05: Execution.xpo

This is all for this post. Next week I'll try to cover code coverage.