Creating an API testing framework 101
Writing a fully featured API testing solution is a lot of work. You have to deal with things like test case management, integration with code coverage tools or test execution on server farms. However, the core of a (reflection-based) API testing framework is quite simple. This post explains the basics with EMTF as an example.
Implementing asserts
Assertions are the building blocks for actual API test cases (my previous post Introducing EMTF contains a simple example). A single failing assertion fails the whole test and causes the execution of the current test method to be cancelled. How does a method that doesn't even return a value achieve this? Fairly simple: it throws an exception.
[DebuggerHidden]
public static void IsTrue(Boolean condition, String message)
{
if (!condition)
throw new AssertException("Assert.IsTrue failed.", message);
}
Since it throws a specific exception type, a failed assertion can be distinguished from a crashing test method based on the type of the exception. But isn't it bad to use exceptions to return from an operation? It usually is and this is one of the few exceptions to this rule. First of all, a failing test is (hopefully) still an exceptional event. Usually we want and expect our tests to pass. Second, there is no better way to allow the test runtime to cancel the execution of a test method when an assertion fails.
Another little detail is the DebuggerHiddenAttribute on all methods of the Assert class (Assert.cs) as well as the public constructors of the AssertException class (AssertException.cs). This tells the debugger that the methods/constructors marked with this attribute should be omitted from stack traces so they won't show up when debugging assertion failures.
Identifying test classes and test methods
A common way of marking classes as test classes and methods as test methods is the use of attributes. This allows the test runtime to automatically find and execute all tests and has the advantage of keeping the object model of the test suite flexible. An alternative is to define a base class or interface that test classes must derive from or implement respectively and treating all public, non-abstract, non-generic, void returning, parameter less methods as test method. Though this simplifies both the test code and the code that finds all test methods through reflection a little bit it comes at the expense of pretty much dictating what a test class looks like.
EMTF contains the type TestClassAttribute (TestClassAttribute.cs) for test classes and TestAttribute (TestAttribute.cs) for test methods. These attributes can contain additional (optional) information or be supplemented by additional attributes. EMTF's TestAttribute for example has a property for the test description.
[AttributeUsage(AttributeTargets.Class)]
public sealed class TestClassAttribute : Attribute
{
}
[AttributeUsage(AttributeTargets.Method)]
public sealed class TestAttribute : Attribute
{
private String _description;
public String Description
{
get
{
return _description;
}
}
public TestAttribute()
{
}
public TestAttribute(String description)
{
_description = description;
}
}
Finding all tests
private static Collection<MethodInfo> FindTestMethods()
{
Collection<MethodInfo> testMethods = new Collection<MethodInfo>();
// Iterate through all assemblies in the current application domain
foreach (Assembly assembly in AppDomain.CurrentDomain.GetAssemblies())
{
// Iterate through all exported types in each assembly
foreach (Type type in assembly.GetExportedTypes())
{
// Verify that the type is:
// 1. Marked with the TestClassAttribute
// 2. Not a generic type definition or open constructed type
// 3. Not abstract
if (type.IsDefined(typeof(TestClassAttribute), true) &&
!type.ContainsGenericParameters &&
!type.IsAbstract)
{
// Iterate through all public instance methods
foreach (MethodInfo method in type.GetMethods(BindingFlags.Instance | BindingFlags.Public))
{
// Verify that the method is:
// 1. Marked with the TestAttribute
// 2. Not a generic method definition or open constructed method
// 3. Its return type is void
// 4. Does not have any parameters
if (method.IsDefined(typeof(TestAttribute), true) &&
!method.ContainsGenericParameters &&
method.ReturnType == typeof(void) &&
method.GetParameters().Length == 0)
{
// Add method to the list of
// test methods to execute
testMethods.Add(method);
}
}
}
}
}
return testMethods;
}
Executing the tests
The last piece missing is a method that executes all test methods and logs the results or at least raises events which can be handled by the caller or a dedicated logger. The basic logic for test execution used by EMTF's TestExecutor (TestExecutor.cs) and other API testing runtimes looks like this:
private void ExecuteImpl(IEnumerable<MethodInfo> testMethods)
{
// Raise event signaling the start of a test run
OnTestRunStarted();
// Instance of the current test class
Object currentInstance = null;
// Iterate through all test methods
foreach (MethodInfo method in testMethods)
{
// Get the TestAttribute on the method
object[] testAttribute = method.GetCustomAttributes(typeof(TestAttribute), true);
string testDescription = null;
// Get the test description if TestAttribute is defined on the method
if (testAttribute.Length > 0)
testDescription = ((TestAttribute)testAttribute[0]).Description;
// Try to instantiate the test class if necessary
// Skip the test if an instance cannot be created
if (!TryUpdateTestClassInstance(method, testDescription, ref currentInstance))
continue;
// Raise an event signaling the start of a test
OnTestStarted(new TestEventArgs(method, testDescription));
// Invoke the test method in a try block so we can
// catch assert failures and unexpected exceptions
try
{
method.Invoke(currentInstance, null, true);
}
// Assert failed
catch (AssertException e)
{
// Raise event signaling the test failure
// then immediately run the next test
OnTestCompleted(
new TestCompletedEventArgs(
method,
testDescription,
e.Message,
e.UserMessage,
TestResult.Failed,
null));
continue;
}
// Test threw unexpected exception
catch (Exception e)
{
// Raise event signaling the test failure
// then immediately run the next test
OnTestCompleted(
new TestCompletedEventArgs(
method,
testDescription,
String.Format(
CultureInfo.CurrentCulture,
"An exception of the type '{0}' occurred during the execution of the test.",
e.GetType().FullName),
null,
TestResult.Exception,
e));
continue;
}
// No exceptions were thrown
// Raise event signaling that the test passed
OnTestCompleted(
new TestCompletedEventArgs(
method, testDescription, "Test passed.", null, TestResult.Passed, null));
}
// Raise event signaling the completion of the test run
OnTestRunCompleted();
}
This posting is provided "AS IS" with no warranties, and confers no rights.