Verily, 'Tis Truth

Your Logical Functional Model lets you write test cases from your user's point of view, test cases that focus on what is being done rather than how it is being done. Your Physical Object Model lets your test cases ignore the details behind interacting with your user interface. Execution Behaviors let you push into your test automation stack decisions about which of the plethora of ways each user action can be executed. Data Manager does the same for the values your test cases use. Together these let your test cases be straightforward and clear and simple to write. Just one thing is lacking: verification.

Loosely Coupled Comprehensive Verification lets you verify everything you care about all the time. The first step is to notify Verification Manager every time an action starts or finishes:

public static void Logical.Question.SelectAnswer(string answerId)
{
Verification.ActionStarting();

    Logical.MainWindow.CurrentQuestion.GetAnswer(answerId).Invoke();

    Verification.ActionEnding();
}

That's it as far as the LFM is concerned! You see why we call it Loosely Coupled: the only point at which verification is coupled to anything else is here, where the LFM tells the verification system when things happen.

Things are rather more interesting on the verification side of things.

public static void VerificationManager.ActionStarting()
{
string executingActionsName = this.GetNameOfCaller();
object[] executingActionsArguments = this.GetArgumentsForCaller();
this.ExpectedState = this.SnapshotCurrentState();
this.NotifyExpectedStateGenerators(executingActionsName, executingActionsArguments, this.ExpectedState);
}

When Verification Manager is notified that an action is starting, Verification Manager determines the name of and arguments for the action being called. (In .Net, the name is simple to determine via Reflection. Due to security reasons, however, the arguments are impossible to determine, so they must be retrieved through some other mechanism.) Next it baselines its state to the current application state. Finally it passes this information on to its Expected State Generators.

private static void VerificationManager.NotifyExpectedStateGenerators(
string executingActionsName, object[] executingActionsArguments, State expectedState)
{
// Magic happens here!
}

Notifying the Expected State Generators can be done any number of different ways. Your options include:

  • Switch on the action name and then notify a hard-coded set of Expected State Generators. You will have to make modifications any time actions or Generators are added, but if both sets are mostly stable the simple implementation may be worth the occasional edit.
  • Pass every event on to every Generator; each Generator then has to determine whether to respond to or ignore each event. This keeps knowledge of which events a particular Generator handles isolated within the Generator, but notifying every Generator every action will have a runtime cost.
  • Have each Generator register for the events it is interested in. This gives the Generators the most responsibility for managing themselves while ensuring you notify the minimum set of events, but you do have to manage all those registrations.

When an Expected State Generator is told that an action is starting, it does whatever it does to decide how the application state should change and then it updates the expected state to record that information:

internal void OnSelectAnswer(object[] executingActionsArguments, State expectedState)
{
foreach (State.Answer answer in expectedState.Questions[expectedState.CurrentQuestion].Answers)
{
answer.IsSelected = false;
}

    expectedState.Questions[expectedState.CurrentQuestion].Answers[executingActionsArguments[0]].IsSelected = true;
}

In the case of selecting an answer, I mark every answer as not selected and then go back and mark the selected answer as selected.

After every Expected State Generator has done its thing, control returns back to the LFM which then goes about executing the action. When execution is complete Verification Manager goes back into action:

public static void VerificationManager.ActionEnding()
{
State actualState = this.SnapshotCurrentState();
this.ExpectedState.CompareTo(actualState);
}

This is the easy part: compare the expected state to current state and log any differences as failures.

A common question about this scheme is performance. "Don't all those snapshots slow things down?" "Won't all those calculations take too long?" Our experience has been that they don't. However, while the snapshots must of course occur in realtime, both generating expected state and comparing that to actual state can occur at some time after the test has completed. This change minimizes the runtime cost while still allowing you to verify everything (or as much of everything as you care about) all the time.

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: michhu at microsoft dot com. Great coding skills required.