True Stories In Test Automation
[The following is a true story. The feature has been changed to protect the unannounced, but the code and test progression is a near verbatim account of a coding session I had recently.]
Say you are testing a timeline. Specifically, you are testing sliding a timeline item around on the timeline. An item can be moved anywhere along the timeline, but moving an item before the origin or off the end of the timeline is not allowed.
The timeline has several different timescales it can use:
- The typical linear timescale
- A logarithmic timescale
- A "fisheye" timescale, where the timeline is magnified within the area of focus, has normal scale around the area of focus, and is anti-magnified further out from the area of focus
You need to write a helper method that manipulates an item on the timeline using its user interface; in other words, to simulate interacting with the timeline the way the user will. This seems relatively simple for the linear case, and not too much harder for the logarithmic case, but the fisheye case sounds like it may require some complicated math -- essentially reproducing the logic the timeline uses to decide how to render itself. That's a scary thought, so let's avoid it as long as we can.
The first step is to create a list of test cases:
- Drag an item on a linearly scaled timeline to a later time value (i.e., to the right).
- Drag an item on a linearly scaled timeline to a earlier time value (i.e., to the left).
- Drag an item on a linearly scaled timeline past another timeline item.
- Drag an item on a linearly scaled timeline to a time value that does not exist (i.e., off the end of the timeline).
- Drag an item on a linearly scaled timeline to a negative time value (i.e., past the beginning of the timeline).
- Drag an item on a linearly scaled timeline exactly onto another timeline item.
- Do each of the preceding tests for a logarithmically scaled timeline and for a fisheye scaled timeline.
Next let's define some terms and invent some functionality that we'll pretend already exists in our automation library:
- The horizontal line that defines the timeline's axis we'll call the Timeline. It has a (zero-based) collection of its TimelineItems. It knows its start time, end time, and timescale.
- Each TimelineItem in the Timeline's TimelineItemCollection knows its current time value.
- The bubble that represents an item on the timeline we'll call an Adorner; specifically, a TimelineItemAdorner.
- An Adorner can give us a clickable point. Dragging the mouse starting from a timeline item adorner's clickable point will drag the corresponding timeline item.
- Our automation library provides methods for clicking the mouse (i.e., a mouse down, pause, mouse up sequence) at a specific absolute location and for dragging the mouse (i.e., a mouse down, pause, one or more mouse moves, pause, mouse up sequence) to a specific absolute location. It also provides mouse down, mouse move, and mouse up primitives.
- Our automation library provides a helper method that returns a reference to the Timeline.
With that in hand we can get started. Let's start with the simplest case: dragging an item along a linearly scaled timeline. Neither moving right nor moving left seems significantly simpler nor more complicated than the other, so let's arbitrarily decide to start with the former.
(I'll use C#-ish psuedocode throughout. You should be able to translate this into your language of choice without much trouble. I won't explicitly show verification of results, but you should always verify every last thing you can think of.)
[Setup]
TestCaseSetup()
{
Create a timeline // Defaults to length 1000.
Add an item at time 100
Add an item at time 500
}
[Teardown]
TestCaseTeardown()
{
Delete the active timeline
}
[TestMethod]
MoveTimelineItemToLaterTime()
{
ActiveTimeline.MoveTimelineItem(0, 300)
}
This is a straightforward test. If we assume that our timeline displays itself such that one logical pixel is the same as one time point (e.g., time 100 is 100 pixels from the timeline's origin), the implementation to make this work is straightforward as well:
Timeline.MoveTimelineItem(itemIndex, newTimeValue)
{
adorner = Get the adorner for item #itemIndex
originPoint = Get the absolute location of the timeline's origin
startPoint = adorner.GetClickablePoint
destinationPoint = newTimeValue + originPoint
Mouse.Drag(startPoint, destinationPoint)
}
Simple stuff. What's more, it works for several of the other linearly scaled tests as well:
[TestMethod]
MoveTimelineItemToEarlierTime()
{
ActiveTimeline.MoveTimelineItem(0, 50)
}
[TestMethod]
MoveTimelineItemPastOtherTimelineItem()
{
ActiveTimeline.MoveTimelineItem(0, 600)
}
[TestMethod]
MoveTimelineItemToOtherTimelineItem()
{
ActiveTimeline.MoveTimelineItem(0, 500)
}
Whether we need any changes to handle the illegal cases depends on how the timeline handles out of range items. If it silently stops dragging timeline items when they hit either end of the timeline, we just need to make our verification code handle those cases. If it pops a message box or otherwise throws an error for these cases, however, we may decide to write specific test cases for these edge conditions:
// Somewhere amongst our timeline test cases:
[TestMethod]
MoveTimelineItemPastBeginningOfTimeline()
{
this.MoveTimelineItemToIllegalTimeValue(0, -10)
}
[TestMethod]
MoveTimelineItemPastEndOfTimeline()
{
this.MoveTimelineItemToIllegalTimeValue(0, ActiveTimeline.Length)
}
MoveTimelineItemPastBeginningOfTimeline(itemIndex, newTimeValue)
{
adorner = Get the adorner for item #itemIndex
originPoint = Get the absolute location of the timeline's origin
startPoint = adorner.GetClickablePoint
destinationPoint = newTimeValue + originPoint
Mouse.Drag(startPoint, destinationPoint)
if (the timeline error dialog is visible)
{
Log a pass
Dismiss the timeline error dialog
}
else
{
Log a fail
}
}
This leaves our automation system free to throw an exception in these cases:
// Back in our unit tests:
// Somewhere amongst our timeline test cases:
[ExpectedException(ArgumentOutOfRangeException)]
[TestMethod]
MoveTimelineItemPastBeginningOfTimeline()
{
this.MoveTimelineItem(0, -10)
}
[ExpectedException(ArgumentOutOfRangeException)]
[TestMethod]
MoveTimelineItemPastEndOfTimeline()
{
this.MoveTimelineItem(0, ActiveTimeline.Length)
}
Timeline.MoveTimelineItem(itemIndex, newTimeValue)
{
if ((newTimeValue < 0.0) or (timeline.Range < newTimeValue))
{
throw new ArgumentOutOfRangeException("newTimeValue", newTimeValue
, "The new time value must be within the timeline's limits.")
}
adorner = Get the adorner for item #itemIndex
originPoint = Get the absolute location of the timeline's origin
startPoint = adorner.GetClickablePoint
destinationPoint = newTimeValue + originPoint
Mouse.Drag(startPoint, destinationPoint)
}
(Of course, we could also decide our automation library should look for and dismiss the error dialog, in which case we would neither do the check nor throw an exception. This would work if we do verification inline in the automation library methods, but not if verification is done external to the automation library -- that verification needs to know if the error dialog appeared, but it wouldn't have any way to get this information.)
Since GetAdornerForTimelineItem throws if the specified timeline item doesn't exist, we don't need to explicitly check whether the requested timeline item actually exists; instead we just pass the index along and let GetAdornersForTimelineItem throw when necessary.
That's all the linearly scaled cases, so let's move on to the logarithmically scaled cases. Calculating the drag point is a bit more complicated than it was for the linear case, but not too much more:
Timeline.MoveTimelineItem(itemIndex, newTimeValue)
{
if ((newTimeValue < 0.0) or (timeline.Range < newTimeValue))
{
throw new ArgumentOutOfRangeException("newTimeValue", newTimeValue
, "The new time value must be within the timeline's limits.")
}
adorner = Get the adorner for item #itemIndex
originPoint = Get the absolute location of the timeline's origin
startPoint = adorner.GetClickablePoint
if (timeline is linearly scaled)
{
destinationPoint = newTimeValue + originPoint
}
else
{
destinationPoint = CalculateXCoordinateForLogarithmicValue(newTimeValue) + originPoint
}
Mouse.Drag(startPoint, destinationPoint)
}
We check the timeline's type and either do the simple calculation we did before or call out to a helper function we get someone else to write. Simple! <g/>
This would work (even if we have to write the helper ourselves), and we could do much the same thing for fisheye scaled timelines. We would now have three separate algorithms to maintain, however, and some of them are likely to be complicated. If additional timeline types are added in the future, or zooming, or any number of other things, the code gets ever more complicated.
Not to fear; there's a better way. What if we just dragged the timeline item until it reached the desired time value? A naïve implementation might look like this:
Timeline.MoveTimelineItem(itemIndex, newTimeValue)
{
if ((newTimeValue < 0.0) or (timeline.Range < newTimeValue))
{
throw new ArgumentOutOfRangeException("newTimeValue", newTimeValue
, "The new time value must be within the timeline's limits.")
}
adorner = Get the adorner for item #itemIndex
startPoint = adorner.GetClickablePoint
while (the timeline item's current time value < the desired time value)
{
Mouse.MoveTo(startPoint, 1)
startPoint = adorner.GetClickablePoint
}
while (the desired time value < the timeline item's current time value)
{
Mouse.MoveTo(startPoint, -1)
startPoint = adorner.GetClickablePoint
}
}
If the timeline item is currently to the left of the desired value, drag it right one pixel; if it's to the right, drag left. Repeat until it reaches the desired value. Moving a pixel at a time we're bound to hit the desired value spot on.
Try it on our existing tests; most pass, but MoveTimelineItemPastOtherTimelineItem fails. Why? Run it again, watching carefully. Oh…we're doing a series of drags -- in other words, a mouse down, mouse move, and mouse up each time. At one point we move the timeline item directly over the other timeline item, so that both timeline items now have identical clickable points. The next mouse down grabs the second timeline item, not the one we want.
Is this the correct behavior? The spec should say, and if not a three-way between us, our dev, and our PM is in order. In this case we find that this behavior was specifically left unspecified. Thus the first item might get selected, or perhaps the second, and what happens this time isn't necessarily what happens next time.
Okey dokey. A series of mouse drags isn't going to work, then; instead we need to do one long drag so that we never release the timeline item:
Timeline.MoveTimelineItem(itemIndex, newTimeValue)
{
if ((newTimeValue < 0.0) or (timeline.Range < newTimeValue))
{
throw new ArgumentOutOfRangeException("newTimeValue", newTimeValue
, "The new time value must be within the timeline's limits.")
}
adorner = Get the adorner for item #itemIndex
startPoint = adorner.GetClickablePoint
Mouse.Down(startPoint)
while (the timeline item's current time value < the desired time value)
{
Mouse.MoveTo(startPoint + 1)
startPoint = adorner.GetClickablePoint
}
while (the desired time value < the timeline' items current time value)
{
Mouse.MoveTo(startPoint - 1)
startPoint = adorner.GetClickablePoint
}
Mouse.Up(startPoint)
}
With that change all our tests are passing again. Add a new test for a logarithmically scaled timeline:
[TestMethod]
MoveTimelineItemToLaterTimeOnLogarithmicallyScaledTimeline()
{
Toggle the active timeline to be logarithmically scaled
ActiveTimeline.MoveTimelineItem(0, 5000)
}
That passes, so try a fisheye timeline. That passes too.
So we're set, right? Yes and no. Our algorithm does work, but it's inefficient. Moving timeline items any significant distance will take forever moving just a pixel at a time. Let's see what happens with a really long timeline:
[TestMethod]
MoveTimelineItemAReallyLongDistance()
{
Stretch the active timeline to be ten times as long
ActiveTimeline.MoveTimelineItem(0, (the active timeline's length - 10))
}
That took a while!
Another incentive to enhance our algorithm is our wish to test like the user. Users won't generally inch along; they'll move in large jumps until they get close to the desired value, then scale down to more and more exact movements. Let's see if we can't duplicate that behavior.
We'll start by dragging a large distance rather than the one pixel we drag now. What should this new distance be? We could choose some arbitrary number (50, say, or 10), but this number is likely to be too small for long timelines and too long for short timelines. Basing the drag distance on the timeline's length should generally give us a decent starting point:
Timeline.MoveTimelineItem(itemIndex, newTimeValue)
{
if ((newTimeValue < 0.0) or (timeline.Range < newTimeValue))