Testing the Toolkit
When we first started working on the Toolkit, we evaluated several different testing methods. First, we looked at the testing features built intoVSTS specifically for ASP.NET but they mostly just watch page round trips, and therefore wouldn't work for our stuff that's modifying the DOM at runtime.
We also looked at some of the Javascript testing frameworks out there like JUnit, as well as something that the Atlas team had put together. Each of these had short comings as well, for example they don't deal well with post-backs. Of course, with AJAX you want to minimize postbacks, but we've designed the Toolkit to work well with traditional ASP.NET server-side programming as well, so thre are scenarios where they make sense. Plus we needed to make sure it was okay for us to publish, etc.
So we came to the conclusion that we'd have to hand-roll something, which we did just to a level that was good enough for our uses and we've used that for the releases up until now.
But since we've started taking contributions, it needed to be better, so one of my dev's Ted, did a great job of pulling together a great test harness framework that solves these needs. He's been banging away at getting it all working and published for the last week or so. If you want to take a look at it, just go to CodePlex and download one of the recent checkins of the Toolkit.
There are two pieces to the testing functionality. One is a TestHarness page and framework that allows you to pick and run the various tests, and the other is the test pages themselves.
In the test pages, you can write either JavaScript to excercise your behaviors on the client side or server-side code to modify them via the extender on the server side. Or both; it's very flexible.
The real innovation in the test harness framework, however, is that it unifies the way that you handle synchronous steps, asynchronous steps, and postbacks. The real challenge with writing these tests is knowing when something completes. For example, if you're testing the CollapsiblePanel, you can initiate a collapse or expand, but since it runs an animation, you have to write a bunch of code to wait for the test to complete. You'd like to write this:
var
cb = $object("collapse1");
cb.doOpen();
verifyPanelExpanded();
cb.doClose();
varifyPanelCollapsed();
But that won't work. Why? Because the open/close calls kick of an animation that won't complete when the next line of code is run.
So you end up with something like this, where you chain a bunch of stuff together with timeouts.
var
cb = $object("collapse1");
cb.doOpen();
window.setTimeout(verifyPanelExpanded, 250);
// ...
function
verifyPanelExpanded() {
var cb = $object("collapse1");
// check to make sure it's expanded
cb.doClose();
window.setTimeout(verifyPanelCollaped, 250);
}
This is bad for several reasons. First, the timeouts are arbitrary, and anyone who's experienced with using timeouts for this stuff knows that they don't work reliably. There's always a case where your timeout isn't long enough, and they make everything run really slow. Second, if you have more than 2 steps, the logic gets extremely difficult to follow as things jump from method to method. And this still doesn't deal with postbacks.
After we got the last Tookit release out the door and got our stuff moved up to CodePlex, it became a top priority to get testing out there with the framework itself, especially if we're going to be taking external contributions. So we sat down to figure out a way to unify this. The idea came up of being able to just define setups and put them onto a queue, and have the test harness just crank through them. This is nice for several reasons.
First, it's easy to have the harness wait or pause for things to complete, and second it's also easy to resume a test after a postback, since the harness knows what "step" it's on. And it let's you easily define other callbacks for checking status, etc.
Along with that, we added functions that do the normal TDD assert stuff, like test.assertTrue, etc., and you can now define your tests in a much more understandable fashion, with very little plumbing code.
function registerTests(harness) {
testHarness = harness;
var test = testHarness.addTest('Initial State');
test.addStep(checkBothExpanded);
test = testHarness.addTest(
'Collapse');
test.addStep(checkBothExpanded);
test.addStep(clickBoth, waitCollapsed, 150, 5000);
//...
}
So each test case is a"test" above, and to it gets added a series of steps. In the last line, for example, what that's saying is:
- Call the function "clickBoth"
- Then, every 150ms, call the function "waitCollapsed". If it returns true, move to the next step, otherwise wait 150ms and try again.
- Fail if the elasped time goes longer than 5000ms
The only real caveat is that the functions you call can't take paramters, but that's easy to get around with Javascript closures, which we ended up using extensively. They're very handy. Anyway, clickBoth and waitCollapsed are simple methods for checking or modifying state:
function
clickBoth() {
testHarness.fireEvent(panel,
'onclick');
testHarness.fireEvent(hpanel,
'onclick');
}
function waitCollapsed() {
try {
checkBothCollapsed();
return true;
}
catch (ex) {
return false;
}
}
function checkBothCollapsed() {
testHarness.assertNotEqual(panelParent.offsetHeight, 0,
"Content should be vertically expanded, it's " + panelParent.offsetHeight + 'px');
testHarness.assertEqual(label.innerHTML,
'Opened', "TextLabel should display 'Opened'");
// ...
}
So if any of the "assert" functions fail, the test that's currently running fails and will show up in the UI.
We were able to convert all of our existing tests to this new framework in just a few hours, it was very smooth and the tests work well and are reliable. Here's how it looks in action:
As with the Toolkit itself, this framework is included in full source under the MS-PL license, so you're welcome to use it in your projects.
