Tests aren’t methods!


In the feedback section of one of my blogs, Aaron Junod wrote:


“In writing this, I just realized something. I think someone should draw a line between writing tests, and actual program code, and then offer different options based on that. They are totally diff coding paradigms, and really merit diff behaviors not only in the compiler, but the IDE.. Hmm.. think I gotta think about this more.”

I told him I wasn’t going to respond and I’d let Jay get a chance to voice his opinion on this, and he has! (Hurrah!)   The issue arose when dealing with our frustration that NUnit forces both your TestFixture class (which is unnecessary) to be public (which is unnecessary) and requires that all tests in that class (which is unnecessary) to be public (which is unnecessary).  Wow.  That’s a whole lot of unnecessary stuff.  What irked us was that when we were trying out the new VS Team System tools, they also had these restrictions.  Jay inquired into the reasoning behind these decisions and raised several good points for why this restrictions was a “Bad Thing”.  They were:

  1. Nothing about the concept of a unit test has anything to do with a method.  A unit test is just code that runs and tests something for you.  We package it in a method because a method is a convenient way of naming something and packaging a set of statements together. However, a method carries unnecessary overhead that just isn’t necessary in a test. The concept of it having accessibility, input or output parameters is just meaningless and unnecessary.  Imagine if a unit test required you to take an integer as an argument.  It would be pointless and would just burden the programmer. Similarly, by forcing a test to be contained in a method which is public you are unnecessarily burdening the programmer.  This makes the test visible to people interacting with the system.  Which is completely unnecessary.  The test should only exist so that the engine can run them and report the results back.  Of course, if a user wants to make his tests publicly accessible, he can.  But he should not be forced to.
  2. Forcing the tests to be public goes against the XP philosophy of “do the simplest thing that could possibly work”.  In order to have this restriction someone had to go out of their way to prevent it from happening.  Why would you ever do that?  Why do extra work in order to limit what the customer wants to be able to do?  If you listen to the new Office System commercials, they espouse the philosophy of “Do more with less”.  But here we are doing the opposite.  We are doing extra work to make sure that you can’t do something. 
  3. The Team System software isn’t supporting a completely reasonable coding style.  With the team system method you will end up with a loosely couple system where you have “class MyClass” and a “public class MyClassTests”.  Jay wants to code in the manner of “class MyClass{ class Tests {} }” so that he has tight coupling between the class and the code that tests that class.  That way if he moves the class around, the tests stay with it.  If he renames the class he doesn’t have to worry about renaming the test class, and he doesn’t have the redundancy of “MyClass” specified twice.

Note: Jay is not saying that everyone should test in his style.  He is saying that both styles can be supported with less work on the Team System team. 


I’d like to know your thoughts on this so I can take back some feedback on what the community thinks.  I would appreciate a discussion on why you think it’s a good or bad idea to force tests to be public.  Addressing all of Jay’s issues would be very useful, and if you can introduce points as to why limiting what the user can do here is a good thing, then we’d like to here them.  Remember, jay is not saying “force the test to be private”, he’s saying “support this behavior which doesn’t prevent anything that a user can currently do with the system”


Thanks!


Comments (24)

  1. AT says:

    1. Hey ? Are use sure that all test methods must be void TestFoo() ?

    How about cool tests inheritance, then you create base abstract class to test all methods in interface and then you need to test class implementing this interface you simply add corresponding testTarget object with new class ?

    Do you believe if you test your business objects for cash balance 100USD and 200USD you must go to source code, change value and recompile to test for 200.000.000.000USD ?? Or you must be able to do this using config file or even more – flexible UI ?

    Current testing like a baby, it need to grow up. Parameters for tests are a must. Without params it’s like a C "void main(void)" function – which works, but all problems to parse command line offseted to programmer.

    2. What’s wrong with public ? IMHO, correct Unit test code must looks like

    public class TestFoo {

    [TestMethod]

    public TestFooResult TestDoFoo(FooConfigParam params) {

    TestFooResult result = new TestFooResult();

    Initialize(params);

    try {

    // Check everything and report any warnings to result

    } finaly {

    Cleanup();

    }

    return result;

    }

    }

    While Initialize and Cleanup must be called by developer, not by system. Currently if you need some complex structure for one of your tests and you are lazy you usualy put it in Initialize. As result you get extremely long test runs because system make useless initialization and cleanup for *each* test case. Not everybody understand this. Even more – sometimes even I’m totaly lost in my unit tests and unable to test static members.

    But this Initialize problem can be workaround-ed by complex attribute with initialization level like a

    [TestMethod(InitilizeLevel.Basic | InitilizeLevel.IgnoreStatic)]

    But it’s no way flexible 🙁

    Each test method must work regarles on current system state, it must not assume that it’s the only one method executed. It must reveal all side-effects from other activities in system (even in case if it will be hard to diagnose).

    You must be able to execute your ATM cashing tests successfuly even if test case from another developer who is responsible for complex reporting currently running in the same process on the same database and modify/lock your data.

    That’s why public modifed is aceptable.

    3. I realy like class Foo { class Tests {}} idea. But partial classes not supported across assemblies, currently the only one way is to use it is #ifdef for debug versions.

    This kind of nested test class solve problems I’ve with private (internal and protected) method/members access without reflection tricks.

    I like this code (everything except #ifdef ;o)

    #if DEBUG

    partial

    #endif

    class Foo { private void DoFoo() { } }

    #if DEBUG

    partial class Foo { class [Test] TestClass { void [TestMethod] TestFoo() { Foo f = new Foo(); f.DoFoo(); } } }

    #endif

  2. MartinJ says:

    We have this excellent system of adding XML comments to just about every member. Why not use this same general idea and extend it to testing.

    I heard that tests should not necessarily be methods. So, don’t expose them as methods to the developer. Expose them as a different type of comment. The compiler can generate the actual executable goo to make this work invisibly. Then, it falls to the compiler to determine if it should just embed the code in the same assembly and expose the meta data necessary for the testing framework to run. The compiler could also split the tests from the "stuff" being tested. Again, the meta data would also be generated for the testing framework.

    Does the developer really care if the tests live in the same phsyical assembly as the tested code? No. They just care that when they run the tests, the tested code is actually tested. The test framework can use reflection if necessary to get to the items it needs to.

    The whole notion of needing to use reflection is kind of moot. Does the test compiler need to follow the same rules as real languages? No. The idea is that the tests are compiled into MSIL. i believe visibility can be ignored in this low level view.

    I don’t have an answer for what "language" the tests should be written in. It would be nice to be able to use any .net language. It would also be nice to be able to use some type of specifications instead of the standard languages (VB, C#, C++, J#, …). I’m thinking of a way of accepting sets of inputs and making sure that the expected results happen (either a given value is returned, underlying data is altered correctly, or an expected exception thrown).

    What would be nice is that you could then write tests for snippets of code instead of at the method level. Imagine that you have a method that makes calls outside of the developer’s control (calling a third-party component). Instead of writing dummy calls to verify that the external call operates as desired, you could just wrap a block of code in a test. I know, I know. I hear people yelling refactor, extract it out. I’d agree if we were talking about code that was used in multiple places. But, imagine that this call is only made this one time. It would be wasteful to extract that one call out just so that you can test it.

    Ah, well. These are just some crazed ramblings of a naive systems analyst.

    -Martin

  3. AT: I never said that you shouldn’t have public tests. I said that you shouldn’t require public tests.

  4. Martin: I agree. There is no reason to make create your tests as methods, and you do not have to do so. However, currently, a method is simply a convenient container for the test. However, that container need not be public.

    As you stated a test is a test irregardless of visibility.

    You can currently write tests in any language, but the current ones supported directly in team system are any .net language.

    I all agree with testing blocks of code. I was having adebate with neil about that the otherday. If you really think of everything as an object, then a block is just another object that you’d like to be able to test.

  5. AT: "Do you believe if you test your business objects for cash balance 100USD and 200USD you must go to source code, change value and recompile to test for 200.000.000.000USD ?? Or you must be able to do this using config file or even more – flexible UI ? "

    No. I would do this:

    [Test]

    void TestBusinessObject() {

    TestCashBalance(100);

    TestCashBalance(200);

    }

    void TestCashBalance(Dollars dollars) { … }

    But again, there is no reason for the test to take any parameters, return any value, or be public 🙂

  6. AT: "2. What’s wrong with public ? "

    Nothing. As I stated. Public is fine. The question is "what’s wrong with private" 🙂

    You say: "IMHO, correct Unit test code must looks like : [TestMethod] public"

    Why?

  7. AT: "3. I realy like class Foo { class Tests {}} idea. But partial classes not supported across assemblies, currently the only one way is to use it is #ifdef for debug versions. "

    That’s if you believe that you should only test debug code. We don’t think that 🙂

    Testing release code is just as important.

  8. adam says:

    I’m not at all keen on shipping tests with the product. I do believe that the tests should be run against the released binaries. Which means that my tests always go in a separate dll. It also means that any code that I want to test must also go in a dll because the IDE does not allow exe’s as references.

    I see no reason that tests be public, the test methods are loaded by reflection with full trust so it makes no odds on their protection level.

    To me the idea of nested classes, or worse still, code in comments justs adds clutter to the code.

    I do think that test attributes could be extended to allow for special setup methods, parameters, etc.

    [TestFixture]

    class addTest : Assertion

    {

    // default setup

    [Setup(true)] void fullInit(){}

    [Setup] void smallInit(){}

    [TearDown()] fullTear(){}

    // default tear down

    [TearDown(true)] anotherTear(){}

    [Test("fullInit")] void testAdd1(){}

    [Test("smalllInit", "fullTear")] void testAdd2(){}

    [Test] void testAdd3(){}

    // load the resource into the

    // parameter before calling the method

    [Test, Param("ds", "resource.DsTestAdd4")]

    void testAdd4(DataSet ds){}

    }

  9. Adam: How does a nested test add clutter? Specifically because they’reinter mixed with the code?

    It’s quite possible to do this:

    File MyClass.cs

    —————–

    public partial class MyClass {

    //MyClass Stuff

    }

    File MyClass.Tests.cs

    ———————–

    public partial class MyClass {

    class Tests {

    }

    }

    That leaves the code nice and isolated and clearly shows where the tests are placed.

    Adam: Strings are smelly 🙂

    Can you think of a typesafe way to accomplish the same thing?

    Also, tangential qustion. Why are you against shipping tests with your product? (nothing wrong with that,just curious)

    Note: I don’t like the IDEs rules on exes/dlls. But I do believe that exes should just be a thin wrapper that loads a class andstarts running it. I.e. just a shim that loads another dll. In that kind of model, it’s easy to test a dll.

    Doing this also makes it easy to write an application that has a web/command-line/WinForms based UI.

  10. David Levine says:

    Cyrus,

    re: visibility

    I want the test code to be available in both debug and release builds…actually, especially in release builds, because the automated build program runs the unit tests and fails the build if a test fails. But we don’t want the test classes to be public because we don’t want to expose these classes for external parties to access them. This is especially true if the classes being tested are private, internal, nested, etc.. We also don’t want to distribute the nunit assemblies so it is especially important that code that references those assemblies not get executed.

    re: coupling

    The tests should be an integral part of the classes being tested…they could be written as nested classes within the class being tested so that if the containing class were moved/renamed the nested test classes would be moved/renamed too.

    I’d prefer it if the test code was in the same file as the code being tested. If the file gets to be huge it’s a good argument for refactoring.

    re: XML

    A method’s xml comments could be used as a series of directives on how to automatically construct test code. Argument range checks could be specified, etc. This would tie together the documentation of a method with its test code, and if the arguments to a method were to change the test code could be regenerated automatically.

    re: error handling and exception testing.

    Very little support exists for this and I believe this is the least understood and the most poorly tested areas of an application, yet is one of its most critical areas (reliability and robustness are features too). It’s difficult to compose unit tests that force coverage of all the error handling paths. Automatically generating unit tests for error handling paths would be a tremendous feature.

    re; unit tests as test jigs.

    Often times I want to load/run code to test out an external DLL, interface etc. but I don’t want to compile/run the entire application (it takes 5-10 minutes to compile/load/connect our app). I use nunit as a test jig in these special cases…it would be nice to have that facility built right into the IDE. e.g. mark a class or method as a TestJig entry point, load the assembly and call that entry point. Similar to the unit test entry point but without all the semantics associated with unit tests. It also avoids accidentally confusing testjig code with unit test code – test jig code should always be removed in a release build.

  11. adam says:

    > Specifically because they’re inter mixed with the code?

    yes

    > It’s quite possible to do this:

    > <SNIP partial class example/>

    Ok although I still don’t like it. It just doesn’t feel right (not helpful I know)

    I’ve not really had much call to use nested classes. I see several references (in this thread) to the fact that nested classes enable objects under test to be renamed during refactoring but I can’t see how this would work

    class B // was called A

    {

    class TestA

    {

    [Test]void test()

    {

    A a = new A(); //< compiler error unless refactored also

    }

    }

    }

    > Adam: Strings are smelly 🙂

    Yes :o)

    >Can you think of a type safe way to accomplish the same thing?

    No, I can’t. I vaguely thought about some sort of enum name matching with the test environment ensuring all matched but that still is not _clean_. maybe that just means that there needs to be several test classes, one for each required setup/teardown.

    >Also, tangential question. Why are you against shipping tests with your product?

    Every line of code is a liability.

    > Note: I don’t like the IDEs rules on exes/dlls. But I do believe that exes should just be a thin wrapper that loads a class and starts running it.

    I’ve never really thought about only having a void main() in the exe.

    > Doing this also makes it easy to write an application that has a web/command-line/WinForms based UI.

    Not so sure about this. I have found that it is really hard work to shift all the GUI business logic to he totally separate from the UI as in "the humble dialog" method. It’s a nice concept that I would like to strive for.

  12. I find this interesting.

    First, for quite some time, I have been embedding my tests within my class (not nested, just #ifdefed) and I have been testing private methods of my classes. Recently I have been reading Jim Newkirk and others recommendation to only test public interfaces of classes, so I started moving my tests to a separate fixture class but still within the same file as my class under test.

    I agree with Jay. I like the locality and I guess the coupling, though I had not thought of it that way. In fact I like testing my private methods. In some of the places where I have to plug into/extend a frame work, I find it difficult to do only test public interfaces. I am trying to code faster and more reliably, so I don’t mind if a few of my tests on private functions break as I change my understanding, refactor, or expand my needs.

    If you drop the test fixture attribute, then you would have to examine every class in detail looking for tests, not a bad thing, but certainly it would slow you down a bit loading the test engine. One could certainly make a strong argument that most if not all classes should have tests, so I look at this as a mixed blessing.

    I have trouble imagining a system where your tests are not encapsulated in methods. Are you looking for another set of attributes that would define your tests? Kind of like eXtensible C# http://www.resolvecorp.com/Products.aspx? (By the way I really like their work. Declarative asserts are great by me.)

  13. Radu Grigore says:

    When someone requsted exception filters in C# you replied "show me an example why this is useful". What’s your example of a useful private test?

    Sometimes it is preferable to let developers do more work to restrict the users. Another related example comes from a blog post (of ericgu, I think): non-public fields in a class should be protected only if you can think of a useful scenario in which a derived class needs access to them, otherwise they should be private. I guess it is all about being "defensive".

  14. Ron says:

    I’ll preface by stating I’m an NUnit noob.

    Like Adam, I don’t like including test code with a shipped product. I see it as code which should never run out in the wild. The only justification I can see for doing so is in the possibility these test will need to run on the client machine (which I don’t think should happen in the first place). Why would the client need to run the unit tests? Because they are having a problem? If the client really must run the unit tests, this can be mitigated by shipping them a test exe to run the same tests. But even then, you’re only proving your unit tests run successfully on this machine. The problem probably occurred because the unit test were deficient in the first place. In which case, the problem still exists on the client’s machine.

    Yes, I’m generalizing the situation some what. I guess I’m in a similar boat as Radu. I haven’t seen a clear need to include these tests in the shipping assembly.

    I do agree with the points about public vs private test methods, though. It doesn’t make any sense why these methods must be public.

  15. AT says:

    Cyrus:

    a) Can you provide example of test that must be marked as private ?

    b)

    "[Test]

    void TestBusinessObject() {

    TestCashBalance(100);

    TestCashBalance(200);

    }"

    Bad … Really bad… Test run must give you as much as possible information about all failures. Testing both 100 and 200 in same method is wrong. Once 100 failed – you will get no information about test results for 200.

    Instead of

    void TestBalance100() { TestCashBalance(100); }

    void TestBalance200() { TestCashBalance(200); }

    You must be able to specify input params in config files or UI, so can play with test data without source edit/recompile.

    For example I would like to be able test email address validator for different emails or long account creation/money transfer/deletion process. I must be able to pass email address or account number using nice UI. Once I will find inputs that fail I will add them to set of automated test runs.

    In addition to fully-manual and fully-automated testing you must be able to do semi-automated testing with manual input of params.

    c) Replace #ifdef DEBUG -> #ifdef TESTING ;o)

    adam:

    d) What the point of fullTear/littleTear specifications as attribute if you can create more flexible code like this one:



    using(TinyInit(configParams)) {

    // Do magic

    }



    Using attributes for Init/Cleanup do not allow you to pass params. For example if you use Init with 10-items common array (used by all tests) it will use it forever, you will be unable to rerun test/tests with array initialized by 1 or 100 items.

    I hope you understand my concern

  16. Adam: Thanks for the responses. As to the last part "Not so sure about this. I have found that it is really hard work to shift all the GUI business logic to he totally separate from the UI as in "the humble dialog" method. It’s a nice concept that I would like to strive for. "

    I highly recommend going down this route. It makes developing UI so much easier. The easiest way to accomplish this is to write programs and force yourself to make them accessible from the command line and a GUI. Games are a good example of this as well 🙂

  17. Jim:

    "If you drop the test fixture attribute, then you would have to examine every class in detail looking for tests, not a bad thing, but certainly it would slow you down a bit loading the test engine. One could certainly make a strong argument that most if not all classes should have tests, so I look at this as a mixed blessing. "

    I’ll do some perf tests. I’m pretty confident that that could happen iwth negligible overhead.

    "I have trouble imagining a system where your tests are not encapsulated in methods. Are you looking for another set of attributes that would define your tests? Kind of like eXtensible C"

    That’s simply because a method is such a useful container for executing tests. However, i can imagine weird things with attributes used. Or declarative tests through some sort of XAML system. However, I’m not saying "i don’t think methods should be used for tests" just that "you shouldn’t think of a method as a test"

  18. Radu: A useful private test:

    The exact same use of the public test without exposing it to all callers 🙂

    "I guess it is all about being ‘defensive’. "

    You stole the words right out of my mouth 🙂

    private test allows me to test and be defensive, while tightly coupling tests to the code they test.

  19. Ron: Thanks for the input. Just to clarify. We are not saying "tests must be shipped with release software" Merely, developers have that choice. And if we’ve chosen to ship that way, then it should be supported.

    An argument for it is TrustWorthComputing. By showing our tests to our users we achieve yet another level of transparency to help convince people that we do take safety and security seriously.

  20. Timbo says:

    Martin: you wrote "I’m thinking of a way of accepting sets of inputs and making sure that the expected results happen (either a given value is returned, underlying data is altered correctly, or an expected exception thrown)."

    This sounds like customer tests, as described in TDD with .NET (Newkirk, Vorontsov, MSPress). They use a tool called FIT that does just what you describe.

    Cyrus: I’d love to see a game that you can play on the command line or in a GUI.

    Actually, I’m working on some at home as I play with TDD. The MMORPG A Tale in the Desert has a mini-game called Tug – I used TDD and coded a mostly-complete CLI version of Tug in a single evening.

  21. Adam and Cyrus.

    Note that in VS2005 it’s possible to add a reference to an EXE.

  22. Kevin: Whoa! No way!! He’s right! I just tried in C# express. I’m so happy. Oh so happy!

  23. Milen says:

    Your blog is very interesint