No Bounds For You!

A reader writes:

I am working on a client application that manipulates all sorts of COM objects. My job is to test the client application and to see if we are providing good enough support. So far I am writing test cases from the client application and testing if the COM objects are getting created or not. Since there are so many COM components out there, we are running into problem of how to test for each of them. Basically what I want to do is to test the client application for the interface it provides to the outer world.

I know COM works through two interfaces only IUnknown and IDispatch and then uses the QueryInterface to get the pointer to the other functionality supported by the interface. But the problem is how should I test this with so many controls. We also import/export data using this client application and this is another problem area. Do you have any suggestion. How guys at MS tested COM interfaces to/from say VB.

Ah, the joys of testing unbounded scenarios! Let me take each question in turn:

1) How can I verify my application correctly interacts with the COM objects it hosts?

You may or may not have an explicit spec for it, but your application has a contract with its COM objects as to what it will do when. You need to verify your application honors this contract. One solution is to write a COM object that simply reports which methods are called and what data is supplied for each call. Then you can drive your client application however you like and verify that it calls your COM object as you expect.

2) How can I verify my application correctly handles misbehaving COM objects?

This is the other side of the equation: verifying what happens when a COM object does not honor that aforesaid contract. Here also perhaps the simplest solution is to write a COM object that will do exactly what you tell it to do. You can pass these commands via some sort of script, or you could have a UI of some sort that lets you drive your custom object. Either way, you can see how your client app reacts to specific types of misbehavior.

3) How can I verify my application correctly imports data?

Once again your application has a contract with whatever is providing the data it imports. Go through that contract with a fine-toothed comb and test the combinations and scenarios your app needs to handle. You may be able to create much of this data yourself, but even when that is the case you'll want to do some testing with whichever data sources your customers use: other applications, XML transforms, perhaps even your own application! Try to get files from your customers; some will be happy to provide real live data, others will give you sanitized data, and yet others won't be willing (or able) to provide anything.

Well-formatted files are one thing, but you also need to know what happens when your app ingests rotten data. An easy way to generate loads of dirty data is to take a valid file and fuzz it: change one or more characters to something else. You can work through the file character by character, mutate values randomly, or use the file format's spec to target specific values - anything that seems likely to futz up the file. Then load each fuzzed file into your application (hopefully via an API - even targeted fuzzing will generate hundred or thousands of files!) and see what happens.

4) How can I verify my application correctly exports data?

This is straightforward: put your application into a particular state, export the data to every supported format, then verify the exported data. If your app supports custom exporters, think about writing one that simply dumps the data your app gives it. This makes for a simple way to verify your application is giving its exporters correct information, which will help you determine whether export bugs are caused by your app or a particular exporter.

5) Any other suggestions?

In every case I suggest you also review your bug history for your application. While you will likely find many of the test cases yourself, customers are wily creatures that have a tendency to (ab)use your application in ways you would never predict. Whether you find yourself running out of ideas for test cases or you just want some jumping off points, reviewing problems your customers have reported will give you a big help.

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great coding skills required.