Visual Basic 6 and .NET COM Interop

This is a quite popular topic on the Internet, and this has a reason: a lot of developers need to integrate legacy components written in Visual Basic 6 with .NET.
COM Interop is or should be one of the easiest ways to bring these two worlds together.

The problem begins when multiple threads, apartments or custom data types float into the picture.

Today I’d like to share an easy way to tackle with data type interop, plus an unexplained discovery I made during a recent case.

The problem

My customer had a .NET method defined as follows:

public void SetData(Data[] param)

{

...

}

‘Data’ was his custom type, also defined in .NET.

When calling this method from Visual Basic 6 through COM interop, we get a run-time error: “Function or interface marked as restricted, or the function uses an Automation type not supported in Visual Basic”.

It’s all about the TLB

When data is passed from client to server or back, chances are that it must be marshaled (i.e. converted). There’re a number of conditions that dictate this, e.g. different representation of data, the need to serialize and transmit data over network, etc.

However, (automated) marshaling cannot happen without some metadata describing the actual data types that are involved. With the advent of .NET, metadata is stored right in the assembly with the code. But in terms of COM, it was stored in a Type Library (TLB). The TLB itself is just a binary representation of the Interface Definition Language (IDL). TLB/IDL is the “common language” that all COM components or clients speak. It’s worth noting that IDL is a powerful language that can express constructs which are not supported by every programming language or runtime. Also keep in mind that a TLB can be included within an EXE or a DLL as a resource.

As for Visual Basic 6, it also relies on the TLB to make the types of the component usable in the client. When doing COM interop, the TLB is generated by the .NET SDK tools tlbexp.exe or regasm.exe.

Therefore, if using a managed type via COM interop from Visual Basic 6 doesn’t work out-of-the-box, then very likely there’s a problem with the TLB/IDL. The problem is not that the TLB is invalid, since then it couldn’t have been compiled at all. Rather, the most likely issue here is that one party (e.g. .NET) expresses a type with a construct that is not supported by the other party (Visual Basic 6). Some constructs do not exist at all, while others are just expressed otherwise. In the latter case, often the difference is not just syntactical, and thus the run-time of Visual Basic 6, for example, won’t be able to handle it.

Thus, we need to look at the TLB to find the problem. Simply look up the type that you’re trying to use, and see how it was defined.

Tool support

Luckily, we don’t need to read TLBs in binary. There’s a tool included with the Windows SDK, called oleview.exe. Just pass the TLB as a command-line parameter and it will show the IDL. It will also allow passing EXE or DLL files, as long as they contain a TLB.

Finding out what’s wrong

In our case, the relevant method looked like:

[id(0x60020020)]
HRESULT SetData([in] SAFEARRAY(_Data*) param);

More often than not, we don’t have an idea about why Visual Basic 6 doesn’t like a given construct. An easy way to tell this is to create a simple ActiveX project in Visual Basic 6 and define a method or type that resembles the construct on the .NET side the most. Then, compile the project, pass the DLL/EXE to oleview.exe and check out how Visual Basic 6 expressed what you wrote in IDL.

We did just the same, and what we saw was:

[id(0x60030000)]
HRESULT SetData([in, out] SAFEARRAY(_Data*)* param);

The difference was an additional level of indirection, highlighted above.

System.Runtime.InteropServices.MarshalAsAttribute comes at the rescue

Since you have close to zero ways to customize things on the Visual Basic 6 size, you need to look at the managed code. Since now you know how the managed type or method should appear to COM consumers, you can change the .NET definition to accommodate this. Fortunately, this attribute allows to change the IDL (and the behavior of the .NET marshaler) without changing the .NET semantics. It goes far beyond the scope of this post to describe what this attribute can do, you can have a look at MSDN.

Sometimes, however, you do need to change .NET semantics, like in our case. Since we need an additional level of indirection, we need to pass the array by reference – thus use the ‘ref’ keyword:

public void SetData(ref Data[] param)

{

...

}

Bonus tip

During this recent support case with my customer, we proceeded up to this point, but still Visual Basic 6 threw run-time errors when using the modified method. After a lot of trial and errors, we discovered that invoking the method as a procedure (i.e. without parentheses) was always successful, while the function-like invoking style (i.e. with parentheses) always failed.

Seasoned Visual Basic 6 developers may be able to explain this, I could not. If anyone has an idea, feel free to add it to comments.