How to modify an Interop assembly to change the return type of a method (VB.NET)

Article
08/04/2008

Hi all,

In some situations we may need to reference a COM dll in our Visual Studio project in order to use a specific COM object within our .NET application, and thanks to COM Interop we can do it. When we add a reference to a COM dll in our project, a COM Interop assembly gets generated so we can use COM objects in the dll almost like any other .NET object. And I say almost because i.e. .NET needs to translate COM types to .NET types and it doesn't always do the conversion the way we like or need.

Let's see this with the following real example:

We are writing an Exit Module for our Certification Authority. We are trying to obtain the raw binary data for a certificate issued by our CA. We are calling CCertServerExit.GetCertificateProperty("RawCertificate", PROPTYPE_BINARY) (ICertServerExit::GetCertificateProperty Method) to achieve that. But GetCertificateProperty is returning a String and not an array of bytes, because .NET interop layer is converting a VARIANT of type BSTR (vt = VT_BSTR) to a .NET String. Because of this return type, .NET is trimming the last byte of the binary certificate when the number of bytes is odd. .NET strings cannot represent BSTRs that contain an odd number of bytes.

Fortunately we can modify the Interop assembly and choose a different return type for that method. This modification will allow us to get all the bytes in the certificate. These are the steps to do this:

1) Extract IL from Interop.CertClientLib.dll interop assembly generated by VS IDE when we added CertCli COM reference to our VS project:

ildasm.exe Interop.CertClientLib.dll /out:temp.il

2) Delete Interop.CertClientLib.dll file.

3) Delete CertCli COM reference from our project.

4) Modify temp.il and change the declaration of GetCertificateProperty in base and derived classes to take an IntPtr for the last output parameter instead of a VARIANT *:

 instance void 
GetCertificateProperty([in] string  marshal( bstr) strPropertyName,
                       [in] int32 PropertyType,
                       [out] native int pvarPropertyValue) runtime managed internalcall

5) Generate a new Interop.CertClientLib.dll which includes these modifications:

ilasm.exe /DLL temp.il /res:temp.res /out=Interop.CertClientLib.dll

6) Modify our project's references to use this new DLL.

7) Change our .NET code to use the IntPtr calling convention. Manage Variant objects from IntPtr and free the memory after its use. Sample:

 <DllImport("Oleaut32.dll", PreserveSig:=False)> _
Private Shared Sub VariantClear(ByVal var As IntPtr)
End Sub

'Exit Module Notify callback
Public Sub Notify(ByVal ExitEvent As Integer, ByVal Context As Integer) Implements CERTEXITLib.ICertExit.Notify

    Trace.WriteLine(">> Notify")

    Dim oCSE As New CERTCLIENTLib.CCertServerExit
    oCSE.SetContext(Context)

    Trace.WriteLine(String.Format("ExitEvent = {0}", ExitEvent))
    If ExitEvent = 1 Then 'Issue

        ' Allocate memory to hold VARIANT
        '
        Dim variantObjectPtr As IntPtr
        variantObjectPtr = Marshal.AllocHGlobal(2048)

        ' Get VARIANT containing certificate bytes
        '
        oCSE.GetCertificateProperty("RawCertificate", DataTypeEnum.Binary, variantObjectPtr)

        ' Read ANSI BSTR information from the VARIANT as we know RawCertificate property 
        ' is ANSI BSTR. Please note that the below code is written based on how the 
        ' VARIANT structure looks like in C/C++
        '
        Dim bstrPtr As IntPtr
        bstrPtr = Marshal.ReadIntPtr(variantObjectPtr, 8)

        Dim bstrLen As Integer
        bstrLen = Marshal.ReadInt32(bstrPtr, -4)

        Dim certBytes(bstrLen - 1) As Byte
        Marshal.Copy(bstrPtr, certBytes, 0, bstrLen)

        Trace.WriteLine(String.Format("Certificate length = {0}", certBytes.Length))

        ' Clear the VARIANT which will free the memory allocated for the VARIANT members
        '
        VariantClear(variantObjectPtr)

        ' Get certificate
        '
        Dim x509 As New X509Certificate(certBytes)

        ' Store the certificate in the certificate store
        '
        Dim store As New StoreClass
        Dim cert As New CertificateClass
        Dim data As String = Convert.ToBase64String(x509.GetRawCertData)

        cert.Import(data)
        store.Open(CAPICOM_STORE_LOCATION.CAPICOM_LOCAL_MACHINE_STORE, _
                    "My", _
                    CAPICOM_STORE_OPEN_MODE.CAPICOM_STORE_OPEN_READ_WRITE)
        store.Add(cert)

        ' Get VARIANT containing Serial Number
        '
        oCSE.GetCertificateProperty("SerialNumber", DataTypeEnum.String, variantObjectPtr)

        ' Construct Variant .NET object from IntPtr
        '
        Dim variantObject As Object
        variantObject = Marshal.GetObjectForNativeVariant(variantObjectPtr)

        ' Get Serial Number
        '
        Dim serialNo As String = variantObject

        Trace.WriteLine(String.Format("Serial Number = {0}", serialNo))

        ' Free memory
        '
        VariantClear(variantObjectPtr)

        ' Get VARIANT containing Distinguished Name
        '
        oCSE.GetCertificateProperty("DistinguishedName", DataTypeEnum.String, variantObjectPtr)

        ' Construct Variant .NET object from IntPtr
        '
        variantObject = Marshal.GetObjectForNativeVariant(variantObjectPtr)

        ' Get Serial Number
        '
        Dim distinguishedName As String = variantObject

        Trace.WriteLine(String.Format("DistinguishedName = {0}", distinguishedName))

        ' Free memory
        '
        VariantClear(variantObjectPtr)

        ' Get VARIANT containing Not Before date
        '
        oCSE.GetCertificateProperty("NotBefore", DataTypeEnum.Date, variantObjectPtr)

        ' Construct Variant .NET object from IntPtr
        '
        variantObject = Marshal.GetObjectForNativeVariant(variantObjectPtr)

        ' Get Not Before date
        '
        Dim validFrom As Date = variantObject

        Trace.WriteLine(String.Format("Not Before = {0}", validFrom.ToString))

        ' Free memory
        '
        VariantClear(variantObjectPtr)

        ' Get VARIANT containing Not After date
        '
        oCSE.GetCertificateProperty("NotAfter", DataTypeEnum.Date, variantObjectPtr)

        ' Construct Variant .NET object from IntPtr
        '
        variantObject = Marshal.GetObjectForNativeVariant(variantObjectPtr)

        ' Get Not After date
        '
        Dim validTo As Date = variantObject

        Trace.WriteLine(String.Format("Not After = {0}", validTo.ToString))

        ' Free memory
        '
        VariantClear(variantObjectPtr)

        ' Free the memory block allocated to hold VARIANT 
        '
        Marshal.FreeHGlobal(variantObjectPtr)

    End If

    Trace.WriteLine("<< Notify")

End Sub

Note: Becareful when translating the sample to C# : "Dim certBytes(bstrLen - 1) As Byte" will translate to "byte[] rawCert = new byte[bstrLen]; ".

I've also seen this kind of issues when referencing CAPICOM.dll (Interop.CAPICOM.dll interop assembly). These errors may appear when we are i.e. decrypting a string:

System.Runtime.InteropServices.COMException: ASN1 unexpected end of data.
System.Runtime.InteropServices.COMException: ASN1 bad tag value met.

For instance, CAPICOM.Utilities.ByteArrayToBinaryString returns a BSTR that represents the binary data being passed as an input parameter. The interop layer assigns the BSTR to a .NET string which removes the last byte and makes the data even. By using an IntPtr as return type we can access the complete binary data as we've already seen. In this case we'll replace "string marhal(bstr) " to "native int" just before each ByteArrayToBinaryString occurrence in the IL file.

I hope this helps.

Cheers,

Alex (Alejandro Campos Magencio)

Credits: Sample code has been provided by my colleague Prabagar Ramadasse. Thank you Prabagar!

How to modify an Interop assembly to change the return type of a method (VB.NET)

Additional resources