Troubleshooting PInvoke Related Issues

I am back with some more PInvoke Stuff. Recently I was working on a PInvoke issue which I found interesting.

I have a C++ dll which has a function whose signature is

int TestFunc(IN_STRUCT in_Params, RET_STRUCT * pret_Par).

I wanted to call this function from C#. Function has two arguments. First argument is input structure which will be filled from C# code and passed to C++ code. Second argument is output structure which is filled in C++ code and output to C# code.

Here are the C struct definitions and a function that needs to be marshaled

#include "stdafx.h"

#include <stdio.h>

#include "Objbase.h"

#include <malloc.h>

typedef struct IN_STRUCT

{

      BYTE CMD_PType;

      BYTE CMD_PTType;

      BYTE CMD_OC;

      BYTE CMD_Seq;

     

};

typedef struct RET_STRUCT

{

  

      BYTE RET_OC;

      BYTE RET_Seq;

      BYTE RET_RetBytes;

      char *str;

      BYTE RET_PD[10];

     

};

extern "C" __declspec(dllexport) \

        int TestFunc(IN_STRUCT in_Params, RET_STRUCT * pret_Par)

{

      int iRet = 0;

      pret_Par->RET_OC = in_Params.CMD_OC;

      pret_Par->RET_Seq = in_Params.CMD_Seq;

      pret_Par->RET_RetBytes = 6;

      pret_Par->RET_PD[0] = 0;

      pret_Par->RET_PD[1] = 10;

      pret_Par->RET_PD[2] = 20;

      pret_Par->RET_PD[3] = 30;

      pret_Par->RET_PD[4] = 40;

      pret_Par->RET_PD[5] = 50;

      pret_Par->str = new char(30);

      strcpy(pret_Par->str,"This is sample PInvoke app");

      return iRet;

}

Managed Structure equivalent to Native Structure:

namespace ConsoleApplication1

{

    class Program

    {

      //This structure will be filled up by C++ Test.dll and returned back

      With values to C# code.

           

       [StructLayout(LayoutKind.Sequential)]

        public struct RET_STRUCT

        {

            public byte RET_OC;

            public byte RET_Seq;

            public byte RET_RetBytes;

            [MarshalAs(UnmanagedType.LPStr)]

            public String RET_STR;

            [MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]

            public byte[] RET_PD;

        };

        //The values of this structure will be used to fill up IN_STRUCT and

          passed to C#

        [StructLayout(LayoutKind.Sequential)]

        public struct IN_STRUCT

        {

            public byte CMD_PT;

            public byte CMD_PTType;

            public byte CMD_OC;

            public byte CMD_Seq;

          

        };

        //C++ dll containing the func

        [DllImport("Test.dll")]

        public static extern int TestFunc(IN_STRUCT i, ref RET_STRUCT r);

        static void Main(string[] args)

        {

            IN_STRUCT cmd_params = new IN_STRUCT();

            RET_STRUCT ret_Params = new RET_STRUCT();

            //Fill up the cmd_params

            cmd_params.CMD_OC = 0x02;

            cmd_params.CMD_PTType = 0x00;

            cmd_params.CMD_Seq = 1;

          

            //Call the C++ function to fill ret_params

            int iRet = TestFunc(cmd_params, ref ret_Params);

         //Print out the returned values

            Console.WriteLine("Returned Values\n");

            Console.WriteLine(ret_Params.RET_OC + " " + ret_Params.RET_Seq +

                " ");

            for (int i = 0; i < ret_Params.RET_RetBytes; i++)

                Console.WriteLine("\n" + ret_Params.RET_PD[i]);

           Console.WriteLine(ret_Params.RET_STR);

            Console.ReadLine();

           

        }

    }

}

After executing the code I was expecting a valid output. But I ended up with Access Violation. I used windbg to troubleshoot this issue.

I spawned exe from windbg and tried to see call stack.

0:000> kv

ChildEBP RetAddr Args to Child

002cec30 76fc5883 006515c8 00000001 00000000 ntdll!RtlpLowFragHeapFree+0x31 (FPO: [0,10,4])

002cec44 76b9c56f 000b0000 00000000 049a3a48 ntdll!RtlFreeHeap+0x101 (FPO: [3,0,4])

002cec58 7565dc2c 000b0000 00000000 049a3a50 KERNEL32!HeapFree+0x14 (FPO: [3,0,0])

002cec6c 7565dc53 7573e6f4 049a3a50 002cec88 ole32!CRetailMalloc_Free+0x1c (FPO: [2,0,0])

002cec7c 6c7e8410 049a3a50 002cec9c 6c8084bd ole32!CoTaskMemFree+0x13 (FPO: [1,0,0])

002cec88 6c8084bd 00109d34 00000001 00109d48 mscorwks!FieldMarshaler_StringAnsi::DestroyNativeImpl+0x16 (FPO: [1,0,0])

002cec9c 6c8088e5 00109d30 0065340c 1670b1d2 mscorwks!LayoutDestroyNative+0x3a (FPO: [2,0,0])

002cee8c 6c73539b 002cef58 00000000 1670b182 mscorwks!CleanupWorkList::Cleanup+0x2ea (FPO: [2,116,4])

002ceedc 001cad4c 002cef18 01020000 00109d30 mscorwks!NDirectSlimStubWorker2+0x120 (FPO: [1,12,4])

WARNING: Frame IP not in any known module. Following frames may be wrong.

002cefa4 6c7013a4 00700876 002cefd8 00000000 0x1cad4c

002cefe0 6c6f1b4c 010d2816 00000003 002cf070 mscorwks!PreStubWorker+0x141 (FPO: [1,13,4])

002ceff0 6c7021b1 002cf0c0 00000000 002cf090 mscorwks!CallDescrWorker+0x33

……….

0:000> da 049a3a50

049a3a50 "My Name is Jyoti Patel"

From the call stack it’s clear that InteropMarshaller ( NDirectSlimStubWorker2) is trying to deallocate string using CoTaskMemFree.

There are two solutions to this problem.

1. As deallocation is done using CoTaskMemFree, Allocate the memory using CoTaskMemAlloc.

Changing line of code in C++ from

pret_Par->str = new char(30);

pret_Par->str = (char*)CoTaskMemAlloc(30);

Issue was resolved.

(When a string buffer allocated by native code is marshaled to managed code, CLR Interop marshaller will allocate a managed string object and copy the contents of native buffer to the managed string object. Now in order to prevent a potential memory leak, CLR Interop Marshaller will try to free allocated native memory. It does so by calling CoTaskMemFree. The decision to call CoTaskMemFree is by-design. This can at times lead to crash, if memory was allocated by the called-native function using any API other than CoTaskMemAlloc family of API’s as custom allocators may allocate on different heaps.)

 In this case, memory was allocated by malloc, and ends up being freed by CoTaskMemFree and hence we see an AV)

2. If you are not able to change the C++ code and you want to allocate memory using new/malloc other solution is to use Intptr and do custom marshalling by calling corresponding methods from Marshal class to do marshalling.

[StructLayout(LayoutKind.Sequential)]

        public struct RET_STRUCT

        {

           

            public byte RET_OC;

            public byte RET_Seq;

            public byte RET_RetBytes;

            public IntPtr RET_STR;

            [MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]

            public byte[] RET_PD;

          

        };

……….

………

………

Console.WriteLine(Marshal.PtrToStringAnsi(ret_Params.RET_STR));

Jyoti Patel and Manish Jawa

Developer Support VC++ and C#