Getting a managed socket to talk SSL

  • Article

This is a follow up to my previous post, where I showed a helper class I wrote to enable a managed socket to communicate via SSL on windows mobile. It was my first technical blog post and a friend gave me some very good feedback: blogs can be more interesting if they not only show the solution to a problem, but also how the solution was found. So here I’d like to tell you a little bit of what I tried before finally making SSL work on my mobile device.

I started searching MSDN and came across this article. The article talks about a native implementation using winsock and basically talks about calling setsockopt and WSAIoctl on the socket to enable SSL and register a certificate validation callback, respectively. I found the managed Socket class has corresponding methods: SetSocketOption and IOControl and at this point I thought this would be straightforward. To a certain extent it was, but I still had some work to do.

My first surprise was that SetSocketOption takes a SocketOptionName enum value as the second parameter, but this enum doesn’t have the equivalent of SO_SECURE. However, C# was nice enough to let me cast an arbitrary integer value to the enum I needed:

 

            //The managed SocketOptionName enum doesn't have SO_SECURE so here we cast the integer value

            socket.SetSocketOption(SocketOptionLevel.Socket, (SocketOptionName)SO_SECURE, SO_SEC_SSL);

 

Next in line was calling socket.IOControl to register a certificate validation callback. Turns out this method is a very general purpose one that takes a control code and then simply a buffer whose interpretation depends on such code. For SO_SSL_SET_VALIDATE_CERT_HOOK the first four bytes in this buffer are a function pointer to the certificate validation callback and the next 4 bytes are a pointer to a string containing the name of the host. Since we want to implement the validation callback in C# (more on why later) we need to first get a pointer to a delegate that we can pass to unmanaged code:

 

            hookFunc = Marshal.GetFunctionPointerForDelegate(new SSLVALIDATECERTFUNC(ValidateCert));

 

Note that it is important to maintain a reference to this IntPtr, otherwise the GC could collect it while the unmanaged code still holds a reference to it. I missed this detail the first time around and it seemed to work fine. But this was just because the GC was not reclaiming that pointer right away and that could change for any number of reasons so I was risking running into a very hard to find bug later on.

Now we need to write the host name into an unmanaged buffer, which we first need to allocate:

 

            //Allocate the buffer for the string

            ptrHost = Marshal.AllocHGlobal(host.Length + 1);

            WriteASCIIString(ptrHost, host);

 

This buffer is freed when the object is disposed since it needs to stick around until the certificate validation happens. Finally, we put these 8 bytes into a buffer and call socket.IOControl:

 

//Now put both pointers into a byte[]

            var inBuffer = new byte[8];

            var hookFuncBytes = BitConverter.GetBytes(hookFunc.ToInt32());

            var hostPtrBytes = BitConverter.GetBytes(ptrHost.ToInt32());

            Array.Copy(hookFuncBytes, inBuffer, hookFuncBytes.Length);

            Array.Copy(hostPtrBytes, 0, inBuffer, hookFuncBytes.Length, hostPtrBytes.Length);

 

            unchecked

            {

                socket.IOControl((int)SO_SSL_SET_VALIDATE_CERT_HOOK, inBuffer, null);

            }

 

That’s it as far as enabling SSL over the socket. All that remains is writing the ValidateCert function. This turns out to be relatively simple because .NET has a X509Certificate2 class that can parse the certificate blob passed to us. You can see the implementation of this function in the previous post, where I do the required validations. But I want to make a comment about a particular piece of code that had me scratch my head for a little while. This is the code that takes the unmanaged pointer to the certificate passed in and reads the data into managed code:

 

            //NOTE: The documentation says pCertChain is a pointer to a LPBLOB struct:

            //

            // {ulong size, byte* data}

            //

            //in reality the size is a 32 bit integer (not 64).

            int certSize = Marshal.ReadInt32(pCertChain);

            IntPtr pData = Marshal.ReadIntPtr(new IntPtr(pCertChain.ToInt32() + sizeof(int)));

 

            byte[] certData = new byte[certSize];

 

When I read in the documentation that size was a long, I was thinking C#, so I was trying to read the size as a 64 bit integer which gave back a huge number. Needles to say the allocation of the byte[] threw an OutOfMemoryException. What I was missing was the fact that while a C# long is 64 bits, a C++ long is only 32 bits. And now that I think about it, it wasn’t just the C++ side of my brain that was turned off at that point, but also my common sense. Why on earth would you need 64 bits to indicate the size of the certificate? :)

Lucky for me I decided to just try reading 32 bits instead and then it all worked. However, I came to the wrong conclusion that the docs were mistaken when in fact I was failing to think about the differences in data types across managed and unmanaged. It wasn’t until I was re-reading the code for this post and I realized the mistake I had made.

 

Here are links to some of the material I read when putting this together:

https://msdn.microsoft.com/en-us/library/system.net.sockets.socket_members.aspx

https://msdn.microsoft.com/en-us/library/aa916117.aspx

https://windowsmobilepro.blogspot.com/2006/03/windows-mobile-secure-socket_25.html