The COPYSTREAM “problem” for Sockets
I must say that since I started working in the integration area of NAV (from within NAV), I got very interested in the possibilities that it would bring to NAV developers. Back then, I presented a Web Server for the NAS using the Socket ComCom and since then I have seen how people have used the streams and specifically the MSMQ Bus Adapter for integrating services and data.
When writing the code for the STREAMS (InStream and OutStream), I thought it would be very useful to write a function that could copy from one to another, reading from the InStream and writing to the OutStream. However, when using Sockets, there is an essential problem to completing this task. Since TCP data can contain anything (literally any data), and since the data size is also unknown and un-limited, how can we know when the copying has been completed?
Recently I was dealing with a problem that I found within one of the communities forums (yes, we are reading) and it was regarding XMLPorts. You need to stream the data to and from them, so I rapidly wrote a codeunit that would allow me to read the XMLPort to a file; for that, I used the COPYSTREAM function to read from the XMLPort to a FILE. A couple of minutes later, I found out that my file was not filled out correctly as it was truncated, and immediately went into the code to further debug the problem. I quickly found out that the problem was that the COPYSTREAM was not reading till the End of Stream (EOS). I was puzzled to find an error in the COPYSTREAM code after all this years, but of course I thought that the call to the EOS was simply missing. Once I saw that the call was in fact there, it finally hit me…. InStream.EOS is not working, or, it cannot really work.
The real problem here is, how do we know when a stream will not send us more data? Usually this is done through a protocol (like http, ftp, etc) but without a protocol, it is not possible to have a clear rule when we have the EOS. Is it a NULL termination? Is it 2 x CR+LF? Or what about a dot in an empty line? The truth is that, we could make rules, but we cannot have a generic answer to all possibilities.
Depending on your specific conditions, and specially the size of the data you are sending, you might not see this problem at all. But maybe, from time to time, you have noticed that data being received gets truncated and you are trying to figure out what went wrong.
To further illustrate the problem, let’s receive a file using Sockets and Save it on our local file (I’ll skip the sending code unit, but it can very easily be done following the samples provided in the Development Guide for Communication Components):
In the environment where I am writing this, I can send files without any issue up to approximately 25 Kb. After that, everything seems to work fine, except that the file gets truncated, that is, I am trying to send a 48 kb file, but get a truncated one.
In order to get rid of this problem, I suggest copying the stream one char at a time, so the code could look something like:
The retry is necessary, because we cannot rely on InS.EOS. InS.EOS also suffers from the fact that we don’t know when the stream will stop sending us data.
A nicer way to deal with this would be to first send how big the data is. That way, we don’t have to rely on time to assume that the information has indeed completed its transmission. If you are sending XML documents, you could also use the DOMDocument.load(InS) return value, to see if the received data indeed contains a valid XML document. If you have access to codeunit 7700, you can see an example on how this could be achieved (as you need to first copy the stream).
Of course, as mentioned before… This issue will only happen with the raw SocketBusAdapter.
Jorge Alberto Torres
Software Developer Engineer
These postings are provided "AS IS" with no warranties and confer no rights. You assume all risk for your use.