Streaming large files with WSE
We have a few articles that describe how to enable large file transfers with WSE 3.0(see references section below).
The main thing to note here is that WSE by default buffers the attachments before they are sent, so when the file size is large(typically anything above 10MB can be considered as large), this can lead to high memory usage and can also cause a System.OutOfMemoryException to be thrown.
This condition worsens as the file size increases or the number of simultaneous requests increases.
The reason for this is that the entire amount of data that will be sent in the response is buffered into the Web service computer's memory. This means that the process that is serving the file needs to be able to allocate one contiguous block of memory to accommodate the buffer. If the process cannot find a large enough contiguous block it throws an System.OutOfMemoryException.
Depending on the current state of the process memory, it may be able to accommodate a fairly large buffer or may not be able to accommodate even a small buffer.
So, the best way to avoid this issue is to stream the files. There is a sample for streaming using WSE 3.0 at "WSE Installation folder\Samples\CS\QuickStart\Basic\BinaryDataMTOM" (usually this maps to C:\Program Files\Microsoft WSE\v3.0\Samples\CS\QuickStart\Basic\BinaryDataMTOM).
This requires writing a custom class that implements the IXmlSerializable interface. Additionally, you must stream the MTOM attachments by using the WriteXml method
NOTE: This sample has been updated with the hotfix https://support.microsoft.com/kb/943508 to include an implementation on how to stream from client side as well.
Note:
1) Streaming cannot be implemented when you use message-level security because the entire soap message needs to be buffered into the memory to perform the cryptographic operations. So, if you want to use security the only option is to use tansport-level security(SSL).
2) The BinaryDataMTOM sample says that the streaming can be done only on the server side and not on the client and on the client the whole file is buffered. But, there is a hotfix https://support.microsoft.com/kb/943508 that provides the streaming on client side. You would need to apply this hotfix and then, you must set the SendChunkWithoutBuffering boolean property to the true value in the WSE Web service proxy class. The sample now has a RunPutStreaming method that does this.
Scenarios:
1) Sending files from client to a webservice. (Upload)
a) secured : No streaming support using message-level security.
b) unsecured: Streaming is supported.
2) Sending files from webservice to client. (Download)
a) secured: No streaming support using message-level security.
b) unsecured: Streaming is supported
References:
How to: Stream Large Amounts of Data from a Web Service: https://msdn.microsoft.com/en-us/library/aa528818.aspx
How to: Enable a Web Service to Send and Receive Large Amounts of Data: https://msdn.microsoft.com/en-us/library/aa528822.aspx