Are you getting OutOfMemoryExceptions when uploading large files?


Problem:


Using the WebClient.Upload method for posting large files will eventually leave you stranded with OutOfMemoryExceptions.


Cause:


WebClient.Upload reads the entire file to memory by default.


Resolution:


Build your own uploader.



Scenario:


One of my customers was using WebClient.Upload in a Winforms application to transfer files to a webserver. The idea in itself was fine, but when they transferred a couple of large files they’d get OutOfMemoryExceptions. When uploading a 500 MB file the application would need approximately 520 MB of memory and if you uploaded a few large files after each other you quickly hit the roof. Running GC.Collect(); after each transfer didn’t help. Judging from the number of hits on the internet for this scenario they weren’t the only ones with this problem.


This is the code they were using:



WebClient oWeb = new WebClient();
oWeb.UploadFile(“http://localhost/test.aspx”, “c:\\bigfile.cab”);


Okay, so why was this happening?
Well, first of all I wouldn’t recommend running GC.Collect(); in any application. A lot has been written on this allready, but if you’re interested in why I suggest you look at Rico Mariani’s post on the subject. Anyway, for testing purposes we ran the following instead:



GC.Collect(3);


GC.WaitForPendingFinalizers();


GC.Collect(3);


And this cleared the memory. So why isn’t this a valid solution? Well, like I said, I wouldn’t recommend using GC.Collect(); in any application, and why read the entire file to memory when you can stream it? I looked up the UploadFile-method and it seems like it does read the entire file to a byte array before posting. This is great for smaller files, but in this particular scenario it wasn’t too good. So what I did was to write my own uploader:



public static string MyUploader(string strFileToUpload, string strUrl)

{


    string strFileFormName = “file”;


    Uri oUri = new Uri(strUrl);


    string strBoundary = “———-“ + DateTime.Now.Ticks.ToString(“x”);


 


    // The trailing boundary string


    byte[] boundaryBytes = Encoding.ASCII.GetBytes(“\r\n–“ + strBoundary + “\r\n”);


 


    // The post message header


    StringBuilder sb = new StringBuilder();


    sb.Append(“–“);


    sb.Append(strBoundary);


    sb.Append(“\r\n”);


    sb.Append(“Content-Disposition: form-data; name=\””);


    sb.Append(strFileFormName);


    sb.Append(“\”; filename=\””);


    sb.Append(Path.GetFileName(strFileToUpload));


    sb.Append(“\””);


    sb.Append(“\r\n”);


    sb.Append(“Content-Type: “);


    sb.Append(“application/octet-stream”);


    sb.Append(“\r\n”);


    sb.Append(“\r\n”);


    string strPostHeader = sb.ToString();


    byte[] postHeaderBytes = Encoding.UTF8.GetBytes(strPostHeader);


 


    // The WebRequest


    HttpWebRequest oWebrequest = (HttpWebRequest)WebRequest.Create(oUri);


    oWebrequest.ContentType = “multipart/form-data; boundary=” + strBoundary;


    oWebrequest.Method = “POST”;


 


    // This is important, otherwise the whole file will be read to memory anyway…


    oWebrequest.AllowWriteStreamBuffering = false;


 


    // Get a FileStream and set the final properties of the WebRequest


    FileStream oFileStream = new FileStream(strFileToUpload, FileMode.Open, FileAccess.Read);


    long length = postHeaderBytes.Length + oFileStream.Length + boundaryBytes.Length;


    oWebrequest.ContentLength = length;


    Stream oRequestStream = oWebrequest.GetRequestStream();


 


    // Write the post header


    oRequestStream.Write(postHeaderBytes, 0, postHeaderBytes.Length);


 


    // Stream the file contents in small pieces (4096 bytes, max).


    byte[] buffer = new Byte[checked((uint)Math.Min(4096, (int)oFileStream.Length))];


    int bytesRead = 0;


    while ((bytesRead = oFileStream.Read(buffer, 0, buffer.Length)) != 0)


        oRequestStream.Write(buffer, 0, bytesRead);


    oFileStream.Close();


 


    // Add the trailing boundary


    oRequestStream.Write(boundaryBytes, 0, boundaryBytes.Length);


    WebResponse oWResponse = oWebrequest.GetResponse();


    Stream s = oWResponse.GetResponseStream();


    StreamReader sr = new StreamReader(s);


    String sReturnString = sr.ReadToEnd();


 


    // Clean up


    oFileStream.Close();


    oRequestStream.Close();


    s.Close();


    sr.Close();


 


    return sReturnString;

}

 


One of the things worth noting is that you need to set oWebrequest.AllowWriteStreamBuffering = false; Otherwise you will read the entire file to memory anyway. This is because the default behavior of the WebRequest is to buffer the entire request in case it needs to re-send it due to authentication, connectivity problems, etc. Again, this is a default behavior that normally is a performance boost, but in this case is a performance killer.


 


So what was the end result?


During my first test runs the application needed as much memory as the file I was trying to upload, and then some. So in order to upload a 500 MB .cab-file the application needed at least 520 MB. The application using the custom uploader never went above 23 MB.


End of transmission


/ Johan

Comments (38)

  1. Thanks bunch! Hungry for more!!!!

    Please keep writing :)

    Rahul

  2. Getting OutOfMemoryExceptions when uploading large files is a very common problem and we troubleshoot

  3. Patrick says:

    Hey Johan,

    nice post, but you should complete your code with Closing and Disposing calls.  

    Is a common mistake in many developers that they dont call those methods when a type has them, specially those related with file and streams.

    bye

  4. JohanS says:

    Hi Patrick,

    You’re absolutely, 100% right. I can’t believe I missed this. The code should be fine now.

    I keep telling my customers the importance of calling close and dispose and fail to do this myself in my sample code. – Well, I think I have the topic for my next post. :)

    Thanks for the heads up!

    / Johan

  5. Mike says:

    Can you also list the code for the .aspx page that this uploader would post to and any server-side configurations that should be considered? for example, the maxRequestLength or anything else… Thanks! Already a great help!

  6. JohanS says:

    Hi Mike,

    The file is uploaded "the normal way", so you deal with it as you would with any file uploaded through <input type="file>

    For more info on how to do that, as well as what to take into consideration regarding file size, please take a look at http://support.microsoft.com/kb/323245

    / Johan

  7. Gagan says:

    Hi Johan

    I tried your "MyUploader" code to upload files. Its going thru all the procedure. There is no error, but it is also not uploading the file!. Am i missing something? Do i need to configure something?

  8. JohanS says:

    Hi Gagan,

    The name of the file is simply "file", so if you just change that, you should be able to use the code from http://support.microsoft.com/kb/323245

    Using Request.Files(0) should work fine as well.

    Additional troubleshooting tips:

    Try setting a breakpoint in the code you’re using to receive the file.

    Add a watch on "Request.Files".

    Take a look at the values under the "AllKeys"-property. You should see a key named "file".

    It would also be interesting to add a watch on Request.Files(0). It’s length should definitively be > 0.

    Let me know if this works out or if you need additional help.

    / Johan

  9. Thanks! You’re a savior. I had already written code to upload my file piecemeal, but I missed the mysterious AllowWriteStreamBuffering property (which *sounds* like a good thing). Easiest bug fix ever!

    Thanks again.

  10. warenbe says:

    Hi

    thank you for this solution i was wondering why i got an exception when trying to send large files

    now i have a problem with your solution: when i try to send a file larger than 10MB (at 110 kB/s) i have an exception:

    "the request was aborted: the request was cancelled"

    when i use a packet sniffer to see what happening i see that after 10mB, the server send "error" and close the connection but i don’t know why. have you got an idea?

    thank you and sorry for my english

  11. warenbe says:

    well sorry for the double post i found the solution:

    changing the oWebrequest.Timeout value :)

  12. Dave Nicoll says:

    I actually stumbled across this looking for a solution to the 100 second timeout when using the .net WebClient. I ripped out the webclient code, added your function, and added the following line to prevent timeouts…

               oWebrequest.KeepAlive = false;

    Works great!

  13. joe says:

    I’m getting an error when trying to implement this code:

        This request requires buffering data to succeed

    Google search says that this is an anonymous authentication problem with IIS… but I am using a secured page on an Apache server.

    Any idea?

  14. JohanS says:

    Hi Joe,

    I’m afraid I don’t know much about Apache.

    I’d try the standard troubleshooting stuff, such as:

    * Attempt to contact a page using anonymous access

    * Try a different web server, IIS, etc. with both anonymous and authenticated requests

    * Run Netmon / Fiddler2 in each scenario and compare the logs

    / Johan

  15. John says:

    Hi

    I tried to use your method , but why i get back on the string is the webpage i put in the uri, not textfile i am trying to upload, am i doing anything wrong?

    John

  16. JohanS says:

    Hi John,

    sReturnString will contain whatever the webpage you’re uploading to decides to return in response.

    / Johan

  17. Tony Ha says:

    Hi Joe,

    This is a great post, it works fine with small file, but i have problem with a big file (50M), it throw an exception, here is error message

    "The request was aborted: The request was canceled."

    Do you have any advice?

    Thanks for your help

    Tony

  18. Tony Ha says:

    OK, I fixed the issues today, now I can upload a large file (up to 1G), here is my update code:

     // keep session for a long time, without timeout

      req.KeepAlive = false;

      req.Timeout = Timeout.Infinite;

      //use protocol version 10 instead of 11 by default

      req.ProtocolVersion = HttpVersion.Version10;

    Hope this help for anyone else have same problem with me

    Thnks,

    Tony

  19. Roey says:

    hi, great article, but what if want to upload more than one file??

  20. JohanS says:

    Hi Roey,

    I’m not sure I follow? What would prevent you from calling the function a second time?

    / Johan

  21. Thank you for blogging about the AllowWriteStreamBuffering property.

  22. WKKF says:

    According to RFC 1867 the trailing boundary MUST end with — . Otherwise the upload will fail on certain web servers.

    Replace:

    byte[] boundaryBytes = Encoding.ASCII.GetBytes("rn–" + strBoundary + "rn");

    by:

    byte[] boundaryBytes = Encoding.ASCII.GetBytes("rn–" + strBoundary + "–rn");

    Good job otherwise.

  23. Jose says:

    Everyone seems to post only one file at a time. The challenge I have is streaming multiple files…  can this be done?

  24. JohanS says:

    @Jose:

    Well, like I’ve said before: What prevents you from calling the function a second time?

    / Johan

  25. Fernando says:

    Hi, Nice Code !

    What can be done in case your internet is gone while oRequestStream.Write(buffer, 0, bytesRead);

    i have a try catch, so it goes to catch with the exception "Unable to read data from the transport connection". Then Let’s suppose that i have a Thread.Sleep for 5 minutes, during that time the internet is back and i want that continues writing where it was, but i continue getting the exception.

    Thanks,

    Fernando

  26. JohanS says:

    @Fernando:

    In that case the connection has been closed, so you’d need to restart. To avoid having to restart completely you’d have to write your own method for dividing the transfer into suitable packets which you would then assemble into the final file.

    / Johan

  27. Fernando says:

    @Johan:

    Thanks for the answer, i was looking for a way to avoid that solution but looks like it can’t. I really appreciate your help !

    Thanks,

    Fernando.

  28. 話題の小向美奈子ストリップを隠し撮り!入念なボディチェックをすり抜けて超小型カメラで撮影した神動画がアップ中!期間限定配信の衝撃的映像を見逃すな

  29. Anuja says:

    Thanks for your findings. This helped my team.

  30. Kirill says:

    Thank you!

    Because of System.OverflowException uploading of large files I replaced

    byte[] buffer = new Byte[checked((uint)Math.Min(4096, (int)oFileStream.Length))];

    by:

    byte[] buffer = new Byte[checked((ulong)Math.Min(4096, (long)oFileStream.Length))];

  31. Jairo Portela says:

    Hi there Johan:

    I tried your code but I'm getting this error on IIS Windows XP SP2:

    "This request requires buffering data to succeed"

    Any ideas?

    When I set AllowWriteStreamBuffering to true it works, but that's not the idea right?

  32. Jairo Portela says:

    I found the problem, Authentication was required, now it's ok.

  33. AlienJun says:

    ho

    ,httpHeader, must have "'""  ???  I didn't understand.

  34. Mobileboy36 says:

    Very nice post, thank you.

    I have a question about the server side:

    Can you list a code example for the .aspx page that this uploader would post to? (I don't know how to handle this, the link support.microsoft.com/…/323245 does not work anymore)

    Thank you!

  35. Mobileboy36 says:

    On the server side, I have an aspx file but this doesn't work in combination with your method.

    Do you have a piece of server code illustrating the way to do it.

    Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load

           Dim f As String

           Dim file

           Dim SaveAsPath As String

           For Each f In Request.Files.AllKeys

               file = Request.Files(f)

               SaveAsPath = Server.MapPath("uploadedfiles/") & file.FileName

               'file.SaveAs("c:inetpubtestUploadedFiles" & file.FileName)

               file.SaveAs(SaveAsPath)

           Next f

       End Sub

  36. pnsunil89 says:

    Hi, Can I get a little help here,

    I am getting the the below exception

    {System.Net.WebException: The remote server returned an error: (405) Method Not Allowed.

      at System.Net.HttpWebRequest.GetResponse()

    It happens when i try to call the below method

    System.Net.WebResponse oWResponse = oWebrequest.GetResponse();

  37. Pravin Bhati says:

    While using below code

    byte[] bytes= new Byte[checked((uint)Math.Min(4096, (int)fls.Length))];

    Is there any chance of loosing the data.?  because if we upload file of 5 mb it will only save the 4096 bytes.

  38. Geo says:

    I get the following exception while uploading a 2 MB file to a SharePoint site "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host."

    Please help me…..