Creating something from nothing, asynchronously [Developer-friendly virtual file implementation for .NET improved!]


This blog has moved to a new location and comments have been disabled.

All old posts, new posts, and comments can be found on The blog of dlaa.me.

See you there!

Comments (60)

  1. swythan says:

    Would it be possible to ensure that the StreamContents callback was called on the UI thread (or at least when the UI thread wasn’t blocked).

    I’d like to put a screenshot of some of the UI into the virtual file, but of course I can’t access any WPF elements from anything but the UI thread. Grabbing the bitmap before calling DoDragDrop works, but of course that’s means we always take the perf hit.

    Maybe I should stop worrying. I’d probably want to (eventually) use the bitmap as visual feedback for the DnD operation anyway.

  2. Delay says:

    swythan,

    I considered trying to do that. But then I decided that was probably out of scope for VirtualFileDataObject because there’s no way of knowing what the developer intends to do in their begin/end actions. Like I mentioned in the article, an MVVM approach is probably best here – and for that there’s usually no need to be on the UI thread. Therefore, it seemed like forcing the issue in VirtualFileDataObject would not really be addressing the 80% case very well. And because it’s so easy for the developer to get to the UI thread, I chose the current approach for its simplicity and generality.

    I’d expect that you can do what you want within in the current framework (perhaps via Dispatcher.BeginInvoke), but if you find that you can’t, that would be great feedback to have.

    Thanks for taking the time to write – I hope this is helpful!

  3. onovotny says:

    How about stroring the SynchronizationContext.Current in the ctor instead of a dispatcher?  There could be a ctor parameter or property to control if it should be used.

    That way the the user can tell the object to either marshall callbacks or not and the object doesn’t need a reference to any dispatcher.  

  4. Delay says:

    onovotny,

    The use of Dispatcher in the example above is part of the sample application, not part of the VirtualFileDataObject control; I have specifically tried to keep threading issues out of VirtualFileDataObject.

    I’m not sure from reading your suggestion whether you’re proposing a change to the sample or to VirtualFileDataObject itself. If you’re just talking about the sample, your suggestion sounds like it would work as well. :) I think the current way is a bit clearer for a sample, but your way is probably a more practical solution.

  5. …having to create your own IDragSource. this also provides access to the drag drop visuals feature demonstrated in the Win7 SDK DragDropVisuals sample

    http://code.msdn.microsoft.com/Project/Download/FileDownload.aspx?ProjectName=shellapplication&DownloadId=6769

    there is a managed version of this as well found here

    http://blogs.msdn.com/adamroot/pages/shell-style-drag-and-drop-in-net-wpf-and-winforms.aspx

  6. chris_guzak@msn.com says:

    messed up my post above… the title meant to be "use SHDoDragDrop to avoid…"

  7. Delay says:

    Chris,

    That’s a great suggestion! In my case, IDragSource was pretty easy to write, so I don’t feel too, too silly. :) But you’re right that folks should be aware of the SHDoDragDrop API because it seems like another good option: http://msdn.microsoft.com/en-us/library/bb762151(VS.85).aspx

    Thanks for writing!

  8. Tyler says:

    Cool program… but it doesn't seem to work in Visual Studio 2010.  it throws a exception

    Invalid FORMATETC structure (Exception from HRESULT: 0x80040064 (DV_E_FORMATETC))

  9. Delay says:

    Tyler,

    Folks report that problem to me every now and then and so far it has always been because they're running under the debugger – if they run without debugging (Ctrl-F5) instead, it works just fine. Also, I'm pretty sure hitting F5 to continue after the exception (which is handled) also works fine. Maybe VS 2010 has a different setting for catching first chance exceptions, or maybe everyone's just forgotten that they disabled them in VS 2008 a long time ago. :) Please give that a try and let me know if you're still having trouble.

    Thanks!

  10. Steve says:

    Hey Delay!  Great blog!

    This is exactly what I was looking for; I need to support dragging and dropping of virtual files which need to first be transferred from a hardware device via a slow serial connection.  

    I also get the exception noted above in VS2010 it's caused by the call to Marshal.ThrowExceptionForHR( ) method in theVirtualFileDataObject::GetData( ) method.  It seems to result in an InteropServices.COMException exception being thrown which for whatever reason doesn't get caught by the exception handler.  

    You can disable break on user unhandled System.Runtime.InteropServices.COMException in the Debug | Exceptions dialog to stop this.  

  11. Steve says:

    Delay,

    Just out of interest would it be possible to implement a graceful way of cancelling an asynchronous virtual file transfer?

  12. Delay says:

    Steve,

    Great tip, thanks! Regarding the question about cancellation, I don't see anything explicit, but the docs say to call IAsyncOperation::EndOperation at completion, so I'm thinking maybe you could call that prematurely? Alternatively, consider returning a failure code from any method that gets called during the transfer – I vaguely remember that would cause an abort when I was developing this sample. :)

  13. vhanded says:

    Hi, great piece of code! Now I have a problem: can current code works for multiple files? I saw your previous article has 1 "You can drag multiple virtual objects, you know" article, but the native language is scary for me. Any shortcuts?

    Much appreciate.

  14. vhanded says:

    ok, so I found out that SetData can accept an array of DataObject. problem solved. Thanks for the code anyway.

  15. Delay says:

    vhanded,

    Glad you got that worked out!

  16. Scott Myers says:

    Thank you very much for this article. It is very well written and thought out.

    I am working on code that accepts a Drop from the VirtualFileDataObject class. I have figured out how to make a GetData call with FileGroupDescriptorW and parse out the array of FileDescriptor structures that is returned. I am having trouble figuring out how to get the corresponding file data.

    I've tried making a GetData call with "FileContents" but all I get back is null. Can you give me any ideas how to do this?

  17. Delay says:

    Scott Myers,

    Wow – it's been a while since I wrote this code – and I didn't need to handle the receiving end at the time… :) That said, as I look at the code now, it appears that the "FileContents" data may be special because it's indexed (FORMATETC.lindex != -1) because there may be multiple representations of the same data. So my suggestion would be to check the call you're making to GetData and be sure it's configured correctly for this. Alternatively, maybe try handling a file drop from something "real" like Windows Explorer just in case I've overlooked something in my code.

    Thanks for the kind words – I hope this helps! :)

  18. John Lamberis says:

    This is a fantastic piece of work and exactly what I have been trying to assemble.

    I cannot seem to get this ported over to VB.NET – what are the chances of seeing a version of this code in VB? (saying that is like a Hail Mary pass)

  19. Delay says:

    John Lamberis,

    Thanks for the kind words! :) Regarding a full VB port, I'm afraid that's not something I feel likely to get to given how many other things are on my list. However, if you've got it mostly done and have one or two specific questions, I *may* be able to help. And if you get the whole thing ported, I'd be happy to link to your work so others could use it!

    But you might also consider simply compiling my C# implementation into an assembly and then referencing that assembly from your VB.NET project and using it just like any other .NET assembly. Aside from sticking this code in its own assembly, you shouldn't need to care that it's C# as that will all be hidden away in its own assembly.

    Just a thought… :)

  20. Matt Winckler says:

    Thanks for this code; it's extremely helpful.

    One improvement I have to suggest: implement IStreamWrapper.SetLength to set the size of the IStream, like so:

    /// <summary>

    /// Sets the length of the current stream.

    /// </summary>

    /// <param name="value">The desired length of the current stream in bytes.</param>

    public override void SetLength(long value) {

    _iStream.SetSize(value);

    }

    When using the VirtualFileDataObject to transfer very large files, setting the length beforehand to a reasonable size makes an enormous difference in speed–else IStream.Write can end up being unacceptably slow. (Of course, it's a tradeoff with memory usage…)

  21. Delay says:

    Matt Winckler,

    Thanks, Matt, this sounds like a great suggestion! I've added it to my TODO list to investigate if/when I revisit this topic.

  22. Nick says:

    David, great work. Unfortunately though I have noticed that the operation occurs synchronously when dragging a virtual file into an Outlook message (drag and drop to windows explorer occurs asynchronously as expected). As this is the case in your sample I don't believe this to be due to my implementation, do you have any ideas as to why this is or how it might be avoided?

    Thanks

  23. Delay says:

    Nick,

    Thanks! Regarding the behavior you're seeing, what I would do first is check the value of the IsAsynchronous variable when the synchronous operation is started. IsAsynchronous defaults to true, but Outlook may be calling IAsyncOperation.SetAsyncMode to set it to false and that would lead to the behavior you're seeing, I think.

  24. Steve says:

    How do I implement this with drag and drop within a WPF application?  My app need to drag to the File Explorer and drag items with in itself.  I can't figure out how to make  VirtualFileDataObject.DoDragDrop work with my existing drag drop code.

  25. Delay says:

    Steve,

    If the drag source and destination are both within the same application, can you detect that and bypass the logic here to manage the entire operation internally? That's probably what I'd try (perhaps via a custom drag object or clipboard format?), though I haven't actually done so myself and may be oversimplifying. :)

  26. Joe says:

    I'm trying to do the same as Steve:

    Drag an object either to the desktop or within the application.

    The problem is that i can do nothing with the virtualfiledataobject inside the drop handler of the application because i cannot convert the drop args there into the neccessary object.

    Also i cannot detect when i drag the object, where it would be dropped on: onto the explorer or onto the application.

    I have tried to call windows dragdrop and virtualfiledataobject drag drop at the same time but that does not work also.

  27. Delay says:

    Joe,

    Let's deal with one thing at a time. You say you are having trouble dragging things to the desktop, but my sample application seems to work fine for that. Try dragging either of the bottom two items onto the desktop – a file will be created with the appropriate contents. Is this not working for you, or is there more to it?

  28. Chevon says:

    Question,

    Is there a way to support dragging and dropping from the WPF application to another app (not windows explorer) such as Chrome, Evernote, etc.?

    I have been half successful with my question since I am able to drag to the Windows Live Mail application and Outlook for attachment, but those are the only two that have worked for me.

  29. Delay says:

    Chevon,

    The sample application already demonstrates dropping into another application. :) Dragging the "Text and URL" item onto Chrome or Internet Explorer will open that URL in the browser. (Dragging the "Text only" item onto Chrome's address bar will initiate a search for that text.)

  30. Chevon says:

    David,

    Thanks for the tips. Worked great!

    However, I would like to go a little bit further and allow files (virtual or not) to be dragged to Chrome/Gmail, as is done with Windows Explorer. Do you know the steps I can take to achieve this?

    I believe I need to specify the FileDrop DataFormat, along with an array of strings/filenames. However, your code does not current accept this format.

  31. Delay says:

    Chevon,

    What you suggest around using DataFormats.FileDrop sounds about right to me. I'd expect that could be added to the sample implementation to get the effect you want.

  32. David,

    Loving this work. I'm using it to rename existing files as they are "dropped" into a new location. It's sluggish for large files using the following code:

    new FileDescriptor

       {

           Name ="SOME NEW NAME"

           StreamContents =stream=>{

               FileStream fs = File.OpenRead(actualFilePath);

               fs.CopyTo(stream);

               fs.Close();

           }

       }

    Can you think of a faster alternative?

    Steve

  33. Delay says:

    woodced1979,

    What you're doing looks pretty efficient to me. :) One thing you might do is wrap the FileStream in a "using" which would ensure it's Dispose-ed and also avoid the need for you to call Close. Something else you *might* consider is doing a P/Invoke out to the CopyFile Windows API because that *may* be able to perform the copy a bit more efficiently.

  34. David,

    Thanks for the response.

    I'm finding that a 46Mb file takes a minute or so versus a second or so using the built in System.Windows.DataObject..

    I'm not sure I quite understand you with regard to the CopyFile Windows API method. Where abouts do you think I would implement this?

  35. Delay says:

    woodced1979,

    I'm suggesting that calling msdn.microsoft.com/…/aa363851(v=vs.85).aspx instead of OpenRead/CopyTo *may* be more efficient because it allows the OS to do the work for you and it has all kinds of resources that stream read/write can't take advantage of. That said, it's going to take time to write 46Mb to disk, so this is unlikely to be instantaneous. The speed you're seeing from DataObject may be a result of it taking some shortcuts of its own (ex: if the data is in memory, it doesn't need to be read from disk).

  36. Joe says:

    I have a list of objects. I use .net's DragDrop.DoDragDrop to drop an item of that list to another list inside my application. In the drop handler of the list I drop the item on, I get an object of the list item's type and everything is ok.

    Now I want to drop that item also on the desktop. So I use the VirtualFileDataObject's dragdrop operation to do so. This works also without problems.

    The problem is that I want to be able to drop the item on the other list AND to the desktop. I do not know how to manage this. When I use the dragdrop operation of the Virtualfiledataobject, I get an object of type FileGroupDescriptorW and FileContents in the list's drop handler. How can I get the item out of that?

  37. Delay says:

    Joe,

    You can register multiple types with IDataObject This is how the sample app supports text+URL+file. My thinking is that you can try to include a custom type which contains the info to get back to your list item. (It probably can't be the item itself because of the native/managed transition, but an index should be just fine.)

    Hope this helps!

  38. Joe says:

    Thank you David,

    it works!

    I just serialized my object as byte array, just as you did in the example and defined an own DropType so I can identify my object.

    With this I can deserialize the byte array in the OnDrop Handler of my list so I can get my object.

    Great Work!

  39. Alexandre N says:

    By me it crashes with exception COMException was unhandled by user code, Error in the structure FORMATETC (Exception HRESULT: 0x80040064 (DV_E_FORMATETC)). FORMATETC is taken from namespace System.Runtime.InteropServices.ComTypes. Is there easy way to fix it?

  40. Tom Colvin says:

    David,

    Great article that has really helped me – thanks a lot :)

    I'm finding a potential scale issue. The stream writes for the VirtualFileDataObject take progressively longer as the file is written. I am writing up to 128kb to the stream with each write. The first few writes take about 8ms, but after 200MB or so they are taking more like 90ms.

    This means a 20MB file is written almost instantly whereas a 200MB file takes several minutes – much more than the expected 10x longer.

    Do you know why this may be?

    Thanks

  41. Delay says:

    Tom Colvin,

    Glad you liked it! Regarding the behavior you're seeing, I'm not sure what's going on offhand. My guess based on how you describe it is that something is trying to buffer the entire stream as it's being written. (I looked again at my code briefly and it doesn't seem like it would be guilty of doing that.) It might be interesting to step through the code or profile the memory use of your app in this scenario or even try a simple (yeah, I know this code isn't simple…) native code test app to check if it has the same behavior.

    Hope this helps!

  42. Kei says:

    How can I get Dropping location?

  43. Delay says:

    I don't see that you can – it's likely to be in a different process about which the drop source knows nothing, so it's not clear what kind of information could be provided to the source that would be generally useful.

  44. Vijay G says:

    Hi David,

    Really a great article, thanks a lot!!!

    I am facing one issue, how to force synchronous process to asynchronous? My use case is – I am dragging file to Outlook in that case VirtualFileDataObject doesn't call the GetAsyncMode() function, like it get called when dragging to explorer. I've tried all option to set IsAsynchronous to true, but not succeed. Waiting for your valuable response.

    Thank you.

  45. Delay says:

    Vijay G,

    I'm not sure that's possible. If the target application doesn't want to opt-into asynchronous mode, I think that's its choice and that Windows doesn't try to shim the operation. As a drag-drop source, you should try to support both modes, but be happy to provide data however the target asks for it. :)

  46. Vijay G says:

    Thank you David for your valuable comment.

  47. Joe says:

    Hi,

    I have a problem when trying to drag/drop large files using VirtualFileDataObject.

    What I do is to open a filestream to the source file, reading the bytes from the source filestream and writing those bytes to the Stream,

    which was delivered by the callback function of "fileDescriptor.StreamContents",

    using the "stream.Write" function. But this seems to write all bytes from the source file to memory first, before dumping it all to the harddisc.

    This sometimes leads to an OutOfMemoryException if the file is bigger than several hundered Megabyte.

    Is there any solution to this? I have to admit that I'm not that skilled in COM programming, so I do not know what to change in the VirtualFileDataObject…

  48. Delay says:

    Joe,

    It sounds like you're providing an implementation of FileDescriptor.StreamContents that reads from a file and writes it to a stream. When doing so, you want to read from the file in a series of small chunk and write each chunk to the stream before moving on to the next chunk. For example, create a 1000 byte buffer, read into it from the file, write that out to the stream, read the next 1000 bytes, etc.. In this manner, you'll only be using 1000 bytes of memory at a time no matter how large the file is.

    This is such a common pattern that recent versions of .NET offer the Stream.CopyTo method (msdn.microsoft.com/…/system.io.stream.copyto.aspx) which handles this for you!

    I hope this helps!

  49. Joe says:

    Yeah, that is exactly what I'm doing. In the FileDescriptor.StreamContents callback, I open a stream to the file on disk, read it in chunks of 4096kb, and write each chunk to the stream provided by the VirtualFileDataObject in the callback. I guess that's the common way of doing such things. But it seems the problem is the line  var iStream = NativeMethods.CreateStreamOnHGlobal(IntPtr.Zero, true); in the GetData function of the DataObject. This writes all chunks to the global memory, not to disc… so every time I write to the stream, I write to global memory and at some point this is full, which leads to a "Not enough storage" message. This even occurs, when I use the stream.copyTo function. I wish I could use SHCreateStreamOnFileEx instead of CreateStreamOnHGlobal, but this requires that I know the location of the file before dropping it to the explorer, but the file itself is created INSIDE the FileDescriptor.StreamContents callback.

    So what I want to do is:

    private void onStream(Stream stream)

           {

              // create the file after dropping some items of my application to the explorer to a temporary location

              // waiting until file creation has finished, after this I know where I find the file and how it is named.

             // Open a stream to the created file and "move" it to the location where it has been dropped. I do this by copying the bytes from the source stream to the target stream. But this leads to the problem I mentioned, because all is written to   global memory. SHCreateStreamOnFileEx seems to work if I knew the filename in advance, but this is created after the stream has been opened.

           }

  50. Joe says:

    Sorry, chunks of 4096 bytes, not kb :)

  51. Delay says:

    Joe,

    Got it, thanks for clarifying! Unfortunately, you're working right at the boundary of my memory/experience, so I don't know the right answer. :( That said, I think you're on the right track – what I'd suggest is to look at the SetData method that calls CreateStreamOnHGlobal and see if there are other options (ex: dwAspect or tymed) that would allow you to do what you want (i.e., provide something other than an in-memory stream). Also, I think there's a deferred-render mode for clipboard data that might be relevant/useful here because it waits to provide the data until it's needed and the file name should be available at that time.

    If you figure this out, please let me know what you ended up doing – I'd love to learn! :)

  52. Matthieu says:

    Hi David,

    Thank you for this article, I've finally found a good way to drag and drop stuff from my .net app. I have a question though.

    I'm using the .net 35 framework and in my c# app (complied with the "any cpu" option) I can't get the VirtualFileDataObject class to be initialized. I saw that errors occure when initializing the static properties like below:

    private static short FILECONTENTS = (short)(DataFormats.GetDataFormat(NativeMethods.CFSTR_FILECONTENTS).Id);

    With the debugger I saw that the value of Id returned is over the maximum value for a short. For instance, if I do:

    var d = System.Windows.DataFormats.GetDataFormat("FileGroupDescriptorW");

    d then equals 0xc11e thus casting it to a short raises an OverflowException.

    I'm using windows 7 x64. Is it a know issue? What can I do to fix that?

    Thank you in advance,

  53. Delay says:

    Matthieu,

    This is not a known issue and I'm not seeing the same behavior on my machine with the sample application above. I do see Id values greater than 0x8000 and even with that the cast shown above succeeds.

    My guess is that your project has the /checked compiler option enabled: msdn.microsoft.com/…/h25wtyxf.aspx

    If so, you should be able to run the relevant code in an unchecked { … } block or change the types from short to int to avoid the OverflowException.

  54. Matthieu says:

    Hi David,

    Thank you! I removed the "checked" flag and it works now like a charm :)

    However I was wondering whether we could show the windows copy file form while dropping a large file. I'm trying to extract a 650mb file and nothing appears during the transfert. As you did in your improved version I can handle this in my application but the user won't be able to see the remaining time, right?

  55. Matthieu says:

    For those that do not want the entire file to be cached in memory while being dropped, you have to wrap your stream into an IStream:

           /// <summary>

           /// Simple class that exposes a read-only Stream as a IStream.

           /// </summary>

           private class StreamWrapper : IStream

           {

               private Stream _stream;

               public StreamWrapper(Stream stream)

               {

                   _stream = stream;

               }

               public void Read(byte[] pv, int cb, System.IntPtr pcbRead)

               {

                   Marshal.WriteInt32(pcbRead, _stream.Read(pv, 0, cb));

               }

               public void Seek(long dlibMove, int dwOrigin, System.IntPtr plibNewPosition)

               {

                   Marshal.WriteInt32(plibNewPosition, (int)_stream.Seek(dlibMove, (SeekOrigin)dwOrigin));

               }

               public void Clone(out IStream ppstm)

               {

                   throw new NotImplementedException();

               }

               public void Commit(int grfCommitFlags)

               {

                   throw new NotImplementedException();

               }

               public void CopyTo(IStream pstm, long cb, IntPtr pcbRead, IntPtr pcbWritten)

               {

                   throw new NotImplementedException();

               }

               public void LockRegion(long libOffset, long cb, int dwLockType)

               {

                   throw new NotImplementedException();

               }

               public void Revert()

               {

                   throw new NotImplementedException();

               }

               public void SetSize(long libNewSize)

               {

                   throw new NotImplementedException();

               }

               public void Stat(out System.Runtime.InteropServices.ComTypes.STATSTG pstatstg, int grfStatFlag)

               {

                   throw new NotImplementedException();

               }

               public void UnlockRegion(long libOffset, long cb, int dwLockType)

               {

                   throw new NotImplementedException();

               }

               public void Write(byte[] pv, int cb, IntPtr pcbWritten)

               {

                   throw new NotImplementedException();

               }

           }

    Then, in the SetData method, change its signature:

    public void SetData(short dataFormat, int index, Stream stream)

    {

      …

      var iStream = new StreamWrapper(stream);

      …

      // Ensure the following line is commented out:

      //Marshal.ReleaseComObject(iStream);

      return new Tuple<IntPtr, int>(ptr, NativeMethods.S_OK);

     …

    }

    By doing so, while extracting/dropping large file, you don't have a new memory stream which contains the data!

    Hope it helps

  56. Delay says:

    Matthieu,

    Thanks a lot for the example above for in-memory caching!

    Unfortunately, I don't know of a way to reuse the Windows copy dialog in this scenario.

  57. Matthieu says:

    I've found it -> msdn.microsoft.com/…/ff362447.aspx

    var FILEDESCRIPTOR = new NativeMethods.FILEDESCRIPTOR

                   {

                       cFileName = fileDescriptor.Name,

                       dwFlags = NativeMethods.FD_ATTRIBUTES | NativeMethods.FD_SHOWPROGRESSUI,

                   };

    And now we have the windows copy dialog ;)

  58. Delay says:

    Matthieu,

    Awesome, thanks for sharing! :)

  59. RussellH says:

    David, great article!  

    One question, has anyone tried Matthieu's post on using a IStream?  I ran into some problems after changing the signature of the SetData Method.  Other methods that call it have issues then.

    Matthieu's post on the windows copy was cool!