Win2D 0.0.15 - cleanups and breaking changes


Win2D version 0.0.15 is now available on NuGet and Github.

New this sprint:

We also made a significant breaking change to the effects API:

Breaking changes are going to be a theme over the next few weeks as we approach the Build conference.  We plan to release a somewhat stable version of Win2D, including Win10 UAP support, in time for Build.  We’re not promising zero breaking changes after that point (this will not be our final 1.0 release) but we’re aiming for a greatly reduced rate of churn compared to what you’ve seen so far.  Which, of course, means the next few weeks are our last chance to tear things up!  So we’ll be taking a short break from adding new features in order to revisit some designs that we’re not entirely happy with:

  • DrawImage overloads are inconsistent and frustrating
  • Should we have more nesting underneath the Microsoft.Graphics.Canvas namespace, to remove clutter?
  • Reconcile the various text direction / flow options between XAML and DWrite
  • Weird interaction between CanvasTextFormat.LineSpacing and LineSpacingMethod
  • Is the CanvasSwapChain.ResizeBuffers DPI parameter in the right order?  What about other methods that take a DPI?

Please bear with us and pardon the mess while we fix these things up.  Taking the time to get them right will make the API better in the long run.  Also, please let us know if there are other things you find wrong or confusing about our current API designs.  Not missing things (we can add new APIs easily at any time) but any designs you think are less awesome than they could/should be.

Thanks!


Comments (29)

  1. Andrew says:

    Guys, please make CanvasRenderTarget.GetPixelBytes() accepting a buffer for storing pixels. I understand that in this case the client should know the size of the buffer to allocate, but its essential to have such an overload to achieve good performance in some cases, when you don't want to allocate new big chunk of memory every time. In case of a .Net client, such allocations can fragment large object heap significantly.

    Ideally, it would be nice if there also was method like .WritePixelBytesAsync(...), accepting a stream. In my case, I render map tiles, and I have to provide a stream with the tile bytes. Since InMemoryRandomAccessStream doesn't allow to preallocate memory, I would use .Net MemoryStream, converted to a WinRT stream. And here I'd be happy to write the bytes from the render target directly to the memory stream which I will return to the TileDataSource, avoiding redundant memory copying.

  2. Interesting, thanks for the feedback Andrew!   Would an overload of these methods that accepted a WinRT IBuffer interface be sufficient to meet your goals?

  3. Andrew says:

    Shawn,

    As long as I can convert an array to an IBuffer, this approach will be nice. But its still strange from the point of consistency that currently you return an array, not a buffer; as I understand, array allocation is actually done automatically by the COM/WinRT interop, and the method accepts a memory pointer.

    But this is only half of the story. If .WritePixelBytesAsync(...) implementation will able to write all bytes to a InMemoryRandomAccessStream at once (meaning memory reallocation happens only once), this would be perfect (performance) solution for my concrete case. However, having ability to get pixel bytes to an IBuffer, I of course can use a MemoryStream converted to IBuffer for getting pixels (1st step), and can use that MemoryStream  converted to a IRandomAccessStream for returning to the TileDataSource (2nd step). But this looks somewhat tricky, and possibly slower.

  4. Could you share more info about what other API you are passing this data onto?

    Reading back the contents of GPU rendered bitmaps into a CPU array or stream is quite a slow operation for graphics hardware, so I'm surprised that you're finding the CPU side overhead of allocating these output arrays to be a significant expense  (I'd have expected that cost to be dwarfed by the initial GPU -> CPU readback).

    So it would be good to understand more about where the data is going, to see if there might not be some other more efficient way to do this entirely.

  5. Andrew says:

    Shawn,

    I don't actually find it slow, but I suspect there can be some performance issues (but more likely you are right). For example, in some cases there may be additional copying when using those WinRT-.NET adapters, or in some cases MemoryStream, created implicitly, doesn't allow accessing its bytes directly, requiring reading it in chunks using an intermediate buffer. Also, on a mobile device, every extra code execution consumes battery, and while copying all bytes from GPU memory to CPU memory at once can be relatively efficient (especially on SoC systems, or/and when CPU doesn't participate in this process at all), having many interop calls may add to power consumption significantly. Again, these are primarily only my fears, I just mean its always better if you have the shortest path for the data.

  6. Andrew says:

    Regarding my code case: first of all, look at the sample in "Overlaying custom tiles" section at msdn.microsoft.com/.../dn632728.aspx.

    In particular, they suggest this code for converting image bytes to a stream refrence:

    // Create RandomAccessStream from byte array

               InMemoryRandomAccessStream randomAccessStream =

                   new InMemoryRandomAccessStream();

               IOutputStream outputStream = randomAccessStream.GetOutputStreamAt(0);

               DataWriter writer = new DataWriter(outputStream);

               writer.WriteBytes(bytes);

               await writer.StoreAsync();

               await writer.FlushAsync();

               return RandomAccessStreamReference.CreateFromStream(randomAccessStream);

    Here, in addition to already allocated bytes, another block of data is allocated for InMemoryRandomAccessStream, and a copy is done.

    My code is somewhat shorter, but involves a WinRT-.Net adapter:

    var randomAccessStream = new InMemoryRandomAccessStream();

               await randomAccessStream.WriteAsync(bytes.AsBuffer());

               return RandomAccessStreamReference.CreateFromStream(randomAccessStream);

    But again, it allocates additional memory of the same size and does copying.

    You could suggest the following approach:

    var memoryStream = new MemoryStream(bytes);

               var randomAccessStream = memoryStream.AsRandomAccessStream();

               return RandomAccessStreamReference.CreateFromStream(randomAccessStream);

    But it turns out that this will crash with an exception, because this random access stream adapter doesn't support stream cloning, an MapControl API needs it for some reason:

    "This IRandomAccessStream does not support the CloneStream method because it requires cloning and this stream does not support cloning."

    > System.Runtime.WindowsRuntime.NI.DLL!System.IO.NetFxToWinRtStreamAdapter.ThrowCloningNotSuported(string methodName) Unknown

    System.Runtime.WindowsRuntime.NI.DLL!System.IO.NetFxToWinRtStreamAdapter.CloneStream() Unknown

    Remembering this, I've just realized, that the approach I described in step 2) in the earlier  post won't work for for the same reason. So we return to the need of having a method which could write pixel bytes directly to a WinRT stream, namely, to InMemoryRandomAccessStream. But of course, a method accepting IBuffer is a must have too.

  7. Is this app far enough along that you could take some profile measurements of this code in action, or possibly share a version that I could do some profiling on?

    My instincts are that compared to all the other expensive operations you have going on here, the details how the pixel data gets marshalled is going to be insignificant noise, therefore not worth focusing optimization effort on.

    But this is just an instinct - I'd love to get some actual data one way or the other!

  8. Andrew says:

    I can't provide you the exact code I work with, but I've created a sample project which resembles the actions I do in my code (but the drawing is fake), take it here: http://1drv.ms/1C7m9fd . At this address you will also find

    two screenshots I made from profiler session (on a real device) showing how much time was spent on copying in Win2D (6.76%) and how much in DataWriter (5.74%) while the CPU was loaded at more than 80% (I did intensive map scrolling). You can also pay attention to percentage spent in different interop routines (btw, I included .diagsession and .vspx too).

    But as I said, my main concern is not about interop, but about double memory allocations (remember, GC LOH fragmentation, tough memory conditions on a mobile) and copying. I think, wasting 5-6% of CPU time is not good, especially on a mobile device. While on my Lumia 920 there was no visible performance hit, it can be visible on a low end device. But what is more important, again, is wasting battery power.

  9. Andrew says:

    And finally: we are talking about my specific case. But somebody may have need to render and copy much bigger images, much more intensively, and there will be no MapControl which consumes most resources, so relative CPU consumption may be much higher comparing to other code.

  10. Thank you for the test app, that will be very helpful.

  11. Lukas says:

    I think you should support both writing to an IBuffer and writing to a byte[]. The IBuffer is the native WinRT way to handle and move data. For native development, this will be the best choice. For .NET developers, the byte[] will be much better, since they can access it directly.

    The problems with getting byte[] into a RandomAccessStream is something that should be solved on WinRT platform level. I do not understand at all why we cannot create a InMemoryRandomAccessStream (or an equivalent) from a pre-allocated byte[], or with a fixed size, without unneccessary allocations and copying. This is really a big problem for .NET applications which interface with databases or other libs. But this is a WinRT platform issue and does not have much to do with Win2D. Perhaps you can convince your colleagues to be a bit more convenient here? A lot of people struggle with this - not only with Win2d but also with any kind of data access (databases and other libs which work with byte[]).

  12. Andrew says:

    Lukas, I mostly agree with you, but regarding passing a .Net array as a buffer - it looks like its a tricky thing. Here is what Jeffrey Richter writes regarding arrays marshaling to WinRT:

    "WinRT can marshal an

    array’s elements in or out of a method; never in and out. Because of this, you cannot pass an

    array into a WinRT API, have the API modify the array’s elements and then access the modifid

    elements after the API returns.4 I have just described the contract that should be adhered to.

    However, this contract is not actively enforced, so it is possible that some projections might

    marshal array contents both in and out. This usually happens naturally due to improving performance. For example, if the array contains structures, the CLR will simply pin the array, pass

    it to the WinRT API, and then unpin it upon return. In effect, the array’s contents are passed

    in, the WinRT API can modify the contents and, in effect, the modifid contents are returned.

    However, in this example, the WinRT API is violating the contract and this behavior is not guaranteed to work. And, in fact, it will not work if the API is invoked on a WinRT component that

    is running out-of-process." (C)

  13. I've added the request to support IBuffer to the Win2D feature backlog, and will do some digging to find out who owns the byte[] -> RandomAccessStream area.

    Thank you so much for the input and detailed discussion about this!  I find it extremely helpful, and really appreciate you both taking the time to provide such thoughtful feedback.

  14. Andrew says:

    Shawn, thank you for listening us 🙂

  15. Moondevil says:

    Any hope to get any of this work in standard Windows 10 APIs?

    Just went through the available documentation and it seems it still going to be C++ route for DirectX with no C# bindings available. 🙁

    This is really sad as other mobile platforms do expose the graphics APIs to all the official languages.

    Anyway thanks for Win2D.

  16. Apologies Moondevil, but I don't understand your question.  Are you asking about C++ access to Win2D  (which is absolutely possible) or C# access to other APIs apart from Win2D?

  17. juanpablogc says:

    First of all absolutely great project it's like a windows.forms render and xna. I have compiled the example and great. Then I created a Windows 10 UAP project and I get Win2D from nuget and it says "Could not find SDK "Microsoft.VCLibs, Version=12.0". TestingW2D" I have added the 14.0 version that I understand is made for UAP, but does not work. I do not know if it is possible to download a compatible dll from somewhere.

    I have dig into the gallery and is there an option about making XOR between text and a rectangle? for instance, to make TextClipping basically. Because I think there is no TextGeometry at the moment doesn't it?

    Thanks anyway, in my W8 project I tested some blurs and displacementEffect and it's awesome

  18. Juan,

    Win2D does not yet include Windows 10 UAP support.  We are working on that at the moment.

    You can clip text (or any other drawing) to a rectangle using the CreateLayer method:  microsoft.github.io/.../M_Microsoft_Graphics_Canvas_CanvasDrawingSession_CreateLayer_8.htm

  19. juanpablogc says:

    Thank you for the answer, I will start making some controls in a W8 app an then I will copy them to the W10 when it's released. Yes I explain a bit bad myself (that's so awesome that I think to many things at the same time) I mean create an image with a text hole.

  20. juanpablogc says:

    I have made a inherited Grid and I inserted a CanvasControl with blur effect, absolutely great. The matter is that when I translate the Grid and I update the grid position to the canvascontrol per two updates there is a flickering about the real position that it should be, and I do not see what could be wrong, because I have tried to move the Grid really slow and happens the same, might be the loop I am doing wrong or the invalidate call now should be in another way. I posted it here expediteapps.com/.../blurgrid-using-win2d.

    I know it's my problem but an idea would be great I have made several changes with the same luck.

  21. juanpablogc says:

    My fault I initialized the events twice, now I have removed the call to Initialize and it's absolutely marvelous

  22. Glad to hear you got this working!  I like how you're using that blur control, looks neat...

  23. DamyanPepper says:

    @Juan: it's exciting to see new and interesting ways of using Win2D!

    As an aside, it might be worth checking out blogs.msdn.com/.../async-resource-loading-in-win2d.aspx - this describes a pattern for loading bitmaps from CreateResources that handles device lost scenarios.

  24. juanpablogc says:

    I am preparing some TextControls with shadow, hollow (I will have to dig to make it), outline like

    msdn.microsoft.com/.../ms745816(v=vs.110).aspx (Incredible says that WPF can convert text into geometry).

    But before continue I understand there are no plans to add effects for controls like in WPF for Windows Apps (W10 apps), or are there?

  25. juanpablogc says:

    Hello, I have tested on all the ways and effects that I could, but I cannot make a outlined text (paint just the contour). I think MorphologyEffect is absolutely great

    var effect = new MorphologyEffect

               {

                   Source = renderTarget,

                   Mode = MorphologyEffectMode.Dilate,

                   Height = 5,

                   Width = 5

               };

    If the effect could have another MorphologyEffectMode.Outline, just to have the outline of the text will be really great when having big outlined texts and images.

  26. DamyanPepper says:

    This item on the backlog (github.com/.../Backlog) "All DWrite functionality is available in WinRT" will expose the functionality required to generate geometry from text (specifically, exposing the functionality provided by IDWriteFontFace::GetGlyphRunOutline I believe is what you need).

    Feel free to open feature requests at github.com/.../new.  This is way they can be tracked and commented on by other people.

    Have you tried playing with ConvolveMatrixEffect?  This can be configured to do an edge detect (see the Effect example in ExampleGallery), which may be a way to get towards what you want using effects.  You could feed the results of this into an opacity mask.

  27. juanpablogc says:

    Thank you a lot for your patience and consideration, I have posted my solution here expediteapps.com/.../textblock-with-win2d-effects.

  28. Very cool, it's great to see you got this working!

    Another option that could give more control for thicker borders would be to use MorphologyEffect in Dilate mode to create the border, then subtract the original image from that (using ArithmeticCombineEffect) to cut out the inside and leave only the border.  Or to make a border that extends both outside and slightly inside the original, subtract the result of an Erode MorphologyEffect from the Dilate MorphologyEffect.

  29. juanpablogc says:

    I will try that, thank you. My next steps is test what you suggest. Then create the transparent text on a image effect. Just that name ArithmeticCombienEffect  is really interesting I do not know why I did not try it before.

    And to rest a fireworks animation to improve an old one I did in XAML for a W8 game I released

Skip to main content