Combining XAML and DirectX


Since early on in the development of Windows 8, we’ve been thinking about ways to combine UI, such as the rich set of controls and interactivity provided by XAML, and graphics, like the low-level, high-performance rendering of DirectX.

The feedback you gave for the Developer Preview through the Dev Center forums and other channels helped us focus on the set of related scenarios that developers really wanted to build. We saw some common themes: many of you wanted to either add DirectX graphics to what could otherwise be called a XAML app, or easily add Metro style UI to a DirectX app.

Good news! With the release of the Consumer Preview, you no longer have to draw a hard distinction between a XAML app and a DirectX app. You can now tap into the strengths of both XAML as a rich UI platform and DirectX as a high-performance rendering solution within the same app, using a familiar XAML model.

Now that you can combine these, you really get the best of both worlds. This opens up a wide range of possibilities – some of the key scenarios you told us about included things like:

  • Image processing, creativity, and design apps that blend UI and graphics
  • Large-scale map and document viewers, that mix huge DirectX graphics and some UI
  • Games and simulations with full-screen, high-performance DirectX graphics and minimal overlaid UI

where the combination of XAML and DirectX can make developers more productive and apps richer and faster.

Metro style UI in Windows 8

XAML – in parallel with HTML/JavaScript – is a UI toolkit for all types of apps in Windows 8, providing constructs for interactivity, Metro style controls and animations, and rich functionality like accessibility support, databinding, media, and HTML hosting. You also benefit from design-time support in Visual Studio and Expression Blend, all using your choice of programming languages.

One area of continual investment for the engineering team that spans across all these areas in both XAML and HTML/JavaScript is performance: we’ve put a lot of effort into this in Windows 8, making it easier than ever for you to build fast and fluid Metro style apps. In the graphics area in particular, we’ve continued to improve rendering and composition performance, worked to use GPU hardware wherever possible, and even made it so that common types of animations run independently off the UI thread to ensure they remain smooth regardless of any work your app is doing on the CPU. We’ve worked toward ensuring that everything the UI frameworks do, they do well — fast, fluid, and with lower power requirements. When using a UI framework, you’re already using the power of DirectX behind the scenes; in many cases this provides everything you need to achieve great graphics performance.

DirectX in Windows 8

Still, there are definitely times when you need the raw immediate-mode rendering performance and direct access to graphics devices that DirectX provides. Starting with Windows 8, DirectX SDK components are now included as part of the primary Windows SDK, and DirectX contains a host of new features such as a unified 3D API, improved 2D graphics and text performance with Direct2D, a rich image processing pipeline, and improved printing support.

Everything in Windows 8 is optimized for and built around DirectX, from the developer platforms to the OS to hardware design. This way, you can use DirectX to achieve the highest performance rendering in Windows 8. The major tradeoff is that pure DirectX solutions can be quite complex to create and maintain, and (unless you’re using a third-party wrapper) you must implement them in C++.

This is where DirectX and XAML interop comes into play: you can now build your XAML UI using C#, Visual Basic or C++ and include C++ DirectX components. You may also be wondering about HTML/JavaScript and DirectX – as mentioned earlier we’ve done a lot to bake the power of DirectX into the UI platforms, but for combining UI and native DirectX graphics we’ve focused on XAML for Windows 8.

Design goals

With the above in mind, we developed a few principal goals we wanted to meet when enabling the combination of DirectX and XAML:

1. Performance

  • Support low-latency input and interactivity
  • Allow incremental redrawing of both XAML and DirectX content
  • Minimize overhead from using XAML

Chiefly, we wanted to avoid imposing a “XAML tax”: if your app is primarily DirectX, using XAML should be an easy choice if you need Metro style UI. We know this isn’t important just to you: we wanted to help ensure your customers see great performance and long battery life from your app.

2. Flexibility

  • Enable all functionality of both XAML and DirectX 11

The result is that you can design and develop a XAML interface in Blend and Visual Studio, and combine it with any DirectX content. We wanted to make sure we weren’t limiting the power of either XAML or DirectX — just letting you combine them.

3. Integration

  • Ensure smooth integration of XAML and DirectX

You can combine DirectX graphics and XAML UI in the same app using familiar XAML concepts.

After breaking down the set of scenarios listed earlier, we found they fell into three broad categories:

  1. Combining small islands of DirectX into a primarily XAML UI
  2. Combining large scalable DirectX surfaces with XAML UI
  3. Overlaying Metro style XAML UI onto a primarily DirectX app

In the end, we found that a one-size-fits-all approach didn’t work well across all these categories. So, we added three XAML types that each targets one of these key types of scenario. We believe this provides a high degree of flexibility for you to choose the option(s) best suited to the requirements of your app. These options aren’t exclusive — you could even use all three approaches for different components in the same app.

Scenario #1 – SurfaceImageSource: islands of DirectX in a XAML UI

As the first option for combining DirectX and XAML, you can use the new SurfaceImageSource XAML type to add areas of DirectX content to a Metro style XAML app. An instance of this type provides you with a shared DirectX surface you can draw to using all the functionality of DirectX, then composes the surface into the rest of your XAML app’s interface.

This option is particularly useful when you want to use DirectX to draw specific components that are part of a larger XAML UI. For example, you might want to use SurfaceImageSource when applying complex image effects to a photo, or when creating high-performance 3D data visualizations.

If you’ve ever used a XAML ImageSource, the usage will be quite familiar: you can set a SurfaceImageSource as the source of an ImageBrush and then use it to paint nearly any XAML element (or elements — brushes are reusable!). For example, you could use a SurfaceImageSource to fill a XAML Rectangle:

SurfaceImageSource^ surfaceImageSource =  
ref new SurfaceImageSource(rectangle1->Width, rectangle1->Height, true);
ImageBrush^ brush = ref new ImageBrush();
brush->ImageSource = surfaceImageSource;
rectangle1->Fill = brush;

You can then use DirectX to draw the content of the Rectangle, and update it as often as you need to. The best part is that this surface is seamlessly integrated with the XAML composition model: properties applied to that Rectangle also affect the content you draw with DirectX. You can easily overlap elements, and apply render transforms, projections, z-index, or opacity, all in XAML, without any of the airspace issues often experienced when mixing rendering technologies:

A diagram showing a XAML app that mixes a DirectX surface and XAML content. The DirectX surface is rotated and composed with other XAML content that is overlaid on top of the surface.

Rectangular DirectX surface composed with XAML content

If you’re thinking this seems conceptually similar to the D3DImage type in WPF, then you’re right! However, we listened to your feedback on D3DImage and worked to make SurfaceImageSource both more powerful and easier to use. In particular, SurfaceImageSource: supports DirectX 11, including Direct2D; removes many of the limitations that D3DImage had (for instance, SurfaceImageSource works over a Remote Desktop connection); improves performance; and manages the surfaces for you — no more locking and unlocking, or manually creating and managing buffers! A SurfaceImageSource can even support surfaces larger than your Direct3D device’s maximum texture size.

The first thing you might notice when creating a SurfaceImageSource is that at a glance it doesn’t seem to expose any new methods. The implementation is in fact exposed via a backing native interface; this is because it needs to directly reference DirectX APIs, which are available only in C++. This means you must use C++ when calling DirectX APIs, either directly in a C++ XAML app or in a separate C++ WinRT component that you can then include in any C# or Visual Basic app. Once you include the required header file:

#include "windows.ui.xaml.media.dxinterop.h"

Then you can query for the ISurfaceImageSourceNative interface, which gives access to the three methods you need to get started:

interface ISurfaceImageSourceNative: IUnknown
{
SetDevice(IDXGIDevice *pDevice);
BeginDraw(
RECT updateRect,
IDXGISurface** pSurface,
POINT* offset
);

EndDraw();
};

BeginDraw() takes an updateRect parameter that enables you to constrain the area you want to update; this allows you to improve performance by making only smaller incremental updates to the surface. The method also returns a surface and a point specifying an offset where you should begin drawing: even if you’re updating the whole surface, you should still make sure to check this offset, because your surface may be part of a larger one the framework maintains internally as an optimization. From here you can create and set a DXGI Device, and then start drawing with DirectX! The DirectX and XAML interop Dev Center topic provides a good overview of the details.

Scenario #2 – VirtualSurfaceImageSource: large-scale DirectX content + XAML UI

VirtualSurfaceImageSource extends SurfaceImageSource to better support huge surfaces. Its foundation lies in the concept of virtualization — that is, drawing areas of a surface only as they become visible on the screen.

We designed this type to be used when displaying complex content that is larger than the screen, especially when it can be panned or zoomed. Scenarios where this might apply include anything from a document viewer to a medical imaging app — apps with content that could potentially be many times larger than the maximum resolution of an average monitor. A map control is another good example: using a VirtualSurfaceImageSource, you could implement a large vector-based map with DirectX and let the system do the work of tracking the area that’s on-screen at any given time. You can also overlay on the map XAML elements such as pushpins, and easily keep their positions in sync with the DirectX content.

VirtualSurfaceImageSource implements this virtualization using tiling and a callback model. After you create your full-size VirtualSurfaceImageSource, your DirectX content gets split up into a series of rectangular areas behind the scenes; the framework automatically keeps track of the visible rects at any given point (plus potentially some extra just out of view, to keep panning smooth), and invokes a callback that your app implements to provide the content for newly visible regions when needed. This works especially well when you place your VirtualSurfaceImageSource content in a ScrollViewer to enable smooth panning with touch or a mouse.

A large DirectX surface with area that is currently visible framed by a screen. Blank regions outside the screen represent parts of the surface that have not been drawn yet because they aren’t yet visible on the screen.

Large surface split into rectangular tiles

This approach isn’t just easier to implement than tracking coordinates yourself: it can provide a large boost to your app’s performance. Because the system caches your content internally, after you draw the content of a region it’s kept in memory and will automatically be reused by the GPU even if it’s panned out of view and then back again — no redrawing required! If you want to update any region you’ve already drawn, you can simply call BeginDraw() as you would with a SurfaceImageSource. As a further performance optimization, the system automatically recycles memory used for old regions that are out of view after a certain point, helping to bound memory usage and ensure your app will perform well across a range of form factors and hardware profiles even with very large surfaces. This performance benefit becomes especially apparent on portable devices such as tablets.

Similar to using SurfaceImageSource, you can query for the backing native interface, in this case IVirtualSurfaceImageSourceNative, to access the required methods. You can check out the DirectX and XAML interop Dev Center article or the Direct2D magazine app sample for details.

Scenario #3 – SwapChainBackgroundPanel: DirectX with a XAML overlay

The third and final option for combining XAML and DirectX content, SwapChainBackgroundPanel is a XAML element for developing full-screen, primarily DirectX-focused apps. Unlike SurfaceImageSource and VirtualSurfaceImageSource, SwapChainBackgroundPanel is a XAML element rather than an ImageSource; it inherits from the XAML Grid layout panel and enables apps to create and fully control a full-screen DirectX swap chain, on top of which all other XAML content is overlaid:

Illustration of composition model of SwapChainBackgroundPanel, where DirectX content always sits behind a XAML overlay

A SwapChainBackgroundPanel sits behind overlaid XAML content

We think this is definitely the best option for games, and for other apps where most of your graphics are DirectX; you get great performance and an update model close to a pure DirectX app, and you can use XAML at the same time to easily create a heads-up display interface and databind it to your game model, add interactive controls, and include Metro style elements such as AppBars.

To achieve this, we impose a couple restrictions on the SwapChainBackgroundPanel: it must be the root XAML element of your app, and a few inherited XAML properties, such as Opacity and RenderTransform, won’t have any effect on it (though you can of course apply the equivalent DirectX operations to the swap chain content to achieve these effects). This allows you to make a tradeoff between the close integration with the XAML framework’s composition model that you get with SurfaceImageSource, and the lowest-latency interaction and extra control over presentation timing possible when you directly manage the swap chain.

The backing native interface in this case has only a single method:

interface ISwapChainBackgroundPanelNative: IUnknown
{
SetSwapChain(IDXGISwapChain *pSwapChain);
};

You can start presenting your content as soon as you call SetSwapChain() to associate your swap chain with the panel. At that point SwapChainBackgroundPanel also becomes hit-testable and you handle input using normal XAML input events. Any child XAML elements you add to the panel are displayed on top of your swap chain.

The interop Dev Center topic has more info on SwapChainBackgroundPanel usage, and you can take a look at the source for a sample DirectX shooting game built using SwapChainBackgroundPanel that shows the end-to-end implementation of a Direct3D game with a XAML interface:

A screenshot of a DirectX and XAML first-person shooter game. The DirectX surface contains walls and targets overlaid with a XAML UI containing a menu and an AppBar control.

A game built with DirectX and XAML

In closing

We’re excited about the new possibilities that open up when you combine the rich UI support and design-time productivity of XAML with the high-performance graphics power of DirectX. We hope you find these three new options for building Metro style apps useful when you create compelling apps for your customers.

— Jesse Bishop

    Program Manager, Windows

Comments (35)

  1. Dee says:

    Any plans on porting XNA to Windows 8?

  2. small_mountain says:

    This is a very cool feature, one that I've wanted for desktop applications for as long as I've been using WPF.  What are the chances that the ability to use XAML with native code and DirectX will show up on the desktop application development side of the house?  Or is the desktop application development platform just not going to see significant enhancements going forward?

    Thanks,

    Eric

  3. Morten says:

    I've been trying to create a C++/DirectX custom control, that I can consume from a C#/XAML app, but so far with no luck. Could you please post a sample that does this? You can keep it as simple as a possible in the DirectX as you want (since this sample is not about doing advanced rendering but how the stuff interplays), and just focus on having a C++ control render something simple to a SurfaceImageSource, and then the C# app consumes this control in a XAML page – perhaps with a simple binding flowing back to the C++ layer.

    Another thing I'd like to see covered is doing this so it works on x86, x64 and ARM. The dev story for this as far as I can tell is slightly confusing for the average .NET dev, since they usually use AnyCPU.

  4. Rick Barraza says:

    OMG. Insanely awesome. You've made a small but crucial core of XD professionals ridiculously happy.

  5. XNA_FTW says:

    DID YOU NOT GET THE **********RESOUNDING****** FEEDBACK THAT WE NEED XNA FOR METRO?

  6. RisingStartDevsNEEDxna says:

    I learnt XNA (and am pretty clueless about graphics engines otherwise) and own a very popular windows phone game that is ranked in the top 20 apps. I don't have the budget/time resources to now go learn directX. There are 100s of developers like me. Dont you want their games on Windows 8? Have you seen how many games ipad has?

    How about at least telling us one way or another if there will be future support for XNA+Metro. Just say yes or no. So other people can about making plans around yours.

    Thank you.

  7. Jose says:

    I've been hacking at getting the SwapChainBackgroundPanel working with SharpDx and used in a XAML/C# app … i'm very close to getting it working BUT have had a brilliant experience so far.. I love the 3 approachs you guys have taken and look forward to seeing how this pushes the state of LOB and Metro-XAML apps  :) :)

  8. Nico Vermeir says:

    I agree with RisingStartDevsNEEDxna, I've spent a lot of time learning XNA and I'm having a blast building games in it.

    please port this over to Win8 or at least give us something similar. I really don't want to go and learn C++

  9. RichardC says:

    5 years ago: "yeah, XNA is great, learn it, you'll use for create great games for Windows AND for Windows Phone. It's a GREAT opportunity for you"

    Septembre 2011: XNA is dead (RIP)

    Now : "C++ is great for Metro, learn it. It's a GREAT opportunity for you"

    Future : ???

    So please: give us back XNA.

  10. Grog says:

    I developed few XNA games for WP7, and I was hoping that with Consumer Preview we will get XNA framework for Windows 8, but since nothing happened and Microsoft is constantly silent about it, last week I bought Mac, iPad and MonoTouch licence, and started to migrate my games to iOS.

    Sadly, I cannot believe that your product managers did not see huge potential of XNA indie developers and you didn't implement XNA for Metro. When Windows Store opens this year there will be probably lot of Metro apps on it, but relatively small number of games because of such decision. I invested lot of energy and time in XNA and my custom game fremework, and I do not want to switch back to C++ or HTML5/JS. Simply I do not have time for that.

    I know that in some moment community will probably release some kind of XNA wrapper for WinRT, but frankly Microsoft doesn't deserve it.

  11. IndieDev says:

    Just give us XNA back. Thats the feedback you should be listening too.

  12. Yes please! XNA 5 for windows 8 with directx 11 and Metro support.

  13. Absolutely Fabulous news!!! Very, Very happy and excited to see this has been done. This is a huge step in opening the road to top level 3D-graphics and 3D-audio, for even "small" devices.

    Thank you so very much for listening!

  14. small_mountain says:

    Unfortunately, Microsoft seems to have decided that they are somehow better off not being up-front about the future of the various development platforms.  "Look, here is this shiny new development framework, don't you love it!?!?!"  Meanwhile. we've been building our businesses around technologies that were the shiny new framework 6 years ago, and we can't seem to get a peep out of Microsoft about what the future of those technologies is.  We need clarity, and Microsoft is determined to provide anything but.

    Eric

  15. John says:

    Please, please support XNA with C# on Windows 8 Metro.

    I do not want to go back to C++ and DirectX.

    You support XNA with C# on Windows Phone and XBox, please do so for Windows 8 Metro.

    I hope Windows 8 Metro succeeds, and I want to go along for the ride.

    Please give us developers the tools to do this.

  16. Jules says:

    With no clear direction on XNA for WinRT  we are now looking MonoTouch Game development for Android and iOS.

    After shafting the XNA community like this,  Microsoft does not deserve the attention of Indie Game Developers  to fill it Marketplace.  

  17. Machaira says:

    I will not be taking a step backwards to use C++ for game development. I've stated repeatedly over the years that the only way I'd use C++ again is if someone like Bioware or Blizzard hired me. There's just no reason for an indie to use C++ for game development when an awesome tool like XNA exists. I'll just be making desktop games for Win 8 non-ARM devices I guess. That's fine with me.

  18. Jonas says:

    Folks requesting XNA should check out the sharpdx project. It will allow you to write Metro DirectX apps with C#. Not XNA obviously, but you should at least be able to reuse a lot of code.  

  19. aL says:

    All this xaml and DX stuff is AWSOME but please PLEEEAASE enable it for desktop apps as well, not just Metro apps! we could really really use this for some apps but they also need the multitasking, multi window support of desktop apps!

    As for XNA, relax guys, once we have these native interfaces a  managed layer can be built intop of that. SlimDx already allows you to get an IntPtr to the swapchain so you could pass that in to these elements and do all your development in c# 😀

  20. IO says:

    This is very sad indeed. I don't want to give up all of the fantastic tools and the excellent content pipeline in XNA to go back to C++, write everything your self and good luck managing memory. Going to a new platform with new features is one thing; going backwards like this is just ludicrous.

    Seriously, who's driving this boat?

  21. Jeff says:

    My vote for XNA with its ease (C#) and platform embracing strategy.

    Please drop a single line on that subject / status. Even if not commenting on XNA itself, I would really appreciate to know more about the "why not commenting?".

  22. Jeff says:

    One last note in addition to last comment: With C# you've done such a great job developing a language so powerful, clean and fluently to use like no other language. Congratulations for that! In my opinion you could even knock out Google and Apple by buying and smoothly integrating Mono / Xamarin. Immediate win on the coolness factor – long term win on the dev community. I mean how cool is that: Embracing Windows, Xbox, WinPhone, iOS, Android, Linux, everything with one language / runtime. Just give it some thought – maybe not winning each single space like phones, but winning on the overall tools and the devs.

  23. Dan says:

    I agree with small_mountain_0705 above.  We need this functionality in desktop apps – either in WPF or allow WinRT/XAML-based desktop apps.

  24. jps says:

    And another vote for small_mountain_0705 — this should be in WPF. I realize you've been spending time tooling up for Metro. Please spend some time allowing those of us who are entrenched in WPF to HAVE a successful future as well.

  25. Add my vote for adding this to WPF.  We didn't even get the airspace fix for WPF that we've been begging for for ages–it was removed between the DP and CP!

  26. Sam says:

    I am also semi-depressed about the lack of XNA. I have put a lot of effort into some top notch games for WP7 and it seems like we are getting screwed here. We all need to make some big decisions about whether we are going to stay with the windows platform or not unless they decide to support XNA.

  27. Samyll says:

    +1 for official GPU-programming support directly from C#, please.

    The lack of this makes development for Metro feel like a huge leap backwards for several developers.

    The Playstation Suite SDK by SONY can live off supporting only C#, one would think MICROSOFT at least adds the same level of support for C# as they do for C++.

  28. Samyll says:

    XNA has one fundamental problem however.

    It is kind of hard-coded for DirectX 9.

    Make something for Metro that is capable of keeping up with constant evolution of GPUs, but is similar to XNA.

    Heaven forbid, take C# into consideration while designing the future version of DirectX…

  29. Stilgar says:

    Comment attempt 4

    There is a need for managed game development framework. It may not be backward compatible with XNA but I will be damned before I learn C++ to try out game ideas.

    BTW is there some centralized place where the "BRING BACK XNA!!!1!" rage is concentrated so I can join?

  30. Phenom says:

    I second that a managed 3D development framework is a must.  I own a product (3D terrain viewer), where I need 3D, but I do not need to squeeze out every fps, nor all the latest shaders bells and whistles.

    I did that piece in C# with the already obsolete Managed DX, and I was successful to pull this out in a few months.  Should have I done it solely in C++, which I have only modest knowledge of, that would take me much more time.

    Microsoft, you really need to give us something similar to XNA.  There is no need for it to be fully compatible, but should be close enough to let developers migrate easily.  And be quick about it.

  31. Siavash says:

    I can see everybody here, like me, is complaining about not having XNA on W8! :(

  32. Personally I'm delighted to find that native C++ is once again a first-class citizen in the Windows development ecosystem. SwapChainBackgroundPanel is exactly what I need to bring my skillset directly to Metro and the app store. Bravo.

    My question is, how will this work on Windows 8 tablet/phone?

  33. Simon says:

    For anybody else who wants to vote on getting SwapChainBackgroundPanel et al into WPF:

    visualstudio.uservoice.com/…/2644759-provide-better-directx-support-in-wpf

  34. Nikolai says:

    I'm using SharpDX in my Win8 metro project.  100% C#, full hardware acceleration, pixel shader support, it's great.  It's not XNA:  it's basically using the DirectX api directly but in C#.  There's an alpha-stage project called "ANX" that wraps SharpDX to create a "source compatible" framework.  Quote:  "Source compatibility means, that you can "translate" a XNA game project to be a ANX game project by simply replacing all namespaces from Microsoft.XNA.Framework to ANX.Framework.  I can't speak to the accuracy of that statement, but I can speak to the utter awesomeness of SharpDX.