Effect compilation and Content Pipeline automation in XNA Game Studio 4.0


Previous XNA versions included an Effect.CompileEffectFromSource method. We do not support runtime shader compilation on Xbox, so this was only available in the Windows framework, which could compile shaders targeting either Windows or Xbox. This factoring was problematic because:

  • The API looked like part of the runtime framework, but was not available on all platforms.

  • Our Windows redistributable had to include the Xbox shader compiler, which was wasted space for people shipping Windows games.

  • As we add more platforms, we increasingly find ourselves having to ship pieces of our product out-of-band for specific platforms. Not such a great versioning story if changing the shader compiler for unrelated platforms requires us to update the Windows framework DLL!

Game Studio 4.0 moves the shader compiler from the Windows framework to the Content Pipeline. Effect.CompileEffectFromSource is replaced by EffectProcessor.

There are several ways you can invoke the Content Pipeline to compile an effect:

  • Add .fx file to the Content folder in Visual Studio, then hit F5. Easy, but not especially flexible.

  • Add .fx file to an MSBuild project (either using Visual Studio, or hand-editing XML), then build this from the command line using msbuild.exe. This approach is powerful and IMHO unfairly neglected.

  • Use the MSBuild API to programmatically create and build a project in memory. Great for big projects like level editors, but a hassle to set up.

  • New in Game Studio 4.0: call directly into an importer or processor from your C# code, bypassing MSBuild entirely. The rest of this article describes this new approach.

To call into the Content Pipeline, you must add a reference to Microsoft.Xna.Framework.Content.Pipeline.dll. But when you open the Add Reference dialog, this will most likely not be listed. What gives?

.NET Framework 4 introduces the concept of a client profile, which is a subset of the full .NET Framework optimized for client applications (as opposed to server or developer tools), and which therefore has a smaller download size. The XNA Framework runtime works with the client profile, and our Windows Game project templates target it by default, but the Content Pipeline requires the full .NET framework. To reference the Content Pipeline assembly:

  • Open your Visual Studio project properties
  • Focus the Application tab
  • Change ‘Target framework’ from ‘.NET Framework 4 Client Profile’ to ‘.NET Framework 4′
  • Choose Add Reference
  • Focus the .NET tab
  • Add Microsoft.Xna.Framework.Content.Pipeline, plus whatever importers you want to use (let’s pick Microsoft.Xna.Framework.Content.Pipeline.TextureImporter as well)

We’re going to import a texture. First some using statements:

    using Microsoft.Xna.Framework.Content.Pipeline;
    using Microsoft.Xna.Framework.Content.Pipeline.Graphics;
    using Microsoft.Xna.Framework.Content.Pipeline.Processors;

Now we make a custom logger class. This version just throws away any warnings or messages, but a real app might want to display these somewhere:

    class MyLogger : ContentBuildLogger
    {
        public override void LogMessage(string message, params object[] messageArgs) { }
        public override void LogImportantMessage(string message, params object[] messageArgs) { }
        public override void LogWarning(string helpLink, ContentIdentity contentIdentity, string message, params object[] messageArgs) { }
    }

Finally we create a custom importer context, which allows the importer to communicate with whoever is hosting it:

    class MyImporterContext : ContentImporterContext
    {
        public override string IntermediateDirectory { get { return string.Empty; } }
        public override string OutputDirectory { get { return string.Empty; } }

        public override ContentBuildLogger Logger { get { return logger; } }
        ContentBuildLogger logger = new MyLogger();

        public override void AddDependency(string filename) { }
    }

Armed with these helper classes, it is easy to call into the TextureImporter:

    TextureImporter importer = new TextureImporter();

    TextureContent texture = importer.Import("cat.tga", new MyImporterContext());

Calling a processor is similar, except we need a custom processor context, which is more complex than the importer context we used before:

    class MyProcessorContext : ContentProcessorContext
    {
        public override TargetPlatform TargetPlatform { get { return TargetPlatform.Windows; } }
        public override GraphicsProfile TargetProfile { get { return GraphicsProfile.Reach; } }
        public override string BuildConfiguration { get { return string.Empty; } }
        public override string IntermediateDirectory { get { return string.Empty; } }
        public override string OutputDirectory { get { return string.Empty; } }
        public override string OutputFilename { get { return string.Empty; } }
        
        public override OpaqueDataDictionary Parameters { get { return parameters; } }
        OpaqueDataDictionary parameters = new OpaqueDataDictionary();
        
        public override ContentBuildLogger Logger { get { return logger; } }
        ContentBuildLogger logger = new MyLogger();

        public override void AddDependency(string filename) { }
        public override void AddOutputFile(string filename) { }

        public override TOutput Convert<TInput, TOutput>(TInput input, string processorName, OpaqueDataDictionary processorParameters) { throw new NotImplementedException(); }
        public override TOutput BuildAndLoadAsset<TInput, TOutput>(ExternalReference<TInput> sourceAsset, string processorName, OpaqueDataDictionary processorParameters, string importerName) { throw new NotImplementedException(); }
        public override ExternalReference<TOutput> BuildAsset<TInput, TOutput>(ExternalReference<TInput> sourceAsset, string processorName, OpaqueDataDictionary processorParameters, string importerName, string assetName) { throw new NotImplementedException(); }
    }

To build for a different platform, or for HiDef as opposed to Reach, change the TargetPlatform and TargetProfile properties.

You can use the AddDependency method to track additional files that are read by the importer or processor. For instance this is called any time the effect compiler encounters a #include statement in the effect source code. This information is useful if you want to implement incremental rebuild, so you can handle things like recompiling effects because their #include files have changed, even when the main .fx has not.

This simple example does not bother to implement the Convert, BuildAndLoadAsset, and BuildAsset methods. These are not neccessary when running simple standalone processors, but you will need to hook them up if you want to support more complex things like ModelProcessor, which uses them to call into the MaterialProcessor, EffectProcessor, and TextureProcessor.

Armed with our custom processor context, we can create an EffectContent object, set its EffectCode property to a string containing our HLSL source, then compile it with EffectProcessor:

    EffectContent effectSource = new EffectContent
    {
        Identity = new ContentIdentity { SourceFilename = "myshader.fx" },

        EffectCode =
        @"
            float4 MakeItPink() : COLOR0
            {
                return float4(1, 0, 1, 1);
            }

            technique Technique1
            {
                pass Pass1
                {
                    PixelShader = compile ps_2_0 MakeItPink();
                }
            }
        ",
    };

    EffectProcessor processor = new EffectProcessor();

    CompiledEffectContent compiledEffect = processor.Process(effectSource, new MyProcessorContext());

We can use processor parameters to adjust the compilation behavior:

    EffectProcessor processor = new EffectProcessor();

    processor.Defines = "ENABLE_FOG;NUM_LIGHTS=3";
    processor.DebugMode = EffectProcessorDebugMode.Optimize;

    CompiledEffectContent compiledEffect = processor.Process(effectSource, new MyProcessorContext());

Note that the Content Pipeline is not part of the XNA Framework redistributable, so this code will only work on computers that have a full Game Studio install.


Comments (28)

  1. xzodia says:

    This is looks great awesome, except for the full game studio install requirement…

    I wonder, would it be possible for there to be an intermediate install, which doesn’t require the whole of visual studio, but supports building content?

    It would be good for non-programmers.

  2. ShadowChaser says:

    I’m not sure how I feel about this – personally, I find the content processor fairly unhelpful and only suited to extremely small games without user created content.

    I was really hoping for the *opposite* in XNA 4 – the ability to use all types without a dependency on the content pipeline. Previously about half of the content types (fonts, audio, etc) could only be constructed via the content system. Now there are more – incredibly frustrating!

    I really wish XNA was treated more like a true API with the content processor/pipeline as an isolated, optional component. Of *all* the things in a game, generally content has the most specific requirements in terms of storage, layout, etc.

  3. David Black says:

    Whats the advantage of doing things this way, rather than using the simpler(in most cases) ContentProcessorContext.Convert<>() ?

    If it is performance, then why doesnt XNA do that anyway?

  4. David Black says:

    Never mind… I think I see way, to allow people who dont genuainly want to use the content pipeline to hack around the fact that the shader functions are missing(perhaps rightly missing).

    Still wouldnt a helper function in the content pipeline assembly have been better, rather than embedding the compilation into the processor?

    (or even an independant assembly distributed with the content pipeline)

  5. clayman_joe@yahoo.com.cn says:

    Finally we can build content without MSBuild,  great improve!

  6. ShawnHargreaves says:

    > Whats the advantage of doing things this way, rather than using the simpler(in most cases) ContentProcessorContext.Convert<>() ?

    The simple answer is that to call Convert, you need a ContentProcessorContext instance, which is provided to you if you are running inside the Content Pipeline / MSBuild environment, but if you are running somewhere else, you need to create your own context object, in which case you are responsible for implementing Convert yourself.

    To implement the Convert method, you typically end up writing code that looks very much like the above: find the requested processor type, instantiate it, then call into it.

    The reason these methods all go through a context object, and why you can’t just use our build-in context if you are running in some different environment to the standard MSBuild setup, is that these methods require a lot of policy to work correctly:

    – They need to know how to locate importer and processor types, which depends on how these assemblies are loaded (for instance our MSBuild implementation loads everything into a worker app domain, marshalling build operations back and forth, in order to be able to selectively reload modified processor assemblies between one build and the next).

    – They need to understand how to track dependencies, which has different requirements depending on if/how you are implementing incremental rebuild.

    – The BuildAsset method needs to be able to kick off other nested build requests, generate ExternalReferences pointing to the final build output locations, etc.

    All these things require intimate knowledge of the environment in which the build is running. We use MSBuild to implement our standard Content Pipeline build environment, and provide a standard context object which implements these things by hooking up to MSBuild.

    If you run somewhere else where there is no MSBuild, you have to make your own choices about how to locate and load pipeline assemblies, how to track dependencies, etc, so you must provide your own context object, and therefore can’t just rely on our built-in implementation of the Convert<> helper.

  7. Hmm.. Sounds like a Problem for me. In my Game, Postprocessing Shaders are generated dynamicly, so that i only need One extra Rendertarget for this. But with this changes I can Not longer compile the shader on the client maschiene…

    Am I missing something, or is there a workaround?

  8. I’m currently relying quite heavily on Effect.Disassemble(), I’m guessing this has gone too?

    I expect I’ll end up calling into fxc directly.

  9. I also have always compiled some of my shaders on the client machine. Is there a workaround as having visual studio installed on all the machines isn’t really an option.

    This seems like the first actual loss in 4.0 however, everything else seems to be an improvement.

  10. Nice ,, this what is was looking for for my editor, so i can make the physics work from inside my editor,, insted of recompile the physics all the time

    i have uploaded a nice physics wideo with the new stuff ,,,

    Please shawn , write more about this and put up a code sample at creators club "windform sample 3"

    best regards

    Michael

    link to physics content pipeline new stuff direct from shawn cool code..

    http://www.youtube.com/watch?v=ffLwJyV1HJU

    i now work in my editor , i am happy

  11. Zeljko says:

    This seems like a change that is going in the wrong direction. I am working on a game that generates shaders dynamically at run-time and this is not good for me.

  12. David Black says:

    The XNA policy of always pre-compiling shaders does seem a bit odd.

    If you look at many native engines the approach seems to often be a mix of pre-compiling a shader cache and dynamically compiling shaders if there is a cache miss(eg out of date cache).

    Why not allow this approach with XNA?

  13. Roonda says:

    I agree that this change is backwards and frustrating… that is until we finally get a content pipeline redist!!. When can we finally write tools that utilise the power of the XNA FX but do not require a full XNA GS install?

  14. AR says:

    In my last game, I had some code in an #if DEBUG #if WINDOWS block that gave my game a hotkey that I could use to recompile shaders within the running game, during development. (Very, very useful!)

    I’d like to do the same thing in XNA 4.0. But I cannot figure out how to reference the content pipeline assembly for a debug build, but not a release build. Thoughts?

  15. Rim says:

    I wonder, is there any policy/EULA similar to the DirectX redist that prevents the content pipeline DLLs from being included with the binaries? If so, could this perhaps be waived for internal/development use so we can do this by the book?

    I personally was planning on sneakily doing this anyway for some internal content tools, to make the deployment easier on the artists. These guys are hard enough to find and I don’t want to frustrate them with cumbersome installation processes. I’ll already frustrate them enough with my art requirements 🙂

  16. ShawnHargreaves says:

    > I wonder, would it be possible for there to be an intermediate install, which doesn’t require the whole of visual studio, but supports building content?

    I would personally love to provide a Content Pipeline redist, but that didn’t make it far enough up the priority list to fit into the schedule for GS4.

    Please vote on this on Connect if it’s important to you, so the relevant folks can prioritize it when planning future versions.

  17. ShawnHargreaves says:

    > If you look at many native engines the approach seems to often be a mix of pre-compiling a shader cache and dynamically compiling shaders if there is a cache miss(eg out of date cache).

    That’s a great approach for use during development time, and something you can totally do using XNA in a level editor, or in debug builds of your Windows version (by calling into the EffectProcessor as described in this article).

    It won’t work in a released Windows game, though (because the Content Pipeline is not available there), and won’t work at all on Xbox (where there is no runtime shader compiler).

    Native engines that target consoles as well as PC have the exact same challenge as XNA developers in this regard, and generally deal with it by using this kind of dynamic caching mechanism in PC dev builds, and then precompiling all the shaders they are going to need while building the final console images.

  18. ShawnHargreaves says:

    > In my last game, I had some code in an #if DEBUG #if WINDOWS block that gave my game a hotkey that I could use to recompile shaders within the running game, during development. (Very, very useful!)

    > I’d like to do the same thing in XNA 4.0. But I cannot figure out how to reference the content pipeline assembly for a debug build, but not a release build. Thoughts?

    This is a great example of where it would be useful to learn MSBuild.

    The Visual Studio UI does not support conditional references, but this is trivial to do using an MSBuild conditional if you hand-edit the .csproj XML.

  19. Brian Lawson says:

    Great post.  Now that I'm working with XNA again, this post helped me quickly get run-time shader compilation up and running again.  However, it's not clear to me how #include is supposed to be handled?  Is the EffectProcessor supposed to load those files (or request a load via callback) or is on us to pre-parse shaders and do the #include substitution?

    I also asked on the forums: forums.create.msdn.com/…/71970.aspx

  20. xcorporation says:

    > In my last game, I had some code in an #if DEBUG #if WINDOWS block that gave my game a hotkey that I could use to recompile shaders within the running game, during development. (Very, very useful!)

    > I'd like to do the same thing in XNA 4.0. But I cannot figure out how to reference the content pipeline assembly for a debug build, but not a release build. Thoughts?

    This is a great example of where it would be useful to learn MSBuild.

    The Visual Studio UI does not support conditional references, but this is trivial to do using an MSBuild conditional if you hand-edit the .csproj XML.

    Pre-Build events are useful for detecting output folder .Contains("Debug" / "Release" )

    From there you'll be able to set the flag on program load.

    Global::Debugmode = true etc.

  21. Pre-compiling an effect... says:

    Shawn, your blog has been my one frail tether to sanity throughout my DirectX to XNA transition.  But I've spent the last 5 hours trying to figure out something simple, and am feeling equal parts stupid and enraged – I desperately could use your help.  I want to precompile a simple .fx effect.  Per your suggestion in this article, I tried the custom, hand-written MSBuild project approach, and was pleasantly surprised – I quickly learned a lot about, as you said, this powerful but neglected side of building.  I have my multiple effects cleanly building from the command line with a simple call to MSBuild.  However, I still use Visual Studio for editing my .fx files, and am routinely forgetting to switch back to the command line to compile them – I find myself wasting time trying to figure out why my recent changes didn't seem to do anything (- because I forgot to build them again, of course!).  I'd like for them to be automatically compiled when my solution is built.  I thought this would be easy – I have a completely working .contentproj file, that requires no special arguments to MSBuild, everything works well from the command line; the project is already part of my solution; I just can't figure out how to get it to build with everything else.  Visual Studio (2010) won't let me click the "build" check box in the Solution Configuration Manager; and I've pulled out what little hair I have left jumping through hoops trying to get it to automatically build some other way.  I've read every shred of documentation on MSBuild I can find, but cannot get this working – please, have mercy on me and point me in the right direction ….

  22. ShawnHargreaves says:

    You don't need to do anything so complex to build a content project.  Just create a default XNA Game Studio game, add your shaders to its content project, F5 to build the thing and off you go, this is all hooked up automatically by default…

  23. Pre-compiling an effect... says:

    I'm grateful for your response, and should have been more specific – I'm using my effects in a WinForms application – I'm working on medical imaging software, I have no use or desire for the Game class, library or anything else – I just want to build my xnb's at compile time, and load them at run time, in a simple windows form (which, unless I misunderstood, you blogged about here blogs.msdn.com/…/using-xna-with-winforms.aspx).  I've got my solution, and under it a winforms project, and an XNA content project that won't build with the solution.  If I add an XNA game to my solution, yes I can get my effects to build with it, but then I'm building a game I don't want or need; if I add an XNA game library to my solution, I can add a content reference to my effects project to the game library, and my effects get built along with the library – but then I'm building a library I don't want or need.  If I choose not to build the game/library, my effects don't get built either; if I delete the source files from the game/library, I don't build the vestigial exe/dll, but my effects don't get built again; if I try to directly edit the game/library/content project file XML, VS2010 throws up all sorts of warnings about "invalid child elements" wherever it sees anything XNA related, and something inevitably breaks, and I'm back to reading incomplete MSBuild documentation and pulling out hair again.  I have to be missing something obvious, please bear with me – I really appreciate your help.

  24. ShawnHargreaves says:

    Visual Studio content projects are built indirectly by XNA Game or XNA Game Library projects.  If you want to build a content project inside VS, it needs to be a child of a Game or Game Library.

  25. ... says:

    That's… disheartening.  So… I've got a project that MSBuild will build and Visual Studio won't, all this project file XML that's supposed to make the build process configurable but can't bring it to the level of a basic Makefile, and 2012's XNA 4.0 can't give me the functionality I used to have in 2000's DirectX 8.  There are 3D applications beyond gaming, managed development is affordable by non-profits, and surgeons aren't happy when they're visualizing a patient's heart and there are XBox references in the software; making the managed DirectX replacement inseperable from (in fact, synonymous with) multi-platform gaming is …  Really appreciate your help and responses, really disappointed again in Microsoft…

  26. ShawnHargreaves says:

    Visual Studio can build shader content just fine, as long as you use an XNA project type to do this.  You can also build these XNA projects from the commandline using MSBuild.  You're complaining because you can't build XNA content from some other, non-XNA project type.  Well, duh!  If you want to use the fancy UI that XNA adds to Visual Studio to integrate building this type of content into the GUI, then you need to use one of the XNA project flavors to do that.  You're the one who chose not to use an XNA project (typically you would want a library project for this situation).

    I don't think it makes sense to choose not to use something, and then complain that the features of this thing no longer work 🙂

    > 2012's XNA 4.0 can't give me the functionality I used to have in 2000's DirectX 8

    Of course it can.  You're complaining about the lack of shader compilation integrated into the IDE, which

    a) Is supported by XNA, as long as you use an XNA project type to do it

    b) Was not a feature of DirectX 8

  27. Alex says:

    Hi Shawn,

    I have converted mu Application from XNA3.1 to XNA4.0 but encountered a problem.

    When running the Application, an InvalidOperationException error message pops up.

    "The current vertex declaration does not include all the elements required by the current vertex shader. Color0 is missing.

    I am lost in this situation. Please help.