Image codecs in XNA Game Studio 4.0


Quick history lesson:

Once upon a time, we were designing XNA 1.0, and decided that instead of just exposing managed wrappers around the old D3DX functions for loading textures and models, it would be better to create a new thing called the Content Pipeline.

But when we came to release our 1.0 Beta, the Content Pipeline was not quite finished. Hmm. Doesn’t seem like people are going to have much fun with a supposedly easy to use graphics API that has no built-in way to load textures! We needed a quick fix, so we slapped a hasty wrapper around the D3DX texture loader, and thus Texture2D.FromFile was born.

FromFile is only supported on Windows, because D3DX was not available to us on Xbox. We considered doing the extra work to port this to Xbox, but it seemed like a waste of effort to duplicate functionality that is also available through the Content Pipeline. We also considered removing FromFile from our final Windows version, but that seemed like an unnecessary breakage for our beta customers.

Game Studio 4.0 takes cross platform consistency more seriously, so this only-on-Windows situation was no longer acceptable. Plus, Windows Phone adds scenarios involving web services and picture library integration, where runtime image loading and saving are important. But the D3DX image code is large, full of obscure features, and would be expensive to port to all our target platforms.

We decided to replace our D3DX image loader with a simpler new API. The new methods are:

  • Supported consistently on Windows, Xbox 360, and Windows Phone
  • Support PNG, JPEG, and GIF formats
  • Provide options for scaling and cropping
  • Designed for dealing with pictures (screenshots, photo library, web services)
  • Not optimized for loading static game textures (you should use the Content Pipeline for that)
  • Less fully featured than the old D3DX loader (no mipmapping, DXT compression, or color key)

The new load method has two overloads:

    static Texture2D FromStream(GraphicsDevice GraphicsDevice, Stream stream);
static Texture2D FromStream(GraphicsDevice GraphicsDevice, Stream stream, int width, int height, bool crop);

Because this reads from a Stream rather than a filename, it can be used with many data sources: FileStream, MemoryStream, NetworkStream, TitleContainer.OpenStream, etc. To read from a file on disk, use it like so:

    using (Stream stream = File.OpenRead("cat.png"))
{
texture = Texture2D.FromStream(graphicsDevice, stream);
}

There are two save methods:

    void SaveAsPng(Stream stream, int width, int height);
void SaveAsJpeg(Stream stream, int width, int height);

Which are typically used like so:

    using (Stream stream = File.OpenWrite("cat.png"))
{
texture.SaveAsPng(stream, texture.Width, texture.Height);
}

The first FromStream overload does minimal scaling. As long as the source fits within the maximum GPU texture size (2048 for Reach, or 4096 for HiDef) it will be returned exactly as-is. For instance this photo of my dog Kess is loaded into a 192×288 SurfaceFormat.Color texture:

source

If the source is larger than the supported maximum, it will be scaled down as little as possible to make it fit, while preserving aspect ratio (so width and height are always shrunk the same amount).

The second FromStream overload offers more control over how the image is scaled, which is useful for photo viewer applications, thumbnail displays, etc. When crop is set to false, the image is scaled to fit entirely within the specified size, while preserving aspect ratio. For instance if I load the above photo with width=256, height=256, crop=false, I get:

256x256-False

The height has been scaled to 256, while the width has been scaled down to 170 (to preserve aspect). If my source image was wider than it was tall, the width would come back exactly 256, while the height would be smaller.

If I load the same photo with width=256, height=256, crop=true, I get:

256x256-True

The image has been scaled to entirely fill the specified size, and cropped as necessary to preserve aspect ratio, so the leaves from the top of the original photo are no longer visible.

Another way to think of this:

  • Crop=false means “scale the longer axis to exactly the specified size, leaving the shorter axis smaller than specified
  • Crop=true means “scale the shorter axis to exactly the specified size, cropping the longer axis to make it fit

Various places where we used to expose a Texture2D property now return a Stream instead. For instance, the Picture.GetTexture method (which returned a Texture2D) is now Picture.GetImage (which returns a Stream). Likewise for album art and gamer profile pictures. The returned stream contains a JPEG encoded image.

This change was neccessary to make these APIs compatible with Silverlight. By decoupling image sources from the XNA Texture2D class, it is now possible to use our media API both from an XNA game (which can use FromStream to convert the stream into an XNA Texture2D) and also from Silverlight (which can load that same stream into a Silverlight BitmapImage).


Comments (34)

  1. I like it alot. Looks very useful.

    On 360 saving pictures was very annoying and hard before, and I couldn’t figure out how to do jpeg compression.

  2. Man without a Name says:

    I’m wondering, after reading all these posts – are there any new features in 4.0? Something you can do that you couldn’t before. It seems like it is all API refactoring and minor tweaks.

    Anyways, my 2 biggest wishes for the content pipeline are:

    1. Debugging support.

    2. Deployment with the game for user-generated content.

    Don’t know if that will ever happen…

  3. David Black says:

    Does the content pipeline still use the D3DX image routines?

    Because for the content pipeline they are not ideal:

    * Initializing a new D3D device every time another file is processed is slow.

    *In addition there are problems with device creation to process an image during desktop switchs(eg requests for admin privlages).

    * Plus wierd problems running the content pipeline in a profiler (eg SlimTune, PIX, NProf etc). ie xna throws exceptions when creating a device to process a texture. [yes I reported this to connect… a few years ago]

  4. Liort says:

    Is this method also available on the Xbox ?

    Will i be able to host a web service on a server that will serve images and access it from the xbox ?

  5. > I’m wondering, after reading all these posts – are there any new features in 4.0? Something you can do that you couldn’t before. It seems like it is all API refactoring and minor tweaks.

    The new features are summarized here: http://blogs.msdn.com/shawnhar/archive/2010/03/09/in-which-hints-become-facts-xna-game-studio-4-0.aspx

    By far the biggest new thing is support for Windows Phone.

  6. > Is this method also available on the Xbox ?

    From the article above:

    – Supported consistently on Windows, Xbox 360, and Windows Phone

    > Will i be able to host a web service on a server that will serve images and access it from the xbox ?

    No. Xbox does not support any network access other than the Xbox LIVE NetworkSession API.

  7. DragonSix says:

    On a slightly different subject about textures: Is the content pipeline gonna support the ATI2N format?

    I’m making do with Swizzled-DXT5 for now, but it would be nicer to have access to the best normal map standard.

  8. > Is the content pipeline gonna support the ATI2N format?

    This is not just about the Content Pipeline: XNA does not support any such non-standard texture formats at all.

    In some distant future if/when we add a third profile level over the top of Reach and HiDef, that would most likely expose the new DX10 block compressed formats, but for now, no.

  9. liort says:

    > Will i be able to host a web service on a server that will serve images and access it from the xbox ?

    No. Xbox does not support any network access other than the Xbox LIVE NetworkSession API.

    How is it possible to use the new method along with a NetworkStream object?

    Also, is this method Synchroneous or Asynchroneous?

  10. > How is it possible to use the new method along with a NetworkStream object?

    On Xbox, it is not, since there is no NetworkStream class on Xbox.

    The point is that image codecs are decoupled from any particular I/O implementation, so you can use the same image load/save API with whatever stream implementations are available on whatever platform you are working on.

  11. wie says:

    Why the heck there is dog on a picture instead of cat…

    It’s not prerecorded for posting, so who are You, who impersonate as Shawn?!

  12. Curious, behind the scenes are you using WIC on windows?

    WIC is mind bogglingly fast.

  13. Fuzzbuster says:

    Simple question: Development must be performed on Windows Vista or Windows 7 (can’t even download the xna 4.0 on my xp system). Games generated in XNA 4.0, are they vista/Windows 7 only or can they also be executed on XP? I’d check myself, but I’m one of those people that didn’t upgrade to Vista and have been putting off Windows 7 for a while longer. Probably after XNA 4.0 is in full release I’ll make my move.

  14. Jonathan Clark says:

    I’m using XNA 3.1 for a 3D level editor. The lack of any reasonable ability to load textures from outside of the content pipeline is a major deal-breaker. I hope you reconsider this.

  15. Fduch says:

    >SaveAsPng, SaveAsJpeg

    I don’t like this anti-OOP API. What happened to good old System.Drawing syntax for image saving?

    What about BMP/TGA?

    This should be texture.Save(stream, ImageFormat.Png)

  16. > The lack of any reasonable ability to load textures from outside of the content pipeline is a major deal-breaker.

    You can load textures either using the APIs described in this post, or by calling directly into the Content Pipeline TextureImporter (and optionally TextureProcessor) as described in my previous post.

  17. > I don’t like this anti-OOP API. What happened to good old System.Drawing syntax for image saving?

    > This should be texture.Save(stream, ImageFormat.Png)

    Such an API is actually less easily able to be extended in a backward compatible way. The way we designed it, if we ever want to add new formats we can do this just by adding new methods (which is a fully backward compatible change), wheras with the format enum approach, we would have to add new values to that existing enum (which is technically a breaking API change).

  18. > Development must be performed on Windows Vista or Windows 7

    I don’t know for sure, but I believe the plan is to fully support XP.  The CTP release requires Vista+ though.

  19. KikiAlex says:

    Are you going to add support for mipmapping and DXT compression on the Texture2D.FromStream ?

    I am asking because without such support this function is almost unusable for almost any game scenario.

  20. Mantheren says:

    > Less fully featured than the old D3DX loader (no mipmapping, DXT compression, or color key)

    Does this mean I cannot load my dds textures with dxt compression and mipmaps without using the Content Pipeline anymore?

    Kind of annoying since the textures are loaded out of a MPQ file into a memorystream on demand, and passed to Texture2D.FromFile at the moment…

  21. > Does this mean I cannot load my dds textures with dxt compression and mipmaps without using the Content Pipeline anymore?

    The only built-in DDS loader we provide is via the Content Pipeline texture importer, so you would have to write your own loader if you want to read these files in some other way (DDS is an extremely simple format so this would not be hard to do).

  22. > Does this mean I cannot load my dds textures with dxt compression and mipmaps without using the Content Pipeline anymore?

    The only built-in DDS loader we provide is via the Content Pipeline texture importer, so you would have to write your own loader if you want to read these files in some other way (DDS is an extremely simple format so this would not be hard to do).

  23. Kainsin says:

    >> I don’t like this anti-OOP API. What happened to good old System.Drawing syntax for image saving?

    >> This should be texture.Save(stream, ImageFormat.Png)

    > Such an API is actually less easily able to be extended in a backward compatible way. The way we designed it, if we ever want to add new formats we can do this just by adding new methods (which is a fully backward compatible change), wheras with the format enum approach, we would have to add new values to that existing enum (which is technically a breaking API change).

    You could have had an ITextureFormatExporter interface and made some built-in classes that implemented it(PNGExporter, JPGExporter). Then make the method Texture2D.Save(Stream, ITextureFormatExporter). After that new additions would just be new classes.

  24. pnsm says:

    It seems the main theme for the 4.0 XNA release is to drop functionality on Windows to increase compatibility with the weaker platforms.

    Disappointed.

  25. Karl says:

    Ugh I’m frustrated with many of these new changes. If DDS is such a simple format to write ourselves then why wasn’t it included in the new API? You say its too large and expensive to support but then say its easy to do?

    I don’t enjoy hearing that the functionality for MY preferred platform (windows) will continue to drop because a gimp phone or mp3 player can’t do it.

  26. Does this meen that tga image format is not supported any more

  27. Jamie says:

    “Sometimes taking a step forward requires a huge step back.”

    stepping back sometimes means losing features in favor of preparing the product to move forward in new directions and to be honest I like the large picture view of "compatible with Silverlight", "DirectX 10+", more platforms vs the few features that where dropped.

  28. David Black says:

    I agree with Jamie, most of the lost functionality can be worked around one way or another without too much trouble.

    (although that doesnt stop me wanting to not have to work around missing features or bugs:-)

  29. This is off-topic, but I doubt you still read comments from the post it refers to: what does "Separate alpha blend" mean in the reach vs. hi-def profiles post (no for reach, yes for hi-def)? Sorry for not asking sooner.

  30. amartinez1660 says:

    (Seconding Jamie and David Black)

    I, too, like a lot the new "breaking changes". The whole scope looks great, and very scalable for the future.

    The more platforms the better! Even if it means sometimes stripping some of the functionality to open way for the others (and as stated before, most of them have workarounds). XNA might become your one-for-all headquarters for game development.

    > "Compatible with Silverlight"

    1. The Silverlight part is only for the windows phone or does it mean XNA on a web browser also? (!!!).

    2. Or I got everything wrong, Silverlight and XNA are still two incompatible different things, except they happen to share some common APIs.

    > "Separate alpha blend"

    I am a little confused by this too. Lets hope Shawn answer this off-topic bit.

    My confusion is: there is no "separate alpha blend" anymore. You always set the blend operations for the color and for the alpha, if they happen to differ, then it’s like doing separate alpha blending.

    So it seems logical to draw this conclusion: GPUs were always doing separate alpha blending from the beginning, so Reach and HiDef will always do separate alpha blending (although it might seem more complex hardwired units/transistors for treating them separately, so older cards may not have that ability).

  31. Content pipeline error messages says:

    This post brought up a point about content pipeline error messages: http://www.kwxport.org/…/142

    When getting an error like "an item with that name has already been added," actually including the name in question would help a lot with debugging.

    As you've been tweeting about improving error messages lately, is the Content Pipeline getting the same love as the DX layer?

  32. Steven says:

    Thank good Xna added "texture.SaveAsPng(stream, texture.Width, texture.Height);", this makes it easier storing the image as "MemorySteam ms()" and maybe in some way posting it to a picture server like Facebook or common services :), thanks Xna 🙂

  33. LeeC says:

    "DDS is an extremely simple format so this would not be hard to do"

    So you create a framework that is designed to ease in non-programmers (and that's what XNA was originally marketed as), and then immediately tell them that they need to start writing image format conversion routines?!?

    The irony of that is almost blinding.

    Things like this just make me so angry!! If it's *easy* (for you) and it will help people, then do it… simple! Or are we dealing with the "I know how, but I'm not going to share" mentality? Really??

    To think people have been through education, where teachers (who found things easy) actually passed on that knowledge to you, without question, it's like you learned nothing from all that. Game development isn't an every-man-for-themselves type lifestyle… or maybe I was just lucky and bypassed the selfish ones.

    I learn, I share, why is that such a hard philosophy to follow for some people? *shakes head in disbelief*

  34. scoy says:

    @LeeC Complaining about an API that was cancelled a couple of years ago by continuing a thread that died over 4 years ago just so you can dis the team that made the API?  *shakes head in disbelief*

    Seriously though, the XNA API is one of the most well thought out APIs I've used.  Part of that is because of what has been left out.  Reading and writing DDS files is not something that most people need to do.  As pointed out in the messages above the only people that seem to want it are building their or game engine.  Even using XNA that's far from what "non-programmers" are doing.  Despite being "easy" that still doesn't make it the right thing to do for the API.  Everything has a cost even if it just means that something else doesn't get done.  Assuming that it wasn't down because the XNA team are somehow selfish is absurd.

    If you _really_ care about DDS support maybe you should step up, implement it, and donate your work to the MonoGame project so everyone can benefit.