At first glance the XNA Content Pipeline may seem to cover similar ground to the old D3DX utility library. For instance both provide a simple way to access graphics from .X files:
Mesh ungulate = Mesh.FromFile("dromedary.x", 0, device);
Model ungulate = loader.Load<Model>("dromedary");
But in fact the Content Pipeline architecture is very different to D3DX. For one thing we include a Visual Studio user interface and incremental build capabilities, where D3DX just provided helper functions and left it up to you to figure out where and how to call them. There are also some fundamental philosophical differences between the two technologies.
First up: extensibility.
D3DX is not extensible. It provides a good starting point for a lot of people, but often isn't suitable for more advanced programmers and engine developers. Games using D3DX run into a brick wall if they ever need to do something that D3DX does not support. Maybe they've written their own level editor, so they want to load from a file format other than .X. Or perhaps their engine wants to do something clever at runtime, beyond what the basic D3DX Mesh class provides.
When I've written my own game engines in the past, I often wanted to use bits of functionality from D3DX. Because I had my own level editor, file format, and mesh class, I had to write lots of laborious code to convert my data to and from .X files that D3DX could process. Not fun!
With the XNA Content Pipeline we have tried to make the easy things easy, while still leaving room for advanced users to customize and extend how everything works. For instance:
We provide importers for common file formats (.X, .FBX, .BMP, .TGA, .DDS, .FX, etc), but it is also easy to add new ones of your own. An XNA Content Importer is a simple .NET object that reads an external file into our standard object model.
We provide a standardized design time object model for data such as meshes, textures, effects, and animations, but this can easily be extended to contain your own custom types. The .NET type system and serialization infrastructure make this sort of extensible object model much nicer to work with than it would have been in C++, and because these types are only intended to be used during the build process, they can take full advantage of .NET goodness like enumerators and dictionaries that would be too inefficient to use in your runtime graphics structures.
We provide implementations of common tasks (generating mipmaps, compressing textures into DXT format, and optimizing mesh data for the GPU vertex cache), but it is also easy to add new ones. An XNA Content Processor is a simple .NET object that manipulates data using our object model. Because we have a standard representation for the most common types of graphics data, processors from many different sources will be able to work together, and writing new processors is often as simple as just stringing together functionality by calling some combination of the existing ones.
We provide a few runtime output types to get you started (Texture, Effect, Model), but it is easy to add more. If you are writing a new engine, you should be able to add new runtime output types without having to reimplement the existing importer and processor functionality. Or if you are using an existing engine, you should be able to support new file formats by writing a new importer, without having to reimplement the existing processing and conversion code. Use the bits you want, replace the bits you don't. It's up to you.
There are a couple of areas that are less extensible. For instance our incremental build system "just works" as long as you use our importer and processor object model. This seemed like something we could get right for a lot of people without requiring anyone to extend it, so we just went ahead and took care of it for you.