Introducing 3D in Avalon and 2D/3D Integration

At the beginning of May, we gave a number of presentations on Avalon at WinHEC.  One thing that we also did, without making much fanfare about it, was make the first public showing of 3D in Avalon.  I'm really excited about 3D in Avalon because of some of the opportunities for innovation it provides (and because I was heavily involved in the design :-) ).  The way in which it's integrated with the rest of the system is what gives it its power, and I'll discuss briefly below.

First, some other pointers.  Daniel Lehenbauer is a member of the team doing 3D in Avalon, and discusses it in this post, and hopefully more in the future.  Joe Beda is also an Avalon-er, presented at WinHEC on 3D (amongst other things), and provided this post that includes a great looking screen shot of 3D in action.  There's an introductory document up on MSDN discussing the basics of the 3D support.  The rest of this post is intended to provide some grounding and motivation for what we're doing in 3D.

An area I'd like to emphasize which really isn't discussed in the documents yet is the integration between 2D and 3D in Avalon.  This is the key area of innovation here, and the one we think is going to drive very interesting applications that incorporate 3D as a enhancer of fundamentally 2D applications.  I'm going to talk about two examples here.

2D/3D Integration

The programming model of Avalon is fundamentally a 2D one.  Elements are rendered via painter's algorithm, back to front.  There's no notion of interpenetration of elements.  A challenge for 3D is figuring out how it fits into this 2D world.  The approach we arrived at is pretty straightforward - each 3D “world” or “scene” (called a Model3D) is an island onto itself.  It takes a Model3D to be paired up with a Camera before it can be incorporated into the 2D world.  The Model3D+Camera can be thought of as a 2D projection of the model from the viewpoint of the camera, and that projection is what gets incorporated into and composited with the 2D scene.  The 2D element that represents this combination is called a Viewport3D, and is just as full-fledged an element as our Image element, Text element, etc.

That's the integration point from 3D to 2D.  To be complete, we need the reverse as well -- the ability to take a 2D 'image' (I quote it because it's not necessarily a bitmap, just anything that can have a 2D realization in the system) and get it into a 3D scene - either as a flat 3D component with a particular z-value, or to be used as a texture map onto 3D content.  We have plans to pursue both of those, but they're not in the builds that are currently available.

This combination of integration points makes for a powerful platform for building a new style of application that integrates 3D in in novel ways.

For a specific example, Joe posted this screenshot earlier:

This particular application has a 2D background behind everything (turned on by the Background button in the screenshot), overlaid with projected 3D content, which in turn is overlaid by both the patient buttons and the buttom buttons -- demonstrating transparency onto the 3D projection.  Furthermore, if the Show Patient Details button is clicked, a details box (semitransparent again) comes up in front of the main graph, with some 2D information and, inside of it, another 3D graph rendering.  Thus the composition proceeds seamlessly between 2D and 3D.  (Dirty little secret -- the little cubes with the patients' pictures on them in the buttons on the left actually aren't live 3D rendered -- but they could be :-) )

Visual Styling with 3D

A key area for innovation in control development and usage in Avalon in general is the ability to “style” controls -- to provide a completely different look for the control while retaining the same object model presented to the consumer of that control.  This is a critical separation required for both the ability for designers to work independently from developers; and for subsequent update of the “look” of an application (and this is much richer than simple “skinning” that many people are familiar with).  For example, a developer can use an off-the-shelf ListBox, and give it an entirely different look (lay out its items in a circle, for instance) unbeknownst to the code that consumes that ListBox.

Now, 3D comes into the picture.  The “visual style” of the control can now contain 3D.  While the control itself will still always be a 2D entity, it may now be made up of 3D models that are projected into 2D via a camera.  Taking the ListBox example from above, this means that one can style the ListBox to present the individual items as, say, images textured onto a sheet that is slanted into the distance, or rotates in 3D when hovered over.  Again, the consumer sees this as the same old ListBox OM, but the opportunities for visual styling just became much richer.