Working with computer graphics in Windows 7 and Silverlight 3
Introduction
Did you WOW when you saw Aero theme for the first time? Have you ever planed your new home using CAD software? Do you like to share photos with your friends? After you turned on your PC or your phone, you will find graphics everywhere. We can live without graphics, but how boring will the world become? So what if we can create our own software to make the world even better? That’s wonderful!
In the past few months, our All-in-One Code Framework team has released several computer graphic related samples to demonstrate the latest features in Windows 7 and Silverlight 3. They will help you to light your applications both on PC and on the web.
All the samples are well documented. Please refer to the ReadMe.mht/txt for the detailed document. This blog post will give you an overview of the samples.
Direct2D, a new graphic API in Windows 7
Cpp/CS/VBWin7Direct2D (contributed by Yi-lun)
This is a sample demonstrating the new vector graphic API of Windows 7: Direct2D. In the application, all elements are vector graphics rather than bitmaps. When you click the planet, it will move around the star.
You may wonder why we decided to create the Direct2D sample. After all, we already have GDI! To answer this question, let’s look back into the history.
No doubt today most Windows applications use GDI as its rendering engine. The 30 years old API (born in ice age of computer era) has been served us well, and it continues to play an important part in a lot of scenarios. However, the end users are demanding better user experience, yet GDI is just ignoring our dedicated graphic card that caused us several hundred dollars, and ordering CPU to do all the work, even though CPU is already complaining he’s busy calculating the complex business report.
To the other extreme, we have Direct3D. Without it, you will not be able to enjoy your favorite games. Direct3D is an outstanding commander who distributes the work between CPU and GPU, make each of them happily doing work that they’re good at. However, most developers are trained to work with a programming language that is understood by CPU, such as C++/C#/VB, while the standard way to write Direct3D (at least starting from D3D 10) is to do as much work as possible in HLSL, which is only understood by GPU. As more and more shaders are introduced with each release of Direct3D, many developers find themselves lost in the ocean of HLSL.
Windows 7 solves this problem by introducing a new graphic API. It allows you to reuse your existing knowledge of GDI (the programming model is similar), yet you can enjoy hardware accelerating. What’s more, it can interoperate with both GDI and Direct3D, allows you to keep your existing investments, and at the same time, you can explore the new world created by HLSL.
The new graphic API contains 3 components:
Direct2D: A vector graphic API. Its programming model is similar to GDI, but it can exploit the latest GPU features.
DirectWrite: A text engine that is independent from the rendering engine. It supports complex text layout.
WIC (already exists in Windows Vista): A bitmap graphic API that supports extended encoders and decoders.
For more information about Direct2D, please refer to the Cpp/CS/VBWin7Direct2D sample. DirectWrite and WIC samples will be provided later.
Silverlight, the best graphic engine on the web
Html is good, but Silverlight is wonderful!
Silverlight supports vector graphics from the very beginning. You can create vector graphics by writing XAML markup manually, by writing C#/VB code, or by exporting XAML markup using a graphic tool such as Expression Design. We will demonstrate vector graphics in Silverlight in the future.
CSSL3WriteableBitmap (contributed by Yi-lun)
Silverlight also has extensive support for bitmap images, and Silverlight 3 adds support for WriteableBitmap, which allows you to edit individual pixels of the image.
Please refer to our CS/VBSL3WriteableBitmap sample for more information. It is a sample demonstrating how to work with WriteableBitmap in Silverlight 3. Try to select 2 bitmap images, and shoot on the top image. This sample can be ported to WPF, but additional code modification is required. See the readme file for more information.
CS/VBSL3PixelShader (contributed by Allen)
WriteableBitmap allows you to leverage CPU to edit individual pixels of bitmap images, and pixel shader allows you o leverage GPU to create effects for any element, including bitmap images, button, textboxes, videos, and so on. You need to learn a new language HLSL in order to use pixel shaders, but it’s definitely worth learning! Do not fear, HLSL 2.0 is very easy to learn, since its feature set is quite limited.
Please refer to our CS/VBSL3PixelShader sample for more information. It isa sample demonstrating how to work with pixel shader in Silverlight 3. With pixel shader, you can write code that is understood by GPU rather than CPU. When you click the image, you will see a wave effect. This sample can be ported to WPF with minimum modification.
CS/VBSL3DeepZoom (contributed by Yi-lun)
Silverlight also allows you to view extremely large image (or image collection) very efficiently, thanks to Deep Zoom. You can also write code to do a lot of interesting things that interacts with Deep Zoom.
Have a look at our CS/VBSL3DeepZoom sample! It is a sample demonstrating how to work with the cool Deep Zoom feature in Silverlight. What if you can zoom from a 粽子(zong zi in Chinese, means wrap in bamboo), to a 种子(zhong zi in Chinese, means seed), and then all the way to a 中子(zhong zi in Chinese, means neutron)? Do not lose your track. J
CS/VBSL3PlaneProjection (contributed by Allen)
If that’s not enough, you can easily integrate 3D contents into your application. Find how easy it is to create 3D effects with the help of the CS/VBSL3PlaneProjection sample. But note Silverlight doesn’t support advanced 3D scenarios as WPF does. If you need it, you will have to go with XBAP. We will write 3D WPF samples in the future.
The same sample can be ported to WPF 4 with minimum modifications (but you can’t port it to WPF 3).
And do not forget graphics in Silverlight are usually integrated with other elements. For example, you can customize the ControlTemplate of a Button so that it looks like a gun, yet you can still click it, and add some animation to tell the user the gun is fired (and maybe fire some bullets in the Click event handler). For a more practical sample, you can customize the DataTemplate of a ListBox, to draw a Rectangle that represents the population of a state. Tell you what? We also have a sample demonstrating how to use ControlTemplate. You can refer to the XAMLSL3StyleControlTemplate sample. We will also write some data binding samples in the future.
Basic concepts
OK, enough introductions to the samples. Now let’s review some basic computer graphic concepts that apply to all samples.
No matter what technology you choose, some basic concepts will not change. Do not forget it is the basic that is the most important. You may already know all of them, but it doesn’t harm to do a quick review.
Vector VS Bitmap
Computer is capable of rendering 2 kinds of graphics, vector graphics and bitmap graphics. They both have advantages and disadvantages.
Vector graphics advantages:
l Usually smaller (but not always)
l Can be scaled to any resolution without quality punishment
l Easier to animate (can animate a part of the graphic, think of eye blink and transitions)
l Integrate with the rest of the system more natively (do not rely on the Image Control or ImageBrush)
Vector graphics disadvantages:
l If a graph is too complex (contains thousands of paths), its size can increase a lot
l Difficult to render real world scenes with photo’s quality
l Difficult to serialize (unless converted to bitmap)
l Takes more time to render (although almost unnoticeable) because it has to be rasterized
Bitmap graphics advantages:
l Size is small when the resolution is low
l With a good camera, bitmaps can reproduce real world scenes in the form of photos
l Very easy to save
Bitmap graphics disadvantages:
l Size goes up as the resolution goes up
l Cannot scale without losing quality unless resolution (and thus size) is increased (Deep Zoom an solve this problem partially)
l Difficult to animate (animated PNG/GIF requires much more effort to create than animated vector graphics)
So you need to choose between vector and bitmap graphics carefully. Generally speaking, use vector graphics when:
l Need to zoom the graphic quickly without quality lose
l Need to render simple graphics in high resolution
l Need complex animations and transitions
l You’re creating a CAD kind of program
Use bitmap images when:
l Render photos
l Render small static images (such as icons)
l Required to save a screenshot (you can use WriteableBitmap in Silverlight to achieve this)
From CPU to GPU
No doubt today people want to delegate more and more work to GPU (think of Direct Compute, AKA Compute Shader, that delegates non-graphic related work to GPU). So let’s do a quick review about what you need to do when writing GPU aware applications.
The easiest solution to leverage GPU is to handle all jobs to an API. This is exactly what Direct2D does. So under the hook, what happens when you tell Direct2D to draw an ellipse? Well, a process called rasterization happens.
Rasterization is the process turning vector graphics into pixel information that can be rendered on the screen monitor. Before describing rasterization, it is important to remember most graphic cards only support 3 kinds of primitive geometries: point, line, and triangle. Triangle is usually used in 3D graphics. For now, let’s focus on 2D.
For a GPU based rasterization (2D only), usually the following steps will occur:
l Turn a path (such as ellipse) into a serial of lines. This step is done on CPU. Line is one kind of primitive geometries that can be handled by GPU directly.
l Map the brushes to textures.
l Pass the lines, as well as textures, to GPU.
l GPU checks if Vertex/Hull/Domain/Geometry Shader exist, and invoke them.
l GPU rasterizes every geometries to pixels.
l GPU checks if Pixel Shader exists, and invoke it.
l Render the scene on screen when needed.
Just for compare, for a CPU based rasterization, usually the following steps will occur:
l Turn a path (such as ellipse) into a serial of lines.
l Sort the lines in y order, and then in x order.
l Use scan lines to determine whether a pixel needs to be filled or not depending on the fill rule (such as even odd).
l Use the brushes to paint each pixel that is required to be painted.
l Perform anti-aliasing if required.
l Blend the rasterized path with the existing scene.
l If CPU supports SSE instructions, simulate Pixel Shader in CPU (slow).
Depending on the algorithm, the performance of CPU rasterization can differ a lot. For example, Direct2D’s software rasterization is much faster than GDI.
Anyway, when working with GPU, the most important concept is shaders. Today there’re 6 kinds of shaders:
l Vertex Shader: exists from shader 1.0, but unfortunately WPF and Silverlight doesn’t support it. It runs for each vertex of a 3D object. The input is each vertex’s position and color, and the output is the modified position and color of this vertex. Usually used to perform transformation.
l Hull Shader: Available in shader 5.0 only. Allows you to transform input control points that make up a patch into output control points. Please refer to Direct3D 11 documents for more information.
l Domain Shader: Available in shader 5.0 only. Used in conjunction with tessellator that allows you to render high detail scenes with low detail models. Please refer to Direct3D 11 documents for more information.
l Geometry Shader: Available since shader 4.0. Allows you to generate geometry primitives dynamically in GPU. This is extremely useful in scenarios such as particle engine and music visualizer.
l Pixel Shader: Available since shader 2.0. This is the only shader that is currently supported by WPF and Silverlight. It is the final stage in the graphics pipeline, and allows you to customize the rasterized pixels. A lot of bitmap effects that used to be done in CPU can be achieved using Pixel Shader.
l Compute Shader: Available in shader 5.0 only. This is unrelated to graphics. Computer Shader allows you to use GPU as a general purpose parallel processor.
If you want to use Pixel Shader 2.0, you can use WPF and/or Silverlight. This allows you to integrate the shader effects with the remaining of your system very easily. For other kinds of shaders, today the only option is Direct3D. Fortunately, both WPF and Direct2D can interop with Direct3D (Direct2D makes the interop even easier).
When you have to use CPU
Even though today most computers are equipped with a modern GPU, there’s still occasions when you have to use a CPU based solution. The most two common scenarios are virtual machines and server based rendering.
Most virtual machines cannot leverage host machine’s GPU. This cannot be workaround until the VM vendors support it.
Server based rendering solution is used extensively today in ASP.NET, usually using GDI. However, we recommend you to consider client side alternatives, such as Silverlight, whenever possible. By delegating more and more work to the client side, your server is free to serve more requests, and client side rendering is usually much faster than serve side rendering. However, we must admit server rendering will still be used by a lot of applications, especially since cloud computing has become more and more popular.
When using CPU based solution, Direct2D is the best choice. Direct2D supports both hardware rendering and software rendering. Its software rendering engine is well written, and is usually much faster than GDI and even WPF’s software rendering engine. So consider to switch to Direct2D from GDI if possible.
Conclusion
Windows 7 and Silverlight 3 provide us with a lot of cool graphic features, allowing you to create traditional software, as well as leverage the power of the latest graphic cards. There’re a lot computer graphic samples, and more will come later.
In the future, we will release more samples that cover desktop, web, mobile, and even cloud. Stay tuned!