Inside Direct3D -- Stencil Buffers

One aspect of advanced rendering we haven't discussed yet is stenciling, a technique that can be useful for developing commercial applications. If you want your 3D applications to stand apart from the crowd, you'd be wise to combine stenciling with the texturing techniques you learned about in earlier chapters. This chapter will detail how to use stenciling and show you the different types of effects you can generate with it.

Many 3D games and simulations on the market use cinema-quality special effects to add to their dramatic impact. You can use stencil buffers to create effects such as composites, decals, dissolves, fades, outlines, silhouettes, swipes, and shadows. Stencil buffers determine whether the pixels in an image are drawn. To perform this function, stencil buffers let you enable or disable drawing to the render-target surface on a pixel-by-pixel basis. This means your software can "mask" portions of the rendered image so that they aren't displayed.

When the stenciling feature is enabled, Microsoft Direct3D performs a stencil test for each pixel that it plans to write to the render-target surface. The stencil test uses a stencil reference value, a stencil mask, a comparison function, and a pixel value from the stencil buffer that corresponds to the current pixel in the target surface. Here are the specific steps used in this test:

By controlling the comparison function, the stencil mask, the stencil reference value, and the action taken when the stencil test passes or fails, you can control how the stencil buffer works. As long as the test succeeds, the current pixel will be written to the target. The default comparison behavior (the value that the D3DCMPFUNC enumerated type defines for D3DCMP_ALWAYS) is to write the pixel without considering the contents of the stencil buffer. You can change the comparison function to any function you want by setting the value of the D3DRENDERSTATE_STENCILFUNC render state and passing one of the members of the D3DCMPFUNC enumerated type.

Creating a Stencil Buffer

Before creating a stencil buffer, you need to determine what stenciling capabilities the target system supports. To do this, call the IDirect3DDevice7::GetCaps method. The dwStencilCaps flags specify the stencil-buffer operations that the device supports. The reported flags are valid for all three stencil-buffer operation render states: D3DRENDERSTATE_STENCILFAIL, D3DRENDERSTATE_STENCILPASS, and D3DRENDERSTATE_STENCILZFAIL. Direct3D defines the following flags for dwStencilCaps:

Direct3D embeds the stencil-buffer information with the depth-buffer data. To determine what formats of depth buffers and stencil buffers the target system's hardware supports, call the IDirect3D7::EnumZBufferFormats method, which has the following declaration:

HRESULT IDirect3D7::EnumZBufferFormats (
REFCLSID riidDevice,
LPD3DENUMPIXELFORMATSCALLBACK lpEnumCallback,
LPVOID lpContext
);

If the method succeeds, it returns the value D3D_OK. If it fails, the method returns one of these four values:

The code in listing 1 determines what stencil buffer formats are available and what operations are supported and then creates a stencil buffer. As you can see, this code notes whether the stencil buffer supports more than 1-bit -- some stenciling techniques must be handled differently if only a 1-bit stencil buffer is available.

Listing 1. Formats and Operations of Stencil Buffers

HRESULT CMyD3DApplication::CreateStencilBuffer()
{
g_bCanOnlyDoOneBitStencil=FALSE;

DWORD dwStencilCaps =
m_pDeviceInfo->ddDeviceDesc.dwStencilCaps;

if( (!(dwStencilCaps & D3DSTENCILCAPS_INCR) &&
!(dwStencilCaps & D3DSTENCILCAPS_INCRSAT)) ||
(!(dwStencilCaps & D3DSTENCILCAPS_DECR) &&
!(dwStencilCaps & D3DSTENCILCAPS_DECRSAT)))
{
// Must do 1-bit stencil buffer.
g_bCanOnlyDoOneBitStencil=TRUE;
}
else
{
// Prefer sat ops that cap at 0/max, but can
use other
// ones as long as enough stencil bits.
g_StencIncOp=(dwStencilCaps &
D3DSTENCILCAPS_INCRSAT)?
D3DSTENCILOP_INCRSAT:D3DSTENCILOP_INCR;
g_StencDecOp=(dwStencilCaps &
D3DSTENCILCAPS_DECRSAT)?
D3DSTENCILOP_DECRSAT:D3DSTENCILOP_DECR;
}

m_pddsRenderTarget->DeleteAttachedSurface( 0,NULL );

// Get z-buffer dimensions from the render target.
// Set up the surface description for the z-buffer.
DDSURFACEDESC2 ddsd;
D3DUtil_InitSurfaceDesc( ddsd );
m_pddsRenderTarget->GetSurfaceDesc( &ddsd );
ddsd.dwFlags = DDSD_WIDTH | DDSD_HEIGHT
| DDSD_CAPS | DDSD_PIXELFORMAT;
ddsd.ddsCaps.dwCaps = DDSCAPS_ZBUFFER;
ddsd.ddsCaps.dwCaps2 = 0;
ddsd.ddsCaps.dwCaps3 = 0;
ddsd.ddsCaps.dwCaps4 = 0;
ddsd.ddpfPixelFormat.dwFlags = DDPF_ZBUFFER | DDPF_STENCILBUFFER;

if( m_pDeviceInfo->bHardware )
ddsd.ddsCaps.dwCaps |= DDSCAPS_VIDEOMEMORY;
else
ddsd.ddsCaps.dwCaps |= DDSCAPS_SYSTEMMEMORY;

// Get an appropriate pixel format from an
enumeration of
// the formats.
m_pD3D->EnumZBufferFormats(
(*m_pDeviceInfo->pDeviceGUID),
EnumZBufferFormatsCallback,
(VOID*)&ddsd.ddpfPixelFormat );

assert(ddsd.ddpfPixelFormat.dwStencilBitDepth!=0);

g_bCanOnlyDoOneBitStencil=g_bCanOnlyDoOneBitStencil ||
((1<

g_dwMaxStencilValue=
(1<

// Leave g_bUseOneBitStencil set for the
window-resize case.
if( !g_bUseOneBitStencil )
g_bUseOneBitStencil=g_bCanOnlyDoOneBitStencil;

SetMenuStates();

// Create and attach a z-buffer.
if( FAILED( m_pDD->CreateSurface( &ddsd,
&m_pddsDepthBuffer,
NULL ) ) )
return E_FAIL;

if( FAILED(m_pddsRenderTarget->AddAttachedSurface(
m_pddsDepthBuffer ) ) )
return E_FAIL;

// The SetRenderTarget() call is needed to rebuild
internal
// structures for the newly attached z-buffer.
return m_pd3dDevice->SetRenderTarget
( m_pddsRenderTarget, 0L );
}

//------------------------------------------------------
// Name: EnumZBufferFormatsCallback
// Desc: Enumeration function to report valid pixel
// formats for z-buffers
//------------------------------------------------------
static
HRESULT WINAPI EnumZBufferFormatsCallback(
DDPIXELFORMAT* pddpf,
VOID* pddpfDesired )
{
if( NULL==pddpf || NULL==pddpfDesired )
return D3DENUMRET_CANCEL;

// If the current pixel formats match the desired
// ones (DDPF_ZBUFFER and possibly
// DDPF_STENCILBUFFER), copy it and return. This
// function isn't choosy--it accepts the first
// valid format that comes along.

if( pddpf->dwFlags == ((DDPIXELFORMAT*)
pddpfDesired)->dwFlags )
{
memcpy( pddpfDesired, pddpf,
sizeof(DDPIXELFORMAT) );
return D3DENUMRET_CANCEL;
}

return D3DENUMRET_OK;
}

Clearing a Stencil Buffer

The IDirect3DDevice7 interface includes the Clear method, which you can use to simultaneously clear the render target's color buffer, depth buffer, and stencil buffer. Here's the declaration for the IDirect3DDevice7::Clear method:

HRESULT IDirect3DDevice7::Clear(
DWORD dwCount,
LPD3DRECT lpRects,
DWORD dwFlags,
D3DVALUE dvZ,
DWORD dwStencil
);

The IDirect3DDevice7::Clear method still accepts the older D3DCLEAR_TARGET flag, which clears the render target using an RGBA color you provide in the dwColor parameter. This method also still accepts the D3DCLEAR_ZBUFFER flag, which clears the depth buffer to a depth you specify in dvZ (in which 0.0 is the closest distance and 1.0 is the farthest). DirectX 6 introduced the D3DCLEAR_STENCIL flag, which you can use to reset the stencil bits to the value you specify in the dwStencil parameter. This value can be an integer ranging from 0 to 2n-1, in which n is the bit depth of the stencil buffer.

Configuring the Stenciling State

You control the various settings for the stencil buffer using the IDirect3DDevice7::
SetRenderState method. Listing 2 shows the stencil-related members of the D3DRENDERSTATETYPE enumerated type.

Listing 2. Control Settings of Stencil Buffer

typedef enum _D3DRENDERSTATETYPE {
§
D3DRENDERSTATE_STENCILENABLE = 52,
// Enable or disable
// stenciling
D3DRENDERSTATE_STENCILFAIL = 53,
// Stencil operation
D3DRENDERSTATE_STENCILZFAIL = 54,
// Stencil operation
D3DRENDERSTATE_STENCILPASS = 55,
// Stencil operation
D3DRENDERSTATE_STENCILFUNC = 56,
// Stencil comparison
// function
D3DRENDERSTATE_STENCILREF = 57,
// Reference value for
// stencil test
D3DRENDERSTATE_STENCILMASK = 58,
// Mask value used in
// stencil test
D3DRENDERSTATE_STENCILWRITEMASK = 59,
// Stencil-buffer write
// mask
§
} D3DRENDERSTATETYPE;

These are the definitions for the stencil-related render states:

The D3DSTENCILOP enumerated type describes the stencil operations for the D3DRENDERSTATE_STENCILFAIL, D3DRENDERSTATE_STENCILZFAIL, and D3DRENDERSTATE_STENCILPASS render states. Here's the definition of D3DSTENCILOP:

typedef enum _D3DSTENCILOP {
D3DSTENCILOP_KEEP = 1,
D3DSTENCILOP_ZERO = 2,
D3DSTENCILOP_REPLACE = 3,
D3DSTENCILOP_INCRSAT = 4,
D3DSTENCILOP_DECRSAT = 5,
D3DSTENCILOP_INVERT = 6,
D3DSTENCILOP_INCR = 7,
D3DSTENCILOP_DECR = 8,
D3DSTENCILOP_FORCE_DWORD = 0x7fffffff
} D3DSTENCILOP;

These members serve the following purposes:

Let's walk through some code that uses the stencil buffer while rendering a scene. This code is from a sample that shows how to draw shadows. For now, don't worry about how all this code generates shadows-the algorithm is described later in the chapter.

The shadow-rendering code starts out by disabling the depth buffer and enabling the stencil buffer:

//--------------------------------------------------
// Name: RenderShadow
// Desc:
//------------------------------------------------
HRESULT CMyD3DApplication::RenderShadow()
{
// Turn off depth buffer and turn on
stencil buffer.
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_ZWRITEENABLE,
FALSE );
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILENABLE,
TRUE );

Next the code sets the comparison function that performs the stencil test by calling the IDirect3DDevice7::SetRenderState method and setting the first parameter to D3DRENDERSTATE_STENCILFUNC. The second parameter is set to a member of the D3DCMPFUNC enumerated type. In this code, we want to update the stencil buffer everywhere a primitive is rendered, so we use D3DCMP_ALWAYS:

//
// Set up stencil comparison function,
reference value, and masks.
// Stencil test passes if ((ref & mask)
cmpfn (stencil & mask))
// is true.
//
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILFUNC,
D3DCMP_ALWAYS );

In this sample, we don't want the stencil buffer to change if either the stencil buffer test or the depth buffer test fails, so we set the appropriate states to D3DSTENCILOP_KEEP:

m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILZFAIL,
D3DSTENCILOP_KEEP );
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILFAIL,
D3DSTENCILOP_KEEP );

The settings in listing 3 are different depending on whether a 1-bit or a multibit stencil buffer is present. If the stencil buffer has only 1 bit, the value 1 is stored in the stencil buffer whenever the stencil test passes. Otherwise, an increment operation (either D3DSTENCILOP_INCR or D3DSTENCILOP_INCRSAT) is applied if the stencil test passes. At this point, the stencil state is configured and the code is ready to render some primitives.

Listing 3. Multibit Stencil Buffer

if(g_bUseOneBitStencil)
{
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILREF,
0x1 );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILMASK,
0x1 );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILWRITEMASK,
0x1 );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILPASS,
D3DSTENCILOP_REPLACE );
}
else
{
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILREF,
0x1 );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILMASK,
0xffffffff );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILWRITEMASK,
0xffffffff );
pd3dDevice->SetRenderState
(D3DRENDERSTATE_STENCILPASS,
g_StencIncOp );
}

Creating Effects

Now that you've seen how to create stencil buffers and configure how they work, let's look at some of the effects you can render with them. The following sections describe several ways Microsoft recommends using stencil buffers. Each of these approaches produces impressive results, but a few of them have drawbacks.

Composites

You can use stencil buffers for compositing 2D or 3D images onto a 3D scene. By using a mask in the stencil buffer to occlude a portion of the render-target surface, you can write stored 2D information (such as text or bitmaps). You can also render 3D primitives -- or for that matter a complete scene -- to the area of the render-target surface that you specify in a stencil mask.

Developers often use this effect to composite several scenes in simulations and games. Many driving games feature a rear view mirror that displays the scene behind the driver. You can composite this second 3D scene with the driver's view forward by using a stencil to block the portion to which you want the mirror image rendered. You can also use composites to create 2D "cockpits" for vehicle simulations by combining a 2D, bitmapped image of the cockpit with the final, rendered 3D scene.

Decals

You can use decals to control which pixels form a primitive image you draw to a render-target surface. When you apply a texture to an object (for example, applying scratch marks to a floor), you need the texture (the scratch marks) to appear immediately on top of the object (the floor). Because the z values of the scratch marks and the floor are equal, the depth buffer might not yield consistent results, meaning that some pixels in the back primitive might be rendered on top of those in the front primitive. This overlap, which is commonly known as z-fighting or flimmering, can cause the final image to shimmer as you animate from one frame to the next.

You can prevent flimmering by using a stencil to mask the section of the back primitive on which you want the decal to appear. You can then turn off z-buffering and render the image of the front primitive into the masked area of the render-target surface.

Dissolves

You can use dissolves to gradually replace an image by displaying a series of frames that transition from one image to another. In Chapter 8, you saw how to use multiple-texture blending to create this effect by gradually blending two textures together. Stencil buffers allow you to produce similar dissolves, except that a stencil-based dissolve looks more pixelated than a multiple-texture blending one. However, stencil buffers let you use texture-blending capabilities for other effects while performing a dissolve. This capability enables you to efficiently produce more complex effects than you could by using texture blending alone.

A stencil buffer can perform a dissolve by controlling which pixels you draw from two different images to the render-target surface. You can perform a dissolve by defining a base stencil mask for the first frame and altering it incrementally or by defining a series of stencil masks and copying them into the stencil buffer on successive frames.

To start a dissolve, set the stencil function and stencil mask so that most of the pixels from the starting image pass the stencil test and most of the ending image's pixels fail. For each subsequent frame, update the stencil mask to allow fewer pixels in the starting image to pass the test and more pixels in the ending image to pass. By controlling the stencil mask, you can create a variety of dissolve effects.

Although this approach can produce some fantastic effects, it can be a bit slow on some systems. You should test the performance on your target systems to verify that this approach works efficiently for your application.

Fades

You can fade in or out using a form of dissolving. To perform this effect, use any dissolve pattern you want. To fade in, use a stencil buffer to dissolve from a black or white image to a rendered 3D scene. To fade out, start with a rendered 3D scene and dissolve to black or white. As with dissolves, you should check the performance of fades on the target systems to verify that their speed and appearance is acceptable.

Outlines

You can apply a stencil mask to a primitive that's the same shape but slightly smaller than the primitive. The resulting image will contain only the primitive's outline. You can then fill this stencil-masked area of the primitive with a color or set of colors to produce an outline around the image.

Silhouettes

When you set the stencil mask to the same size and shape as the primitive you're rendering, Direct3D produces a final image containing a "black hole" where the primitive should be. By coloring this hole, you can produce a silhouette of the primitive.

Swipes

A swipe makes an image appear as though it's sliding into the scene over another image. You can use stencil masks to disable the writing of pixels from the starting image and enable the writing of pixels from the ending image. To perform a swipe, you can define a series of stencil masks that Direct3D will load into the stencil buffer in a succession of frames, or you can change the starting stencil mask for a series of successive frames. Both methods cause the final image to look as though it's gradually sliding on top of the starting image from right to left, left to right, top to bottom, and so on.

To handle a swipe, remember to read the pixels from the ending image in the reverse order in which you're performing the swipe. For example, if you're performing a swipe from left to right, you need to read pixels from the ending image from right to left. As with dissolves, this effect can render somewhat slowly. Therefore, you should test its performance on your target systems.

Shadows

Shadow volumes, which allow an arbitrarily shaped object to cast a shadow onto another arbitrarily shaped object, can produce some incredibly realistic effects. To create shadows with stencil buffers, take an object you want to cast a shadow. Using this object and the light source, build aset of polygonal faces (a shadow volume) to represent the shadow.

You can compute the shadow volume by projecting the vertices of the shadow-casting object onto a plane that's perpendicular to the direction of light from the light source, finding the 2D convex hull of the projected vertices (that is, a polygon that "wraps around" all the projected vertices), and extruding the 2D convex hull in the light direction to form the 3D shadow volume. The shadow volume must extend far enough so that it covers any objects that will be shadowed. To simplify computation, you might want the shadow caster to be a convex object.

To render a shadow, you must first render the geometry and then render the shadow volume without writing to the depth buffer or the color buffer. Use alpha blending to avoid having to write to the color buffer. Each place that the shadow volume appears will be marked in the stencil buffer. You can then reverse the cull and render the backfaces of the shadow volume, unmarking all the pixels that are covered in the stencil buffer. All these pixels will have passed the z-test, so they'll be visible behind the shadow volume. Therefore, they won't be in shadow. The pixels that are still marked are the ones lying inside the front and back boundaries of the shadow volume-these pixels will be in shadow. You can blend these pixels with a large black rectangle that covers the viewport to generate the shadow.

The ShadowVol and ShadowVol2 Demos

The ShadowVol sample on the companion CD in the \mssdk\Samples\Multimedia\D3dim\Src\ShadowVol directory contains a project that shows how to create and use stencil buffers to implement shadow volumes. The code illustrates how to use shadow volumes to cast the shadow of an arbitrarily shaped object onto another arbitrarily shaped object. The ShadowVol2 sample, which the Microsoft DirectX 7 SDK setup program on the companion CD installs in the \mssdk\Samples\Multimedia\D3dim\Src\ShadowVol2 directory on your hard disk, provides some additional capabilities for producing shadows with stencils.

The sample application provides these features in its Shadow Modes menu:

Figure 12-1, Figure 12-2, and Figure 12-3 show three views of the scene generated by the ShadowVol2 sample application. You can see the shadows in Figures 12-1 and 12-3; Figure 12-2 illustrates the shadow volumes.

The Code So Far

In this chapter, we didn't add any new code to the RoadRage project. To see these effects in action, refer to the ShadowVol and ShadowVol2 demo projects included in the DirectX samples.

Conclusion

In this chapter, you learned about stencil buffers and the exciting effects they can produce. In today's market, making your code stand out is a requisite if you want it to sell your applications and keep your users coming back for more. Incorporating strategic stencil-buffer effects into the introduction and into the body of a 3D real-time game might help you win over even the most discriminating game players.

In Chapter 13, we'll discuss how to load and animate 3D models. Creating animated, lifelike characters that your users can interact with is one of the most powerful capabilities you can add to any game.

Peter Kovach has been involved in computer software and hardware development since the mid-1970s. After 11 years in various levels of development and project management, he was eager to being pushing the envelope in 3D virtual world development. He currently words at Medtronic, where he is the project lead developming programmable, implantable medical devices that use a next-generation graphical user interface.