I’ll assume that our DirectX professor, Louis Natanson, was purposefully creating a learning experience when he provided the sample code on how to use the Direct3D stencil buffer.
As you may be aware, a common use of the stencil buffer is in the creation of object shadows. So Louis’s sample code does just that … it renders the shadow of a rotating cube onto a plane. And in his example, in addition to setting up the stencil buffer and related functions, he uses the D3D function D3DXMatrixShadow() to create the shadow.
However, after studying the code, I was having some serious difficulty understanding how the information generated in the stencil buffer was actually being rendered onto the plane. If my memory serves me correctly, in OpenGL you had to capture the resultant stencil buffer as a texture and then apply that texture to a plane (or something like that.)
When I mentioned this to my classmates, one of them (David) expressed the same concern. On further examination, we assured ourselves that in fact, the D3D function D3DXMatrixShadow() does not use the stencil buffer, at all!
I’d now like to take a few moments to describe what I have learned since then about D3DXMatrixShadow(), and issues to consider when implementing it in your own code. In order to view the D3D subtleties better, I use the more complicated shape of a barrel. I’ll start with the following quote form MSDN:
The D3DXMatrixShadow function flattens geometry into a plane, as if casting a shadow from a light.
In other words, the function simply creates a transformation matrix where the entire scene is flattened onto a plane. I believe the wireframe screen shot below helps to illustrate the functionality very clearly.
With that in mind, it should be very clear that the next issue you will face is that the flattened object still contains all the polygons and textures of the regular object. As a result you’ll get a shadow that has serious depth buffer issues.
Removing the textures is an obvious first step, however the depth is still an issue.
So as Louis did in his code, I also disabled the depth buffer. Which leads immediately to a similar issue you’ll find in Louis’s sample program … when viewed from above, without a depth buffer, the shadow is on TOP of the object!
However this is easy to fix. If you turn the depth buffer OFF when drawing your shadow, you just need to ensure that you draw your objects in the correct order.
- draw your surface
- turn off the depth buffer
- draw your shadows
- turn on the depth buffer
- draw you objects
The next thing you might notice about Louis’s program is that he applies a semi-transparent material to the shadow in order to give a nice shadowy effect. Specifically, he uses Alpha-Blending via the following Alpha function code:
Device->SetRenderState(D3DRS_ALPHABLENDENABLE, true);
Device->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_SRCALPHA);
Device->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_INVSRCALPHA);
Although not an issue with the simple cube, the fact that the shadow is an actual flattened 3D object, the alpha blending gives a funny result, since the intensity of the alpha is modified when passing through each level of the object.
Again, an easy fix is to just disable the use of alpha-blending and you finally get a pretty nice looking image. (Of course ignoring the fact that the shadow is not aligned with the image of the sun on the skybox.)
So, now there are three things remaining for me to do.
- Align light and shadow with image of sun in the skybox.
- See if a different alpha function can be used to allow the use of semi-transparent shadows without the multi-layered alpha issue.
- Learn how to actually USE the stencil buffer.
Update
Louis responded via email, and now it’s all clear to me! The purpose of the stencil buffer is to prevent the layering effect of the alpha blending. In other words, as you draw the flattened image of the barrel, only draw it once per-layer. In essence, the stencil buffer keeps track of what pixels have already been drawn once, which can be used to prevent the multiple layers in the alpha blending.
