History: OgreStereoVisionProblems

            !My game looks awfull using stereoscopy with NVIDIA's Vision drivers

This article assumes you're a bit familiar with [http://www.nvidia.com/object/3d-vision-main.html|NVIDIA's 3D Vision] technology, and you're having trouble with it.

Vision technology is great. Turn on a button in the Control Panel, calibrate the glasses, and you're ready to go. You're amazed by how all you're games are now experienced in stereoscopic 3D, except you now go run your own Ogre application... and it's all screwed!

__Just like in this picture:__
{img fileId="1929" thumb="y" alt="" rel="box[g]"}

Your first suspicion is a driver bug. It's a very new technology so this would be no surprise. Even many AAA games don't actually work fine with it. You start playing with your code, and found out it has (most likely) something to do with hardware skeletal animations. You start seeing all NVIDIA's slides from GDC & SIGGRAPH about their vision drivers (since there's not too much documentation) and spot that the Vision driver patches your vertex shaders on the fly to produce left & right eye images.
''Conclusion'': the driver can't handle my skeletal animation code.

Guess again... __you're wrong. It's a bug in your code__. NVIDIA drivers are just fine.

!!Cause of the problem:
Your vertex shader probably looks something like this:
{CODE(wrap="1", colors="c++")}float4 BlendedPos = 0;
int i;
for( i=0; i < NUM_BONES_PER_VERTEX; ++i )
	BlendedPos += float4( mul(worldMatrix3x4Array[inBlendIdx[i]], inPosition).xyz, 1.0f ) * inBlendWght[i];

outPosition = mul( viewProjMatrix, BlendedPos );{CODE}

This shader works fine without stereoscopy, and it's in fact the same code that ships with the Ogre samples, and probably the same code in many, many AAA games.
''But when you turn on stereoscopy, it blows up. What's wrong?''

What's wrong is that this code operates with the __assumption that the sum of all weights are 1.0__.
In other words:
{BOX()}~np~inBlendWght[0] + inBlendWght[1] + ... + inBlendWght[n] = 1~/np~{BOX}

When you're using more than one bone per vertex, usually Ogre does this for you in the function Mesh::_rationaliseBoneAssignments()
But chances are, sometimes (specially when using only one bone per vertex) this function didn't fix it for you.

Let's analyze what happens when inBlendWght~np~[0]~/np~ is 0.8, and NUM_BONE_PER_VERTEX is 1:
{BOX()}
~np~BlendedPos += float4( mul(worldMatrix3x4Array[inBlendIdx[i]], inPosition).xyz, 1.0f ) * 0.8f;~/np~
outPosition = mul( viewProjMatrix, BlendedPos );{BOX}

Later, the GPU projects the vertices, between the vertex and pixel shader stages:
{BOX()}pixelPos = outPosition.xyz / outPosition.w;{BOX}

Let's analyze it again, but putting the weights outside of the variable:
{BOX()}pixelPos = (outPosition.xyz * 0.8) / (outPosition.w * 0.8);{BOX}

0.8 is both in the numerator and denominator, therefore they ''cancel each other'', __and is the same as if all vertices had ~np~inBlendWght[0]~/np~ = 1.0!__
The result, ''just out of luck'', is rendered correctly.


!!!If it's rendered correctly, then why it isn't in stereoscopy?
If you read the GDC papers from NVIDIA again, __the vision drivers' magic lies in using the W component from your vertex__. Althought xyz / w gives a reasonable correct result, __the w value is ''still'' wrong__, and hence you get serious artifacts. Vision driver assumes your W coordinate is correct.


!!!Ok then, how do I fix it?
There are two ways to fix it:
*__The easy way:__ __FIX YOUR MESH__. Go back to Blender/Maya/3DS Max and tell your modeller to assign ''all'' the vertices a blend weight of 1.0, or make a simple tool that loads your .mesh files, modifies the blend weights, and saves the mesh again. Or do this at runtime. If you're using Blender 2.49b, you may want to check out [http://www.ogre3d.org/forums/viewtopic.php?f=8&t=64887&p=429046|use this script] which performs it automatically.

*__The hard way:__ Modify your shader. In some cases, the artist put on ''purpose'' a weight different than 1. And this is critical for you. Therefore, you will need to take into account the vertex position using the world matrix alone (not the one passed from the world array), with the remainder of the weight.

The modified code should look like this:
{CODE(wrap="1", colors="c++")}float4 worldWeight = 0;
for( i=0; i < NUM_BONES_PER_VERTEX; ++i )
	worldWeight += inBlendWght[i];

worldWeight = 1.0f / worldWeight;

float4 BlendedPos = mul( worldMatrix, inPosition ) * worldWeight; //worldMatrix is using world_matrix material binding
int i;
for( i=0; i < NUM_BONES_PER_VERTEX; ++i )
	BlendedPos += float4( mul(worldMatrix3x4Array[inBlendIdx[i]], inPosition).xyz, 1.0f ) * inBlendWght[i];{CODE}


!!Moral of the article
This is a single example of an application going mad when stereoscopy is turned on. When you get artifacts, always ensure you're computing the correct W component. Almost always this is the cause of the problem, since the whole technology actually boils down to this. There's not much more to it.


!!!Useful links:
#[http://developer.download.nvidia.com/presentations/2009/SIGGRAPH/3DVision_Develop_Design_Play_in_3D_Stereo.pdf|NVIDIA SIGGRAPH Paper]
#[http://developer.download.nvidia.com/presentations/2008/GDC/GDC08_stereo-web.pdf|NVIDIA GDC 2008 Paper]