Head Tracking using the WiiMote         Using this tutorial, you should be able to "create" a Desktop VR Display like Johnny Lee demonstrated in his Video "Head Tracking for Desktop VR Display using the WiiRemote"

Introduction

Using this tutorial, you should be able to "create" a Desktop VR Display like Johnny Lee demonstrated in his Video "Head Tracking for Desktop VR Display using the WiiRemote". Changing the view and projection matrix (like Johnny Lee did it) didn't work for me in Ogre, so I had to find my own way to achieve such an experience, but more on that later.

Prerequisites

  • basic Ogre knowledge (you should have read (and understood!) the basic tutorials)
  • running version of Ogre
  • a WiiMote library (e.g. WiiYourself!)
  • basic knowledge of 3D math, if you want to change some values (e.g. FOV values)

Creating a Scene

First of all, we create a Scene like in the Video (you should be familiar with that, so I don't explain it in depth):

The targets

int numInFront = 2;
float screenAspect = mWindow->getWidth() / mWindow->getHeight();
float depthStep = 2.5f;
float startDepth = -numInFront*depthStep;

for (int i = 0; i < 10; i++)
{
    Plane plane;
    plane.normal = Vector3::NEGATIVE_UNIT_Z;
    MeshManager::getSingleton().createPlane("target"+StringConverter::toString(i), ResourceGroupManager::DEFAULT_RESOURCE_GROUP_NAME, plane, 1, 1);
    Entity* target = mSceneMgr->createEntity("targete"+StringConverter::toString(i), "target"+StringConverter::toString(i));
    target->setMaterialName("headtracking/target");
    target->setCastShadows(false);
    SceneNode* targetNode = mSceneMgr->getRootSceneNode()->createChildSceneNode("targetNode"+StringConverter::toString(i));
    targetNode->attachObject(target);

    targetNode->setPosition(Vector3(-2.5f + (rand() % 5) + (rand() % 10) / 10,
                    -2.5f + (rand() % 5) + (rand() % 10) / 10,
                    startDepth + i * depthStep));
    if (i < numInFront)
    {
        targetNode->setPosition(targetNode->getPosition().x * .5f, targetNode->getPosition().y * .5f, targetNode->getPosition().z); 
    }

    // draw line
    ManualObject* line = mSceneMgr->createManualObject("line" + StringConverter::toString(i));
    line->clear();
    line->begin("", RenderOperation::OT_LINE_LIST);
    line->position(targetNode->getPosition());
    line->colour(ColourValue(1.0, 1.0, 1.0));
    line->position(targetNode->getPosition() + Vector3(0, 0, 30));
    line->end();
    targetNode->attachObject(line);
}

The stadium

Plane plane;
plane.normal = Vector3::NEGATIVE_UNIT_Z;
MeshManager::getSingleton().createPlane("stad", ResourceGroupManager::DEFAULT_RESOURCE_GROUP_NAME, plane, 49, 20);
Entity* stadentity = mSceneMgr->createEntity("stadentity", "stad");
stadentity->setMaterialName("headtracking/stad");
stadentity->setCastShadows(false);
SceneNode* stadNode = mSceneMgr->getRootSceneNode()->createChildSceneNode("stadNode");
stadNode->attachObject(stadentity);
stadNode->setPosition(Vector3(0,0,10));
stadNode->setVisible(false);


Note: the picture of the stadium is hidden at the beginning. If you want to hide the targets and show the picture, assign a key for example and change the visibility.

mSceneMgr->getSceneNode("stadNode")->setVisible(false);
for (int i=0; i<10; i++) {
    mSceneMgr->getSceneNode("targetNode"+StringConverter::toString(i))->setVisible(true);
}

Camera position

If you use my above code, a nice camera position is at Vector3(0,0,0) and the camera looks along the positive z-axis.

Additional variables

Now we need some information about the connected hardware, which we define in class variables and some other class variables:

bool mUseWiiMotes = true;                        // to disable wiiMote support
Vector3 mHeadPosition = Vector3::ZERO;                    // position of the users head when the last frame was rendered
float mHeadX = 0;                            // last calculated X position of the users head
float mHeadY = 0;                            // last calculated Y position of the users head
float mHeadDist = 0;                            // last calculated Z position of the users head
const float mRadiansPerPixel = (float)(Math::PI / 4.0f) / 1024.0f;    // don't change this! it's a fixed value for the WiiMote infrared camera
float mIRDotDistanceInMM = 8.5f * 25.4f;                // distance of the IR dots in mm. change it, if you are not using the original nintendo sensor bar
float mScreenHeightInMM = 20.0f * 25.4f;                // height of your screen
bool mWiiMoteIsAboveScreen = true;                    // is the WiiMote mounted above or below the screen?
float mWiiMoteVerticleAngle = 0;                    // vertical angle of your WiiMote (as radian) pointed straight forward for me.

The WiiMote

Connecting a WiiMote

I use the WiiYourself! library to connect my WiiMote to my PC, but you can use any other library you want, if you don't like it. There are only a few changes that have to be done.

Add another class variable for the WiiMote.

#include "wiimote.h"

wiimote mWiiMote;                            // our wiimote object


Now we have to establish a connection to the WiiMote. Let's look for one once a second for three times. If nothing was found after that continue without WiiMote support and give the user an error message. Otherwise enable the first LED and set the right report type. (For detailed information about report types look at the WiiYourself! demo project, LED states can be found here)

if (mUseWiiMotes) {
    int count = 0;
    while (!mWiiMote.Connect(wiimote::FIRST_AVAILABLE) && count < 3) {
        count++;
        Sleep(1000);
    }

    if (!mWiiMote.IsConnected()) {
        MessageBox(NULL, "Can't find any WiiMote. Continuing without WiiMote Support!", "Can't find WiiMote!", MB_OK);
        mUseWiiMotes = false;
    }
    else {
        mWiiMote.SetLEDs(0x01);
        mWiiMote.SetReportType(wiimote::IN_BUTTONS_ACCEL_IR_EXT);
    }
}

Parsing the data from the WiiMote

For parsing I created a new function in my Application, which returns true if it calculated new data or false otherwise. First we check if we use the WiiMote, if the connection is still alive and if the WiiMote has some new data for us.

bool <someClass>::parseWiiMoteData(void)
{
    if (!mUseWiiMotes)
        return false;

    if (mWiiMote.ConnectionLost()) {
        MessageBox(NULL, "Connection to WiiMote lost. Continuing without WiiMote Support!", "Lost Connection", MB_OK);
        mUseWiiMotes = false;
        return false;
    }

    if (mWiiMote.RefreshState() == NO_CHANGE)
        return false;


Now we have to get two points from the infrared camera.

Vector2 firstPoint = Vector2();
    Vector2 secondPoint = Vector2();
    int numvisible = 0;

    for (int index = 0; index < 4; index++) {
        if (mWiiMote.IR.Dot[index].bFound) {
            if (numvisible == 0) {
                firstPoint.x = mWiiMote.IR.Dot[index].RawX;
                firstPoint.y = mWiiMote.IR.Dot[index].RawY;
                numvisible = 1;
            }
            else if (numvisible == 1) {
                secondPoint.x = mWiiMote.IR.Dot[index].RawX;
                secondPoint.y = mWiiMote.IR.Dot[index].RawY;
                numvisible = 2;
                break;
            }
        }
    }


Now comes the "tricky" part: calculating the head position.

if (numvisible == 2) {
        float dx = firstPoint.x - secondPoint.x;
        float dy = firstPoint.y - secondPoint.y;
        float pointDist = (float)Math::Sqrt(dx * dx + dy * dy);

        float angle = mRadiansPerPixel * pointDist / 2;

        mHeadDist = (float)((mIRDotDistanceInMM / 2) / Math::Tan(angle)) / mScreenHeightInMM;

        float avgX = (firstPoint.x + secondPoint.x) / 2.0f;
        float avgY = (firstPoint.y + secondPoint.y) / 2.0f;

        mHeadX = (float)(Math::Sin(mRadiansPerPixel * (avgX - 512)) * mHeadDist);

        float relativeVerticalAngle = (avgY - 384) * mRadiansPerPixel;

        if(mWiiMoteIsAboveScreen)
            mHeadY = .5f + (float)(Math::Sin(relativeVerticalAngle + mWiiMoteVerticleAngle) * mHeadDist);
        else
            mHeadY = -.5f + (float)(Math::Sin(relativeVerticalAngle + mWiiMoteVerticleAngle) * mHeadDist);

        return true;
    }
    
    return false;
}

Adjusting the camera

Now we "just" have to use the above calculated data. To achieve such a VR experience, we need to do several things:

  • a camera translation
  • adjust the cameras orientation
  • adjust the cameras FOV



The camera translation

This is the easy part. Just move the camera from the old head position to the new one.

Changing the orientation

The camera position is now equal to the position of the users head and we also want to rotate the camera a little bit, so if we move left we want to rotate the camera around a point (which (in the real world) is our screen). As we move the camera position in our 3D world, we can't assume that it's always the origin. So we need to calculate the point we look at. This is the camera position + the distance of the users head to the screen into the cameras direction.

Changing the FOV

As the last part, we need to adjust the cameras FOV. So we want to see more of our 3D world if we get closer to the screen, just like we are looking through a real window. The formula gives you a FOV of 100° at 40cm distance and 30° at 4m distance.

Complete Code

Put this code into the frameStarted method.

if (parseWiiMoteData()) {
    Vector3 newHeadPosition = Vector3(mHeadX, mHeadY, -mHeadDist);
    Vector3 lookAt = mCamera->getPosition() + (mCamera->getDirection().normalisedCopy() * -mHeadPosition.z);
    mCamera->move(newHeadPosition - mHeadPosition);
    mCamera->lookAt(lookAt);
    mHeadPosition = newHeadPosition;
    mCamera->setFOVy(Radian(Degree(107 - 0.1944 * mHeadDist * mScreenHeightInMM/10)));
}