Ogre on a Cluster         While many will no doubt never have to run Ogre on a cluster of computers, the procedure to do so could be useful for others who have no root privileges and thus require a special solution

While many will no doubt never have to run Ogre on a cluster of computers, the procedure to do so could be useful for others who have no root privileges and thus require a special solution. Taking into account the following:

We are using Linux:

  • I haven't gotten around to Mac OS X, but I think it's far easier because we only have to configure Chromium and I'm pretty sure there is a DMX package in Macports (if not, you can compile it yourself, no biggie, after all it's BSD).
  • We are using Eihort 1.4.5 (because Freeimage has everything we need)
  • We have at least have Chromium installed (henceforth you have more than one computer and at least a monitor for each). The advantage of the latter setup is that you may expand this to 4 monitors (and active stereo) by tweaking the Nvidia/Ati drivers.
  • For a proper multihead display, we have dmx installed and configured with chromium in mind (this allows us to expand the X desktop on all four monitors). If you choose to have four monitors, using only two nodes, preferably add a third node to make the main node (where your account and most of your graphics apps are) a separate node from the ones that make up the display (to have more control).
  • We have had some prior experience building Ogre and it's dependencies
  • We have a lot of patience.


Image

Introduction


For Chromium and it's configuration, see here.

For DMX and it's configuration, see this page.

For a thorough walkthrough, using Debian, our pals at PennState have a short tutorial.

For the most part, a user of any graphics cluster usually doesn't have access to /usr or /usr/local (where libraries and headers are often found) and most administers are reluctant to ever bless your account with sudoers privileges. Even if you got the administrator to install it himself, the installation will be prone to error and is not exactly an ideal solution if you are developing/extending/tweaking Ogre.

The added caveat of installing it in your home directory is that if anything should go wrong, you just rm -rf your local directory and start from scratch with the src directory that you most obviously saved. There is a trifle matter in which you probably won't ever install anything in /usr or /usr/local again since it is far more convenient to have everything locally (yes even mplayer) when you have to move to another machine, net, etc (in which case you may or may not have to recompile your libraries). These advantages rely a lot on how comfortable you are on handling environment variables.

Installation

Fortunately, Ogre relies on very few non-standard commodity libraries (GTK+, Glademm2 are usually found already installed). For this I will walk you through the installation of Ogre on CentOS 5.0 (which comes to being the closest we can get, legally speaking, to installing Redhat Enterprise WS). I personally use Gentoo all the time, but it's uncommon on most cluster platforms (both commercial and self-made) being that it's tiresome to install on more than 2 machines.

Our first step would be to make our local folder (we'll just call it local) and chmod it 755:

$mkdir local
$chmod 755 local

Now we make a text file in our home containing our environment variables (so that we don't have to type environment variables like nuts every time we open a terminal) like this, where %USER is replaced by your real username:

export PATH=/home/%USER/local/bin:$PATH
export LD_LIBRARY_PATH=/home/%USER/local/lib/:/usr/local/lib:$LD_LIBRARY_PATH
export PKG_CONFIG_PATH=/home/%USER/local/lib/pkgconfig:/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH

The fact that I include /usr/local/ in some environment is precautionary, most likely you will have everything else in /usr and both ldconfig and pkgconfig always look in there for compilation flags and runtime libraries.

Here is another example so you can get the hang of it(Far more cluttered and tailored to run Vrjuggler in local fashion from another user's account):

###########In case I need to separate ENVIRONS for OSG
#export PATH=/home/jp/bin:/home/jp/bin/blender-2.45-linux-glibc236-py24-i386:/home/jp/bin/firefox/:/home/jp/bin/thunderbird:/home/jp/bin/mupen64-0.5:/home/jp/bin/RealPlayer/:/opt/Mercury/Amira4.1
:$PATH
###########Usual Stuff
export PATH=$PATH:/usr/X11R6/bin/
export PATH=/home/jp/local/bin:/home/jp/bin:/home/jp/bin/blender-2.45-linux-glibc236-py24-i386:/home/jp/bin/firefox/:/home/jp/bin/thunderbird:/home/jp/bin/mupen64-0.5:/home/jp/bin/RealPlayer/:/op
t/Mercury/Amira4.1:/usr/local/share/OpenSceneGraph/bin/:/usr/autodesk/maya8.5/bin/:$PATH
export ODEHOME=/home/jp/local
export OSG_EYE_SEPARATION=0.026
###########This next can separated if using OSG 2.0
export VJ_BASE_DIR=/home/curso/vrjuggler-2.0.4-1-src/build.linux/instlinks
export JCCL_DEFINITION_PATH=${VJ_BASE_DIR}/share/vrjuggler/data/definitions
export LD_LIBRARY_PATH=/home/jp/local/lib/:/home/jp/FISICA/osgNewton/lib:/home/jp/FISICA/osgODE:/usr/local/lib/osgPlugins:/usr/local/lib:$LD_LIBRARY_PATH
export PKG_CONFIG_PATH=/home/jp/local/lib/pkgconfig:/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
export OSGHOME=/usr/local/share/OpenSceneGraph
export OSGDATA=/usr/local/OSG_OP_OT-1.2/OpenSceneGraph/OpenSceneGraph-Data
export OSGDEP=/usr/local/share
export OSGFILEPATH=${OSGDATA}:${OSGDATA}/Images
###########Vrjuggler and Navio Specific
export LD_LIBRARY_PATH=/home/curso/vrjuggler-2.0.4-1-src/build.linux/instlinks/lib:$LD_LIBRARY_PATH
export PATH=/home/curso/vrjuggler-2.0.4-1-src/build.linux/instlinks/bin:$PATH
export PATH=/home/curso/dgsca-vrnav-osg-gui/bin/Linux32:$PATH


All of the above could be saved to a file called ENVIRON (this being the most convenient file name) in the home directory, so that everytime we open a terminal we simply execute this line:

$source ENVIRON


Note: Do not attempt to compile, run or install anything that is within your local folder without first calling up this file. It is the bare minimum set of environment variables for which to get anything to work at all.

(this because we will get a number of errors if pkgconfig does not parse our local folder when running autotools)

Note: Alternatively you may want to put this in your .bashrc file, but it's better to have control over what environment variables get loaded.

There are other environment variables which we will define as we go.

In all cases, when we configure, we add the --prefix parameter pointing to /home/%USER/local (again replace %USER with your real user name).
The order of compilation usually goes like this:

  1. Try compiling Ogre: only Ogre can tell us what we lack at this point.
  2. We lack zziplib, try compiling zziplib (some editing required in zziplib.pc).
  3. We lack FreeImage, try compiling FreeImage (a lot of editing required in make.gnu to get it to install locally).
  4. We try to reconfigure Ogre, with LDFLAGS="-L/home/%USER/local/lib/".
  5. We want to compile examples so we try to compile OIS and CEGUI and install locally.
  6. CEGUI requires PCRE, so we compile and install that locally as well.
  7. CEGUI can't find Freeimage, so we compile with CPPFLAGS="-I/home/%USER/local/include" and maybe LDFLAGS="-L/home/%USER/local/lib/".
  8. We uncompress CG (if we have nvidia) to our local folder
  9. Optionally we compile and locally install OpenEXR so we have an openexr plugin


Our last line, in the ogrenew folder, should look somewhat like this:

$ ./configure LDFLAGS=-L/home/jp/local/lib/ --enable-openexr --with-platform=GLX --prefix=/home/jp/local/


If you know you're way around compiling locally than you most likely understood what I did (with the few hints I provided). If not, then continue with the next session.

Step by step

Ogre trial compile

Download the Ogre source into a local folder (I usually put it in ~/Downloads) with wget or firefox (whatever fancies you) and untar into my local folder's src folder.

$mkdir ~/local/src
$cd ~/local/src
$tar -xjvf ~/Downloads/ogre-linux_osx-v1-4-5.tar.bz2

We change into ogrenew, bootstrap and run ./configure

$cd ogrenew
$./bootstrap
$./configure --prefix="/home/%USER/local"

At this point we haven't specified any additional parameters (save prefix)so that we can find out what's missing. Usually it will zonk out when it looks for zziplib.

zziplib

Download zziplib (of which I'm using version 0.13.49). Also untar this file into our local src directory:

$cd ~/local/src
$tar -xjvf ~/Downloads/zziplib-0.13.49.tar.bz2
$cd zziplib-0.13.49


Configure for local installation, make and install

$./configure --prefix="/home/%USER/local"
$make
$make test
$make install
$make clean


Hopefully there will be no errors up to this point. In this particular version, the zziplib.pc file in ~/local/lib/pkgconfig has a dubious variable that will confuse Ogre's configuration. Open this file in a text editor of your choice and either delete, comment out, or leave the variable $datadir blank, like this:

# generated by configure / remove this line to disable regeneration
prefix=/home/jp/local/
exec_prefix=${prefix}
bindir=${exec_prefix}/bin
libdir=${exec_prefix}/lib
datadir=
sysconfdir=${prefix}/etc
includedir=${prefix}/include
package=zziplib
suffix=

Name: zziplib
Description: ZZipLib - libZ-based ZIP-access Library
Version: 0.13.49
Requires: zzip-zlib-config
Libs: -L${libdir}  -lzzip
Cflags: -I${includedir}


Now we have one more library to install to make a minimum build of Ogre.

Building FreeImage locally

Of all the libraries, this one by far represents the hardest to configure (mostly because the author wants you to install it among the stable /usr/lib libraries).
I use version 3.93 (or .393) and so it might not be the latest source (and perhaps the author has decided to go the autotools way by then). In the meantime, we have to edit one of the make files.
First off, uncompress the files into your local source directory

$cd ~/local/src
$unzip ~/Downloads/FreeImage393.zip
$cd FreeImage393


Edit Makefile.gnu and replace these lines:

INCDIR = /usr/include
INSTALLDIR = /usr/lib


With your local directories:

INCDIR = /home/%USER/local/include
INSTALLDIR = /home/%USER/local/lib


Go down to the install: line in the same file and change these lines...

install:
        install -m 644 -o root -g root $(HEADER) $(INCDIR)
        install -m 644 -o root -g root $(STATICLIB) $(INSTALLDIR)
        install -m 755 -o root -g root $(SHAREDLIB) $(INSTALLDIR)
        ln -sf $(SHAREDLIB) $(INSTALLDIR)/$(VERLIBNAME)
        ln -sf $(VERLIBNAME) $(INSTALLDIR)/$(LIBNAME)   
        ldconfig


...so that they remain your files and not root's. Also we should eliminate the last line (we don't have enough privileges to update ld.so.cache):

install:
        install -m 644 $(HEADER) $(INCDIR)
        install -m 644 $(STATICLIB) $(INSTALLDIR)
        install -m 755 $(SHAREDLIB) $(INSTALLDIR)
        ln -sf $(SHAREDLIB) $(INSTALLDIR)/$(VERLIBNAME)
        ln -sf $(VERLIBNAME) $(INSTALLDIR)/$(LIBNAME)   
#        ldconfig -r /home/jp/local/ -n /home/jp/local/lib


Note: I made my own ld.so.cache in the last line, but it's relatively useless.

Now we can build, install and clean

$make
$make install
$make clean

CG

The CG library is necesarry for GLSL scripts in Ogre (I am unaware of how ATI handles this) and as of this writing the latest version of the CG toolkit is 1.5.
This library is relatively simple to install since we mostly have to move the contents of the uncompressed tarball to our local directory (and then some).
Like FreeImage, Nvidia believes itself worthy enough to install in the stable /usr/lib libraries (who am I to argue, these people descend from Silicon Graphics). For the most part, they are right, but again we're not sudoers.

$cd ~/local/src
$tar -xzvf ~/Downloads/Cg-1.5_Sep2007_x86.tar.gz
$mv ./usr/* ~/local


Note: I don't exactly know whether glext.h in ~/local/local/include has any relevance to Ogre builds, but keep it in mind for any future uses of CG.

Minimum build check

Just so to make sure that we have the minimum requirements to build Ogre, we run this line with LDFLAGS environment variables set to our local libraries (this is how Ogre or any distribution checks for FreeImage, via the -l g++ argument)

$cd ~/local/src/ogrenew
$$ ./configure LDFLAGS=-L/home/jp/local/lib/ --with-platform=GLX --prefix=/home/%USER/local/


If any error relating to zziplib or FreeImage come up, it's because you haven't read this page thoroughly (nor the hints I mentioned before, you sodding little...). No I mean it, you must peruse what you might have missed on building these two libraries, as well as double check your ENVIRON files (maybe you made a mistake in the paths or something), especially PKG_CONFIG_PATH.
Anything else could be related to CG (check ~/local for the usr folder).

CEGUI

CEGUI has many dependencies (or had many dependencies when invoked with DevIL) although you might only need PCRE (which is described after this step). We want to enable FreeImage use so we have to define two variables when we configure the source, most importantly CPPFLAGS.

$cd ~/local/src/
$tar -xzvf ../../Downloads/CEGUI-0.5.0b.tar.gz
$cd CEGUI-0.5.0
$./configure --prefix=/home/jp/local/ --enable-freeimage LDFLAGS=-L/home/jp/local/lib/ CPPFLAGS=-I/home/jp/local/include
$make
$make install
$make clean

PCRE (In case CEGUI failed to build)



PCRE is also an autotools package so it's mostly the usual stuff here

$cd ~/local/src/
$tar -xzvf ../../Downloads/pcre-7.4.tar.bz2
$cd pcre-7.4
$./configure --prefix=/home/%USER/local
$make
$make install
$make clean

OIS



This is a straightforward program, we just have to bootstrap before configuring

$cd ~/local/src/
$tar -xzvf ../../Downloads/ois-1.0.tar.gz
$cd ois
$./bootstrap
$./configure --prefix=/home/%USER/local
$make
$make install
$make clean

Almost there



With the above configured and installed, we have but to configure Ogre and compile it, but certain detriments might come along. You might come across an error in the Demo Compilations, one that is particular to the lack of gtkmm or glademm. Either library has a long list of dependencies, so you have 3 choices.

  1. Ask the administrator to kindly install those for you (rpm, the debian pacakges, or even portage will suffice)
  2. Compile these libraries with their dependencies as close to the one installed as you can (you'll see why if you don't)
  3. Compile the latest libraries, taking into account that you will have to compile, in a local manner, everything down to Glib2.0 (that means GTK+, Cairo, Pango, ATK, etc.)


Debian and Ubuntu users don't have much a problem since anyone can download the source packages that make up the packages, so they can go with choice number two easily. I suppose Gentoo could also go with choice number two but it's better if you get the administrator to install these libraries. Suse and Redhat (and all rpm users) mostly have either the first option or can go (as I unfortunately had to) with choice number 3. This last choice is not hard (with the configuration we have) it's just tiring. Fortunately, all these packages use autotools and to not require funny environment variables or editing of their make files.

If you choose to do choice #3, you might as well recompile a local version of FreeType2 (and all it's dependencies). The newer the better (and I'm having a lot of problems with CentOS/Redhat's version of the bloody library).

As a last token, I like to add OpenEXR which is a free library Industrial Light & Magic provide to handle HDR images. If you don't want this plugin, just skip ahead and remove --enable-openexr from the configure options.

OpenEXR

$cd ~/local/src/
$tar -xzvf ../../Downloads/ilmbase-1.0.1.tar.g
$cd ilmbase-1.0.1
$./configure --prefix=/home/%USER/local
$make
$make install
$make clean
$cd ~/local/src/
$tar -xzvf ../../Downloads/openexr-1.6.1.tar.gz
$cd openexr-1.6.1/
$./configure --prefix=/home/%USER/local
$make
$make install
$make clean

Final Build

Our last build with Ogre. This could be the longest build of your life so far (anyone who has ever built Gnome with Avahi on Gentoo can laugh this off). We go to our ogrenew folder and type.

$./configure LDFLAGS=-L/home/jp/local/lib/ --enable-openexr --with-platform=GLX --prefix=/home/jp/local/
$make
$make install


Do not make clean yet, as this will eliminate the Samples (and they are necesarry to figure out what works and what doesn't. Running the demos involves just one last change.

Open up plugins.cfg and change the line that looks like:

# Define plugin folder
PluginFolder=/usr/local/lib/OGRE


Change it so that it'll point to your local folders

# Defines plugins to load

# Define plugin folder
PluginFolder=/home/jp/local/lib/OGRE

# Define D3D rendering implementation plugin
Plugin=RenderSystem_GL.so
Plugin=Plugin_ParticleFX.so
Plugin=Plugin_BSPSceneManager.so
Plugin=Plugin_OctreeSceneManager.so
Plugin=Plugin_CgProgramManager.so


And that pretty much wraps up the local installation of Ogre. The next part is still a work in progress

Configuring DMX and Chromium

DMX and Chromium was configured by the system administrator where I work so I'd have to ask him how he has it working. The thing about this setup is that it's the only away I've been able to run Ogre on a Cluster (DMX can't handle ogre by itself, but Chromium can).

In your user account you should have a file called .crconfig with the following line:

* /usr/local/src/cr1.9/mothership/configs/autodmx.conf %m %p

(or wherever you have the configure scripts installed)

And you also have to load the corresponding Chromium libraries into memory via the environment variable LD_LIBRARY_PATH (I can't remember where they are located).

Launch DMX and set your environment variales to run Ogre and Chromium. You should launch an example in the ogrenew/Samples/Common/bin directory to generate a ogre.cfg file. Modify the file to suit your maximum resolution (it will not autodetect an expanded desktop). In my case it was 3840x1024 and in fullscreen.

Chromium alone

In order to run Ogre Examples (or anything else for that matter) using just Chromium, you will have to edit line 74 of ExampleApplication.h in ~/local/src/ogrenew/Samples/Common/include:

#if OGRE_PLATFORM == OGRE_PLATFORM_APPLE
        mResourcePath = macBundlePath() + "/Contents/Resources/";
#else
        mResourcePath = "";
#endif


You have to add the Path to your executable file and resources (in this case the examples directory) to mResourcePath, like this:

#if OGRE_PLATFORM == OGRE_PLATFORM_APPLE
        mResourcePath = macBundlePath() + "/Contents/Resources/";
#else
        mResourcePath = "/home/jp/local/src/ogrenew/Samples/Common/bin/";
#endif


After this modification, you have to recompile the examples:

$cd ~/local/src/ogrenew/Samples
$make clean
$make
$make install


cd to the Samples/Common/bin directory and run the following:

$ssh-agent $SHELL
$export PATH=${PWD}:$PATH
$python /share/apps/src/cr-1.9/mothership/configs/three_tiles.conf RenderToTexture


The path to your executable must be within your $PATH environment variable (or else Chromium doesn't know what to do) and I'm guessing is that ssh-agent passes along all your environment variables to the other three nodes.

three_tiles.conf is a premade python script to distribute all opengl calls to three graphics nodes (you will see an empty black window on the master node with the dimensions you specified in ogre.cfg), it precedes the call to your executable. This mode of graphics clustering is much faster, more powerful and better for debugging purposes but also more tedious.

Final Outcome

The results are, so far, hopeful, but Chromium still has a number of defects still (like it still only handles up to OpenGL 1.5). DMX can't handle OpenGL calls very well (and even then it sure doesn't handle Ogre). Between the two, Ogre's Geometry gets by (sometimes not so well) and the textures are far from correct. Hopefully by upgrading to GCC 4 and/or using DeVIL I can get rid of these problems.

Image