NOTE: As of right now, this documentation is preliminary, incomplete, and subject to change.

Abstract

This is the documentation for OGRE's visual unit testing framework. The framework allows you to perform image-based comparisons of test scenes between builds.

Introduction

OGRE already uses CppUnit for a selection of unit tests that cover the basics: Vectors, String functions, etc; however, this handful of existing tests is far from comprehensive. Moreover, given the high degree of interdependency and the dependence on graphics API's and the like, traditional unit testing really wouldn't be possible without a very complex testing setup involving mock rendersystems and a huge amount of testing code.

Even if traditional unit testing were doable without significant commitment, it still wouldn't be especially helpful in the context of a rendering engine (you really can't effectively decide "Is the rendered image correct?" with just assertions and so forth).

Given that the output of a rendering engine is an image, why not test it using just that? This framework aims to make testing possible by creating test scenes that can be screen-captured and compared between builds. This allows for features to be tested very simply (just implement the feature in a simple test scene, and the framework does the rest; there's no need for assertions or elaborate test cases).

Usage Details

Build Details

Simply build with the OGRE_BUILD_TESTS option enabled in CMake. Also, note that test plugins are fully compatible with the sample browser.

Running Tests

Due to differences in drivers and so forth, it works best to generate a reference image set for each machine you will be testing with, this greatly reduces the chance of false positives due to driver issues.

1. Run the TestContext executable (see below for a selection of command line options).
2. It will automatically run through the tests and compare the generated images against the reference image set (if you have generated one). It will exit when it is complete.
3. The results will output to your home/My Documents directory (see below for output location details).

Command Line Options

Note that this is using Ogre's built-in findCommandLineOpts which does not allow for combining of options (i.e. foo -abc bar would have to be written as foo -a -b -c bar)
-r
Generate a reference set.

-m "[comment]"
Add an optional comment to be associated with the generated image set.

-ts "[test set name]"
Select the test set to use (default is 'VTests').

-c "[image set name to compare against]"
Select which image set you want to compare this run with (default is 'Reference').

-n "[name]"
Specify a name for this set (omitting this, or choosing 'AUTO' will result in an automatically generated name).

-h -help or --help
Usage details.

--no-html
Suppress html output.

The Output

Whenever a set is created, the test images themselves, along with a small config file containing data about the set (resolution, date/time, name, etc), are created in a new directory (see below for details on directory structure).

The primary output is an HTML document containing an overview of the test, and side-by-side images of the reference image set and the newly generated images. A small linked javascript file allows for some basic diffing to be done within a web browser (requires HTML5/Canvas).

Here is a sample of the html output.

Where to Find the Output

Output is generated in the same directory as the logs and cfg's for the sample browser. This is generally in your My Documents or home directory (or your OS's equivalent). There should be an Ogre directory, with a subdirectory for the Ogre version. From there the structure looks like:

• VisualTests
  • [Test set name]
    • [Rendersystem]
      • out.html
      • [Reference]
        • info.cfg
        • Reference screenshots (.png's)...
      • [Test set name]_[date]
        • Info.cfg
        • Screenshots (.png's)...

Image Comparison

The images are compared to reference images using a selection of common metrics. A failed test will report values for the following metrics in the HTML output:

The most basic is just the absolute difference; how many pixels differ between two images.

Next is the Mean Squared Error (MSE), which, as the name suggests produces the average squared error (difference between the images). Lower is better.

Next is the Peak Signal-to-Noise Ratio (PSNR), which measures the ratio between the maximum signal (in this case, full color values in each channel), and corrupting noise (the differences in the images). Higher values of this metric are better.

Last, is the Structural Similarity (SSIM) index, which is a more recent development (see this 2004 paper for in-depth details), and aims to provide a metric better related to human preception (images with identical MSE may actually be of very different quality levels). It gives a value in the range of -1 to 1, (with 1 being identical).

Creating New Tests

The testing framework is built on top of the existing sample framework, so it is very similar to creating a sample. Tests are created in plugins that the TestContext is able to load dynamically.

General

Create a class derived from VisualTest, override whichever functions you need (the same FrameListener-style functions used in Samples apply here), and add it to a test plugin.

Some Things to Note:

• You will need to specify when you want test screenshot(s) to be taken, with addScreenshotFrame (timing is done by frame to prevent floating point issues).
• Tests must be deterministic, so use the delta (time since last frame) time passed to the frameStarted/frameEnded functions for any timing needs.
• Keep tests simple, the idea is to isolate and test a single feature as completely as possible.

Defining Test Sets

Test sets (a grouping of tests that will be generated and compared together) are defined as a collection of test plugins in the 'tests.cfg' file. Plugins can belong to more than one test set.

An example configuration of test.cfg is below

# where the test plugins are located
TestFolder=[Ogre lib dir]

# A set of all visual tests
[VTests]
TestPlugin=PlayPenTests
TestPlugin=VTests

# Only the playpen tests
[Playpen]
TestPlugin=PlayPenTests

Tests

The following are the initial tests being used with the system:

Known Issues

• None! (so far...)

Future Improvements

• More Tests!
• A more robust image comparison algorithm (e.g. something along the lines of PerceptualDiff)

Additional Links

• Summer of Code forum thread
• Unity3d's test framework