History: Visual Unit Testing Framework

• «
• »

            __NOTE: As of right now, this documentation is preliminary, incomplete, and subject to change.__
{maketoc}
!__Abstract__
This is the documentation for OGRE's visual unit testing framework. The framework allows you to perform image-based comparisons of test scenes between builds.

!__Introduction__
OGRE already uses [http://sourceforge.net/projects/cppunit/|CppUnit] for a selection of unit tests that cover the basics: Vectors, String functions, etc; however, this handful of existing tests is far from comprehensive. Moreover, given the high degree of interdependency and the dependence on graphics API's and the like, traditional unit testing really wouldn't be possible without a very complex testing setup involving mock rendersystems and a huge amount of testing code.

Even if traditional unit testing were doable without significant commitment, it still wouldn't be especially helpful in the context of a rendering engine (you really can't effectively decide "Is the rendered image correct?" with just assertions and so forth).

Given that the output of a rendering engine is an image, why not test it using just that? This framework aims to make testing possible by creating test scenes that can be screen-captured and compared between builds. This allows for features to be tested very simply (just implement the feature in a simple test scene, and the framework does the rest; there's no need for assertions or elaborate test cases).

Additionally, a provided image comparison tool allows for failed tests to be readily debugged in a very natural (visual) manner.
!__Usage Details__
[[any compilation or similar organization notes should go in here somewhere eventually]

!!__Running Tests__
#Run the TestBrowser executable.
#Make any necessary adjustments to basic configuration options (resolution, etc).
#Select an image set to compare against, or none if you wish to simply create a set (which can later be compared with another using the comparison tool).
#Press the 'Run' button.
#Wait while the browser runs through the tests. This is completely automated (if you wish to manually inspect an individual test, you can do so from the main test browser menu).
#After the tests are complete, the image comparison tool will be launched and display an overview of the compared sets. If you didn't choose a set to compare with, you will be returned to the main test browser menu.
!!__The Output__
Whenever a set is created, the test images themselves, along with a small config file containing data about the set (resolution, date/time, name, etc), are created.

When two sets are compared using the image comparison tool, the tool itself will give an immediate overview and allow you to inspect individual images. Additionally, an html document containing the summary of the comparison is generated. This file will give a handful of basic stats (number of tests that passed/failed, overall percent differences, etc), and side-by-side images with some specific stats for each test.

!!__Creating New Tests__
The testing framework is built on top of the existing sample framework, so it is very similar to creating a sample.

Some things to note:
* You will need to specify when you want test screenshot(s) to be taken.
* Tests must be deterministic, so use the delta (time since last frame) time passed to the frameStarted/frameEnded functions for any timing needs.
* Keep tests simple, the idea is to isolate and test a single feature as completely as possible.
!__Tests__

{FANCYTABLE(head="Current Tests")}
__Name:__ | __Description:__ | __Image:__
Texture Blending | Tests basic, fixed function texture blending. | No image yet.
Camera Set Direction | Tests setting of camera direction. | No image yet.
Stencil Shadows | Tests modulative stencil shadows | No image yet.
Skeletal Animation | Tests basic skeletal animation | No image yet.
{FANCYTABLE}

!__Known Issues__
* None! (so far...)
!__Future Improvements__
* More Tests!
* A more robust image comparison algorithm (e.g. something along the lines of [http://pdiff.sourceforge.net/|PerceptualDiff])