Development Info

A lot of this info is out of date, since the plan and schedule changed a good deal, so see the documentation page for more concrete info.

Here is the fork on Bitbucket.

Project Proposal

I would like to add a visual testing framework to OGRE.

Traditional unit testing doesn't help much in a context like OGRE, given the sheer size and heavy dependence on graphics API's. The best way of doing tests without needing significant modifications, mock rendersystems or something to that effect, appears to be a visual solution, wherein test screenshots are compared between builds.

I intend to use the existing sample framework as a a basis, since tests aren't too far off from samples and it just seems like a logical choice. There would be two primary components to the project:

The TestContext

  • Extended from SampleContext, it is to tests what the SampleBrowser is to samples.
  • Initially just runs through the tests as quickly as possible taking screenshots and then exits.
  • However, in week 5 I'll add the ability to manually inspect individual tests, select various options, launch the image comparison tool, etc.

The Image Comparison Tool

  • Built as a Sample for use with the existing SampleBrowser (and later, the TestBrowser itself).
  • Initially just allows you to manually flip through sets of images and compare them.
  • Later in the schedule some more advanced comparison tools (various methods of overlaying and diffing, some automated comparison that can give a quick summary of a whole image set, etc).

The automated image comparison will most likely be a fairly simple custom solution, although "perceptual image comparison" (see this Siggraph '04 slideshow) might be a possible later addition which could reduce the potential for false positives.

If the currently planned comparison solution proves insufficient, there is a GPL implementation of the perceptual image comparison technique (PerceptualDiff) which could be integrated fairly painlessly (GPL isn't ideal, but the testing framework is separate from OGRE and so the licensing could be switched if need be).

The primary focus is on making a useable, well-documented framework, so I won't necessarily be making a lot of actual tests (aside from a few here and there to test the workflow).

The creation of actual tests will be very similar to creating a regular Sample with the existing Sample Framework, with the addition of specifying when the test screenshot(s) should be taken, and using a fixed timestep to ensure deterministic behavior.

Due to hardware/driver differences this is not guaranteed to work across different machines, for the time being the goal is just to make it consistent with images generated on the same machine.

Unity successfully uses a similar system for their testing, see this blog post for some details about their implementation.

How will this project benefit OGRE users?


While it doesn't add any flashy features that would be visible in an end product, it would certainly help in the development and testing of OGRE which is definitely good for users. It would also make testing a lot easier for a user who needs to modify OGRE. Additionally, the image comparison tool would allow users to compare test shots from their own projects (for instance, it could be very helpful in shader debugging or something to that effect).

Is this project within the core scope of OGRE?


Definitely, as a testing framework it's obviously meant to be used directly with OGRE.

Is it realistic to achieve over a summer?


I believe so.

The main area of concern is probably the image comparison tool, since tools inevitably entail a lot of polish and documentation; but given the excellent sample framework and SDK trays stuff already in place, I think it should be feasible. Even if some of the nicer features can't be implemented in time, I can always fall back on a simpler version (I've purposely planned to make a very basic, but functional and deliverable version early on, and then add additional features towards the end).

Schedule


I will have school and exams until June 9th, so I'm aiming to go a little slower for the first weeks and catch up in the following weeks. Other than that, however, I have no vacation plans or other obligations.

April 25th – May 23rd

  • Get to know mentor.
  • Get more familiar with the sample framework, Mercurial, CMake, etc.

Week 1 (May 23rd) –

  • Start coding.
  • Create “VisualTest” class that will form the basis for tests. Derived from the the existing SampleFramework's Sample class, with the addition of options for screenshots, and taking various measures to ensure animations/particles/etc are deterministic (I've already been able to do this by modifying the existing playpen sample base class).
  • Convert the existing playpen samples to use this (the 10 or so that are already set up for the sample browser, and should be trivial to convert).

Week 2 (May 30th) –

  • Write a simple TestContext that automatically runs through a set of tests (no extra menu/options stuff yet, just runs straight through the tests taking the test images, and exits).

Week 3 (June 6th) –

  • My final exams are this week, so I will be mostly unavailable, I'll probably end up tinkering and testing a little bit anyways, but nothing definite.

Week 4 (June 13th) –

  • Create a very basic image comparison tool. At this point it will just allow you to flip through images and manually compare them. Created as a sample plugin, so it can be used with the existing browser, and eventually be run straight from the TestContext.
  • Create a test or two (or perhaps convert a couple old playpen tests), use this to start working out some of the documentation, and work out any kinks in the workflow.

Week 5 (June 20th) –

  • Add functionality to the TestContext (allow you to set various options before running the tests, manually launch individual tests, open the image comparison tool, lots of UI stuff).

Week 6 (June 27th) –

  • Polish up the TestContext's UI and such as necessary.
  • Write image comparison code that can be run automatically (outputs percent differences between images, perhaps some other stats).

Week 7 (July 4th) –

  • Finish up any remaining work on the comparison code.
  • Integrate this into the comparison tool, (display overall stats for the whole image set, allow the user to easily jump to the images that have the greatest deviance from the reference, etc.)

Week 8 (July 11th) –

  • Mid Term evaluation
  • Add additional functionality to the image comparison tool (various modes of overlaying and comparing image sets (see github's new image comparison tool for some potential functionality)).

Week 9 (July 18th) –

  • Add any necessary final polish and documentation for the comparison tool.
  • Begin converting some old playpen tests over.

Week 10 (July 25th) –

  • Convert more playpen tests over to the new system (certainly not all of them, but enough to adequately test the framework and work out any potential issues).

Week 11 (August 1st)

  • Continue converting playpen tests over.
  • The main idea here is testing out the workflow, making sure the framework can handle various features and edge cases, etc; much of the actual work over these weeks will be making fixes and additions to the framework itself, as they become necessary.

Week 12 (August 7th) –

  • Overflow for anything that needs more work, more documentation, cleanup, etc.
  • If there's extra time, look into integrating it with OGRE's existing unit tests.

Week 13 (August 14th) –

  • August 15th is the suggested pencils down date.
  • Final documentation and cleanup.

End August 22nd

Future Extensions


Since the idea here is to create a solid framework, rather than a definite set of tests, the obvious next step would be to write a set of useful, general tests using it.

Some sort of continuous integration server and perhaps a repository of reference images could also be useful next steps.

Why You're The Person For This Project


I'm a 19-year-old student in the US, I'm studying Computer Science at the University of Washington in Seattle.

I've been using C++ for over 4 years now, and OGRE for almost as long. I've been pretty much constantly working on some project or another using OGRE over the years. You can see some of my past projects here (all the projects listed there use OGRE).

I have some code up on my github, much of it is really gross 48-hour Ludum Dare code, but “OryxEngine”, a game engine I've been working with for a number of months is pretty clean.

I'm not an expert by any means, my programming experience is mostly self-taught at this point, but I simply love programming and 3d development. I love a good challenge and I'm eager to learn.

I unfortunately haven't been very involved with the OGRE community lately, but I'll try and change that. In terms of other community involvements, I'm pretty active on the TigSource forums, and I actively participate in Ludum Dare (using OGRE, as an added bonus) every few months.

Why OGRE?


I've been using OGRE for years; it's easily been the single most influential thing on my coding style and learning process over the past years. I think it'd be incredible to be able to give back and try to return the favor.

<HR>
Creative Commons Copyright -- Some rights reserved.


THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED.

BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.

1. Definitions

  • "Collective Work" means a work, such as a periodical issue, anthology or encyclopedia, in which the Work in its entirety in unmodified form, along with a number of other contributions, constituting separate and independent works in themselves, are assembled into a collective whole. A work that constitutes a Collective Work will not be considered a Derivative Work (as defined below) for the purposes of this License.
  • "Derivative Work" means a work based upon the Work or upon the Work and other pre-existing works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which the Work may be recast, transformed, or adapted, except that a work that constitutes a Collective Work will not be considered a Derivative Work for the purpose of this License. For the avoidance of doubt, where the Work is a musical composition or sound recording, the synchronization of the Work in timed-relation with a moving image ("synching") will be considered a Derivative Work for the purpose of this License.
  • "Licensor" means the individual or entity that offers the Work under the terms of this License.
  • "Original Author" means the individual or entity who created the Work.
  • "Work" means the copyrightable work of authorship offered under the terms of this License.
  • "You" means an individual or entity exercising rights under this License who has not previously violated the terms of this License with respect to the Work, or who has received express permission from the Licensor to exercise rights under this License despite a previous violation.
  • "License Elements" means the following high-level license attributes as selected by Licensor and indicated in the title of this License: Attribution, ShareAlike.

2. Fair Use Rights

Nothing in this license is intended to reduce, limit, or restrict any rights arising from fair use, first sale or other limitations on the exclusive rights of the copyright owner under copyright law or other applicable laws.

3. License Grant

Subject to the terms and conditions of this License, Licensor hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright) license to exercise the rights in the Work as stated below:

  • to reproduce the Work, to incorporate the Work into one or more Collective Works, and to reproduce the Work as incorporated in the Collective Works;
  • to create and reproduce Derivative Works;
  • to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission the Work including as incorporated in Collective Works;
  • to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission Derivative Works.
  • For the avoidance of doubt, where the work is a musical composition:
    • Performance Royalties Under Blanket Licenses. Licensor waives the exclusive right to collect, whether individually or via a performance rights society (e.g. ASCAP, BMI, SESAC), royalties for the public performance or public digital performance (e.g. webcast) of the Work.
    • Mechanical Rights and Statutory Royalties. Licensor waives the exclusive right to collect, whether individually or via a music rights society or designated agent (e.g. Harry Fox Agency), royalties for any phonorecord You create from the Work ("cover version") and distribute, subject to the compulsory license created by 17 USC Section 115 of the US Copyright Act (or the equivalent in other jurisdictions).
    • Webcasting Rights and Statutory Royalties. For the avoidance of doubt, where the Work is a sound recording, Licensor waives the exclusive right to collect, whether individually or via a performance-rights society (e.g. SoundExchange), royalties for the public digital performance (e.g. webcast) of the Work, subject to the compulsory license created by 17 USC Section 114 of the US Copyright Act (or the equivalent in other jurisdictions).


The above rights may be exercised in all media and formats whether now known or hereafter devised. The above rights include the right to make such modifications as are technically necessary to exercise the rights in other media and formats. All rights not expressly granted by Licensor are hereby reserved.

4. Restrictions

The license granted in Section 3 above is expressly made subject to and limited by the following restrictions:

  • You may distribute, publicly display, publicly perform, or publicly digitally perform the Work only under the terms of this License, and You must include a copy of, or the Uniform Resource Identifier for, this License with every copy or phonorecord of the Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Work that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder. You may not sublicense the Work. You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Work itself to be made subject to the terms of this License. If You create a Collective Work, upon notice from any Licensor You must, to the extent practicable, remove from the Collective Work any credit as required by clause 4(c), as requested. If You create a Derivative Work, upon notice from any Licensor You must, to the extent practicable, remove from the Derivative Work any credit as required by clause 4(c), as requested.
  • You may distribute, publicly display, publicly perform, or publicly digitally perform a Derivative Work only under the terms of this License, a later version of this License with the same License Elements as this License, or a Creative Commons iCommons license that contains the same License Elements as this License (e.g. Attribution-ShareAlike 2.5 Japan). You must include a copy of, or the Uniform Resource Identifier for, this License or other license specified in the previous sentence with every copy or phonorecord of each Derivative Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Derivative Works that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder, and You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Derivative Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Derivative Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Derivative Work itself to be made subject to the terms of this License.
  • If you distribute, publicly display, publicly perform, or publicly digitally perform the Work or any Derivative Works or Collective Works, You must keep intact all copyright notices for the Work and provide, reasonable to the medium or means You are utilizing: (i) the name of the Original Author (or pseudonym, if applicable) if supplied, and/or (ii) if the Original Author and/or Licensor designate another party or parties (e.g. a sponsor institute, publishing entity, journal) for attribution in Licensor's copyright notice, terms of service or by other reasonable means, the name of such party or parties; the title of the Work if supplied; to the extent reasonably practicable, the Uniform Resource Identifier, if any, that Licensor specifies to be associated with the Work, unless such URI does not refer to the copyright notice or licensing information for the Work; and in the case of a Derivative Work, a credit identifying the use of the Work in the Derivative Work (e.g., "French translation of the Work by Original Author," or "Screenplay based on original Work by Original Author"). Such credit may be implemented in any reasonable manner; provided, however, that in the case of a Derivative Work or Collective Work, at a minimum such credit will appear where any other comparable authorship credit appears and in a manner at least as prominent as such other comparable authorship credit.

5. Representations, Warranties and Disclaimer

UNLESS OTHERWISE AGREED TO BY THE PARTIES IN WRITING, LICENSOR OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE MATERIALS, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU.

6. Limitation on Liability.

EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

7. Termination

  • This License and the rights granted hereunder will terminate automatically upon any breach by You of the terms of this License. Individuals or entities who have received Derivative Works or Collective Works from You under this License, however, will not have their licenses terminated provided such individuals or entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will survive any termination of this License.
  • Subject to the above terms and conditions, the license granted here is perpetual (for the duration of the applicable copyright in the Work). Notwithstanding the above, Licensor reserves the right to release the Work under different license terms or to stop distributing the Work at any time; provided, however that any such election will not serve to withdraw this License (or any other license that has been, or is required to be, granted under the terms of this License), and this License will continue in full force and effect unless terminated as stated above.

8. Miscellaneous

  • Each time You distribute or publicly digitally perform the Work or a Collective Work, the Licensor offers to the recipient a license to the Work on the same terms and conditions as the license granted to You under this License.
  • Each time You distribute or publicly digitally perform a Derivative Work, Licensor offers to the recipient a license to the original Work on the same terms and conditions as the license granted to You under this License.
  • If any provision of this License is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this License, and without further action by the parties to this agreement, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.
  • No term or provision of this License shall be deemed waived and no breach consented to unless such waiver or consent shall be in writing and signed by the party to be charged with such waiver or consent.
  • This License constitutes the entire agreement between the parties with respect to the Work licensed here. There are no understandings, agreements or representations with respect to the Work not specified here. Licensor shall not be bound by any additional provisions that may appear in any communication from You. This License may not be modified without the mutual written agreement of the Licensor and You.