Project Proposal

Motivation

Current terrain component introduced in Ogre3D 1.7 uses chunked level of detail(LOD) together with geomorphing to achieve continuous LOD for terrain(heightmap) rendering. This terrain component performs very well on smaller sets of height data which is it's purpose however problems with limited size of memory come with larger sets of data. Paging of data is common solution for memory issues of this kind and paging component already exists in Ogre3D however it's performance is very poor when used together with terrain component and is not ready to be used in a real project. The current paging strategy which is used for paging terrain tiles in/out is also not optimal in some cases.

Project

There are basically two problems using terrain paging in Ogre3D. Poor performance of terrain paging and simple paging strategy used to select tiles for swapping in/out.

Poor terrain paging performance causes visible tearing/delay when it comes to load another page tile for rendering. This project will address this issue by modifying the way terrain component handles data. That involves:

  • separate load of specific terrain data belonging to specific LOD instead of only possibility to load whole terrain data at once so more distant terrain tiles could be loaded only in low LOD
  • load of terrain data in background or at users own decision
  • notification system for various terrain component events like progress of data loading/preparation so user knows what about the progress


In some cases there is a problem with terrain tiles missing in the view or problem with actually rendering more tiles then needed. This issue is caused by default simple paging strategy used(Grid2DPageStrategy) which loads terrain tiles based on radius around camera. The project will bring new paging strategy which will try to be more accurate using actual terrain shape visibility from camera.

There will be introduced new demo to demonstrate above functionality:

  • import and prepare terrain data
  • load/unload specific terrain LOD
  • use of/react on terrain load events
  • switch between different LOD strategies

Proposal Timeline

Bonding period (27 Apr – 24 May):

  • Play and experiment with Ogre3D terrain and paging components and with its demo and test applications to further improve knowledge about it. Examine threading Ogre3D threading model and other needed core functionality as well.
  • Actively interact with my mentor and Ogre3D community to discuss final details of my intended implementation. After this I will be absolutely sure about every bit of my future work.

Official coding time starts (25 May – 12 Jul 2011):

First work to do is to implement separate LOD load mechanism for terrain component, notification system for it and test it. I divided whole project into stages so progress can be easily followed. Each stage also involves testing of added functionality.

  • Stage 1 (25 May – 14 Jun): Separate LOD loading for t''errain component. Modification of terrain component’s code to allow separate loading of height data.
  • Stage 2 (15 Jun – 21 Jun): Implement loading and preparation methods as background tasks.
  • Stage 3 (22 Jun – 30 Jun): User notification system for background tasks. Make this system configurable by user.
  • Stage 4 (1 Jul – 12 Jul): Make demo using new load mechanism. Option to display various debug statistics as well.

Mid-term deadline (15 Jul):

At this point separate LOD loading should be fully working and integrated in Ogre3D. There should be test/demo showing it’s functionality as well.

After mid-term (17 Jul – 16 Aug 2011):

In this phase implementation of new paging strategy takes place as well as creation of new terrain features showcase demo and its documentation.

  • Stage 5 (17 Jul – 30 Jul): Implement new paging strategy to replace current paging strategy (Grid2DPageStrategy).
  • Stage 6 (31 Jul – 10 Aug): Put together final application to show and demonstrate new features.
  • Stage 7 (11 Aug – 16 Aug): Final tests, bug fixes and documentation.


Any remaining time will be used for overall testing and bug fixing.

Suggested ‘pencils down’ (16 Aug):

All the project’s targets will be fulfilled now and there will be application demonstrating them in a practical way.

Why I’m the one for the project

I’ve been interested in Computer Graphics for a few years now. I have studied CG at Masaryk University in Brno, Czech Republic. I have used Ogre3D in my little pet projects and I’m familiar with it’s design and concepts. I have experience with heightmap terrain rendering as I did that as part of my thesis and worked on game engine which involved heightmap rendering as well. I also have a sound knowledge of C++ coding and project development as I was working as a developer in software company. Now I’ve decided to continue university and so I have whole summer for this GSoC project. I would like to give something useful back to community as well.

Project's Mercurial repository

Repository is hosted on BitBucket: https://bitbucket.org/kuxv/ogre_soc_tpi/overview

What was delivered

Terrain Component changes

Terrain

At the beginning prepare() method is called which reads in terrain stuff like terrain alignment, position but textures as well(lightmap, composite, ...). Terrain's geometry(height and delta) data are not read in thou. That is done by increaseLodLevel() which you call with filename to read data from and tries to increase terrain's LOD level available to display/work with. It runs in background thread if not specified otherwise. It reads in geometry data of next higher LOD level first into temporary memory. After read is done data are distributed into terrain's memory(mHeightData/mDeltaData) in main thread. Then CPU vertex data are created in background and when finished copied to GPU in main thread. If user triggered another increaseLodLevel() in the meantime load of next LOD level is run. decreaseLodLevel() removes highest LOD level loaded or cancels load of one currently in progress. It removes just GPU data as removal of geometry from main memory is unnecessary thanks to organization of mHeightData/mDeltaData as flat array memory.

Terrain file format has been modified to make read of separate LOD data more efficient. Each LOD level is stored in separate chunk and only lowest LOD is saved vertex by vertex. Higher LOD levels are stored as a difference from lower one. File is not compressed as a whole anymore but each geometry chunk is compressed separately so it is possible to read in only specific LOD data. New format moves delta data immediately after height data for easier seeking. Although geometry data are saved separated terrain works with them internaly in a same way as before. Data split/join is done only when data are saved/loaded.

Internaly there are 3 states of terrain LOD:

TargetLOD
Highest LOD level which user wants to be loaded.
LoadedLOD
Highest LOD level which is currently loaded.
StoredLOD
Highest LOD level which is stored in memory.


To retrieve height at point there is bilinear interpolation of height data as it is possible that there is not full LOD level present all the time. Users should be aware that editing terrain heightmap when full data are not present is not possible as well.

There is new TextureAllocator for texture creation/destruction. It's main purpose is to save time when many terrain instances of same size are needed but just a few visible(i.e. paged out). Textures from terrain which is not visible can be reused by other terrain instance which is view.

It is possible to use increaseLodLevel()/decreaseLodLevel() to change LOD level manually as needed there is an interface class TerrainAutoUpdateLod to make that hard work for you. It has just one method:
virtual void autoUpdateLod(Terrain *terrain, const String& filename, bool synchronous, const Any &data) = 0;
terrain
Instance of Terrain which LOD level is going to be changed.
filename
File from which geometry should be loaded
synchronous
Run this as part of main thread or in background
data
Any user specific data.


Idea is that user creates class implementing this method and registers it with terrain using Terrain::setAutoUpdateLod(). Then call Terrain::autoUpdateLod() in main loop to automatically change LOD level. There is one implementation of TerrainAutoUpdateLod as well. It is called TerrainAutoUpdateLodByDistance and changes LOD level according to distance from terrain to camera. It takes one argument holdDistance which tells how much ahead of terrain's actual LOD level change that level should be loaded.

TerrainGroup

Each terrain's LOD level can be changed manually by increaseLodLevel/decreaseLodLevel but terrain has to be prepared first. Although terrain's prepare method runs in main thread TerrainGroup's prepare() runs in background if not specified otherwise.
For easier manipulation when used together with paging TerrainGroup::loadTerrain() behavior is changed. It runs prepare() and increaseLodLevel() in a chain in background to make sure that terrain's lowest LOD level is loaded. Then terrain's auto update LOD mechanism should takes care of load/unload of rest of data. For convenience there is TerrainGroup::autoUpdateLodAll() to update all terrains' LOD level. When new terrain is added shared texture allocator is assigned to it as well.

Terrain Paging Sample

To demonstrate functionality above new sample is added. It is very similar to Terrain Sample as there are not many differences in actual creation/destruction of terrains. Debug info to show status of terrain's load status was added. It shows what LOD level should be loaded(TargetLOD), what level is already loaded(LoadedLOD) and what level is stored in memory(StoredLOD). It allows users to switch on/off automatic LOD level change. It switched off you can increase LOD level using Page Up and decrease using Page Down. By default there is just one terrain but you can specify TerrainGroup's size at compile time using TERRAIN_PAGE_MIN_X/TERRAIN_PAGE_MAX_X/TERRAIN_PAGE_MIN_Y/TERRAIN_PAGE_MAX_Y in a same way as in Terrain Sample. To enable paging just uncomment #define PAGING. Just make sure that all terrains' dat files are created already. You can't use separate LOD loading mechanism when importing from image. So first specify dimensions of TerrainGroup then run sample. After all derived data are calculated and terrains saved close sample. Then #define PAGING, recompile and run sample again.

<HR>
Creative Commons Copyright -- Some rights reserved.


THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED.

BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.

1. Definitions

  • "Collective Work" means a work, such as a periodical issue, anthology or encyclopedia, in which the Work in its entirety in unmodified form, along with a number of other contributions, constituting separate and independent works in themselves, are assembled into a collective whole. A work that constitutes a Collective Work will not be considered a Derivative Work (as defined below) for the purposes of this License.
  • "Derivative Work" means a work based upon the Work or upon the Work and other pre-existing works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which the Work may be recast, transformed, or adapted, except that a work that constitutes a Collective Work will not be considered a Derivative Work for the purpose of this License. For the avoidance of doubt, where the Work is a musical composition or sound recording, the synchronization of the Work in timed-relation with a moving image ("synching") will be considered a Derivative Work for the purpose of this License.
  • "Licensor" means the individual or entity that offers the Work under the terms of this License.
  • "Original Author" means the individual or entity who created the Work.
  • "Work" means the copyrightable work of authorship offered under the terms of this License.
  • "You" means an individual or entity exercising rights under this License who has not previously violated the terms of this License with respect to the Work, or who has received express permission from the Licensor to exercise rights under this License despite a previous violation.
  • "License Elements" means the following high-level license attributes as selected by Licensor and indicated in the title of this License: Attribution, ShareAlike.

2. Fair Use Rights

Nothing in this license is intended to reduce, limit, or restrict any rights arising from fair use, first sale or other limitations on the exclusive rights of the copyright owner under copyright law or other applicable laws.

3. License Grant

Subject to the terms and conditions of this License, Licensor hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright) license to exercise the rights in the Work as stated below:

  • to reproduce the Work, to incorporate the Work into one or more Collective Works, and to reproduce the Work as incorporated in the Collective Works;
  • to create and reproduce Derivative Works;
  • to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission the Work including as incorporated in Collective Works;
  • to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission Derivative Works.
  • For the avoidance of doubt, where the work is a musical composition:
    • Performance Royalties Under Blanket Licenses. Licensor waives the exclusive right to collect, whether individually or via a performance rights society (e.g. ASCAP, BMI, SESAC), royalties for the public performance or public digital performance (e.g. webcast) of the Work.
    • Mechanical Rights and Statutory Royalties. Licensor waives the exclusive right to collect, whether individually or via a music rights society or designated agent (e.g. Harry Fox Agency), royalties for any phonorecord You create from the Work ("cover version") and distribute, subject to the compulsory license created by 17 USC Section 115 of the US Copyright Act (or the equivalent in other jurisdictions).
    • Webcasting Rights and Statutory Royalties. For the avoidance of doubt, where the Work is a sound recording, Licensor waives the exclusive right to collect, whether individually or via a performance-rights society (e.g. SoundExchange), royalties for the public digital performance (e.g. webcast) of the Work, subject to the compulsory license created by 17 USC Section 114 of the US Copyright Act (or the equivalent in other jurisdictions).


The above rights may be exercised in all media and formats whether now known or hereafter devised. The above rights include the right to make such modifications as are technically necessary to exercise the rights in other media and formats. All rights not expressly granted by Licensor are hereby reserved.

4. Restrictions

The license granted in Section 3 above is expressly made subject to and limited by the following restrictions:

  • You may distribute, publicly display, publicly perform, or publicly digitally perform the Work only under the terms of this License, and You must include a copy of, or the Uniform Resource Identifier for, this License with every copy or phonorecord of the Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Work that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder. You may not sublicense the Work. You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Work itself to be made subject to the terms of this License. If You create a Collective Work, upon notice from any Licensor You must, to the extent practicable, remove from the Collective Work any credit as required by clause 4(c), as requested. If You create a Derivative Work, upon notice from any Licensor You must, to the extent practicable, remove from the Derivative Work any credit as required by clause 4(c), as requested.
  • You may distribute, publicly display, publicly perform, or publicly digitally perform a Derivative Work only under the terms of this License, a later version of this License with the same License Elements as this License, or a Creative Commons iCommons license that contains the same License Elements as this License (e.g. Attribution-ShareAlike 2.5 Japan). You must include a copy of, or the Uniform Resource Identifier for, this License or other license specified in the previous sentence with every copy or phonorecord of each Derivative Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Derivative Works that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder, and You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Derivative Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Derivative Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Derivative Work itself to be made subject to the terms of this License.
  • If you distribute, publicly display, publicly perform, or publicly digitally perform the Work or any Derivative Works or Collective Works, You must keep intact all copyright notices for the Work and provide, reasonable to the medium or means You are utilizing: (i) the name of the Original Author (or pseudonym, if applicable) if supplied, and/or (ii) if the Original Author and/or Licensor designate another party or parties (e.g. a sponsor institute, publishing entity, journal) for attribution in Licensor's copyright notice, terms of service or by other reasonable means, the name of such party or parties; the title of the Work if supplied; to the extent reasonably practicable, the Uniform Resource Identifier, if any, that Licensor specifies to be associated with the Work, unless such URI does not refer to the copyright notice or licensing information for the Work; and in the case of a Derivative Work, a credit identifying the use of the Work in the Derivative Work (e.g., "French translation of the Work by Original Author," or "Screenplay based on original Work by Original Author"). Such credit may be implemented in any reasonable manner; provided, however, that in the case of a Derivative Work or Collective Work, at a minimum such credit will appear where any other comparable authorship credit appears and in a manner at least as prominent as such other comparable authorship credit.

5. Representations, Warranties and Disclaimer

UNLESS OTHERWISE AGREED TO BY THE PARTIES IN WRITING, LICENSOR OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE MATERIALS, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU.

6. Limitation on Liability.

EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

7. Termination

  • This License and the rights granted hereunder will terminate automatically upon any breach by You of the terms of this License. Individuals or entities who have received Derivative Works or Collective Works from You under this License, however, will not have their licenses terminated provided such individuals or entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will survive any termination of this License.
  • Subject to the above terms and conditions, the license granted here is perpetual (for the duration of the applicable copyright in the Work). Notwithstanding the above, Licensor reserves the right to release the Work under different license terms or to stop distributing the Work at any time; provided, however that any such election will not serve to withdraw this License (or any other license that has been, or is required to be, granted under the terms of this License), and this License will continue in full force and effect unless terminated as stated above.

8. Miscellaneous

  • Each time You distribute or publicly digitally perform the Work or a Collective Work, the Licensor offers to the recipient a license to the Work on the same terms and conditions as the license granted to You under this License.
  • Each time You distribute or publicly digitally perform a Derivative Work, Licensor offers to the recipient a license to the original Work on the same terms and conditions as the license granted to You under this License.
  • If any provision of this License is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this License, and without further action by the parties to this agreement, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.
  • No term or provision of this License shall be deemed waived and no breach consented to unless such waiver or consent shall be in writing and signed by the party to be charged with such waiver or consent.
  • This License constitutes the entire agreement between the parties with respect to the Work licensed here. There are no understandings, agreements or representations with respect to the Work not specified here. Licensor shall not be bound by any additional provisions that may appear in any communication from You. This License may not be modified without the mutual written agreement of the Licensor and You.