Project Information

Proposal and discussion thread: http://www.ogre3d.org/forums/viewtopic.php?f=13&t=69553
Mercurial repository: https://bitbucket.org/xiaoxq/ogre_soc_tpi/overview
The basement project: SoC2011 Terrain Paging Improvements

Known Issues

Nothing currently.

How to Test

• Move around to see the terrain blocks loading and unloading.
• Move close and away to see the terrain LOD loading and unloading.

Screen Record

Implementation Design

New Terrain File Format

To be able to load LOD asynchronously, we should save height/delta data separately according to the lowest level it belongs to. So the terrain file format has been adjusted as follows:

The new TERRAIN_CHUNK_ID has been upgraded to 2.
The LOD saving process happens at TerrainLodManager::saveLodData(); When a LOD is requested, the TerrainLodManager just open the file, skip the TERRAINGENERALINFO_CHUNK chunk, and skip the unmatched TERRAINLODDATA_CHUNK chunk, then read the height/delta data into memory. These are done at TerrainLodManager::readLodData();

Particial Compression

To support the new terrain file format, the StreamSerialiser is extended:

class StreamSerialiser
{
  ...
  /** Start (un)compressing data
  @param avail_in Available bytes for uncompressing
  */
  virtual void startDeflate(size_t avail_in = 0);

  /** Stop (un)compressing data
  */
  virtual void stopDeflate();
  ...
}

It's very convenient, and may be also useful for other developers. You can call it like:

...
//write
stream.write("abc",3); // "abc" won't be compressed
stream.startDeflate();
stream.write("def",3); // "def" will be compressed
stream.stopDeflate();
stream.write("ghi",3); // "ghi" won't be compressed
...
//read
stream.read(buffer,3); // buffer = "abc"
stream.startDeflate();
stream.read(buffer,x); // note that x is the length of compressed "def", buffer = "def"
stream.stopDeflate();
stream.read(buffer,3); // buffer = "ghi"

By using particial-compression, the new terrain file saves 65% disk space.

LOD State Machine

Previously, the terrain has only 3 states: Not Prepared, Fully Prepared, Fully Loaded. But now LODs are loaded asynchronously, so the terrain may be only partially prepared/loaded. Every LOD gets its own 3-states, as shown below:

LOD Requests Interface and Implementation

The state machine looks somewhat complicated. Luckily we'll hide the details in TerrainLodManager. The interface for users is very simple:

class Terrain
{
  ...
  void load(int lodLevel, bool synchronous); //request a specified LOD
  void increaseLodLevel(bool synchronous); //increase the current LOD by 1
  void decreaseLodLevel(); //decrease the current LOD by 1
  ...
}

In fact, they are just proxies for TerrainLodManager, which holds three LOD status: HighestLodPrepared, HighestLodLoaded, TargetLodLevel.
Every LOD request will finally call the updateToLodLevel(int lodLevel, bool synchronous). It acts like:

• Update the TargetLodLevel.
• If TargetLodLevel is higher than HighestLodPrepared, prepare it/them
• If TargetLodLevel is higher than HighestLodLoaded, load it/them
• If TargetLodLevel is lower than HighestLodLoaded, unload it/them

Currently we won't unprepare LODs. If wanted, the rule is:

• If TargetLodLevel is lower than HighestLodPrepared, unprepare it/them

Asynchronous LOD Loading

Asynchronously-loading of LOD data is the core feature of this project, the logic is very simple, as shown below.

It's also the biggest source of bugs. My work is to make it safe and stable.

Divide Page Loading Procedure

Terrain page loading is a high cost task, as we should define and load the terrain data, and generate materials for rendering. Currently they all have to be done in main thread, so without any enhancement screen hiccup will occur.
To resolve this we add two strategies. Firstly, queue the page loading requests in a list. Fire another loading only after one is finished. As we should also leave some time for the derived loading, a time slice is inserted between two loadings, which is set as 900ms currently. See TerrainPagedWorldSection::mPagesInLoading for detail.

The most costly task before rendering a terrain is to generate materials, including composite map. It's done when the Terrain::getMaterial firstly called. The main call tree is like:

• Terrain::getMaterial
  • generate
    • addTechnique
      • generateVertexProgram
      • generateFragmentProgram
    • addTechnique
      • generateVertexProgram
      • generateFragmentProgram
  • generateForCompositeMap
    • addTechnique
      • generateVertexProgram
      • generateFragmentProgram

Each generateXxxxxProgram will cost about 20ms on my PC. So the getMaterial() will cost more than 0.12s.
Finally, we queue the generate() and generateForCompositeMap() so that they execute separately. The result is acceptable. If we queue procedures at generateXxxxProgram level, the code becomes too tricky and unreadable. See Terrain::handleGenerateMaterialResponse() for detail.

Introduction to the EndlessWorld Sample

To show the terrain generation and paging feature, we add the EndlessWorld sample. See How to customize my own page loader such as using a Terrain-Generator for the basic knowledge. It's based on the Terrain sample, but for fast terrain loading and unloading, we closed the lightmap, terrain-edition and terrain-saving.
When a new terrain page wanted, a definer will define it. The default one is PerlinNoiseTerrainGenerator. You can think it as an endless 2D perlin noise map, using the coordinates (x,y) as offset to get the value. As a result, the terrain gets ups and downs, and everywhere continuous. With some parameter adjustment, it looks natural, except that there are no sharp edges and corners.
Another EndlessWorld::SimpleTerrainDefiner also implements the interface, which only uses the terrain.png as import data for every terrain, just the same as Terrain sample.

Guide for Users

How to make use of the Asynchronous-Loading feature

The simplest way is to manipulate the terrain LOD manually, I mean, calling the APIs above directly. As an example, you can run the EndlessWorld sample, and use PAGE-UP/PAGE-DOWN to increase/decrease terrains' LOD manually.
Another way is to ask an Auto-Update-Lod strategy to call those APIs for you. To support this, the TerrainGroup supplies APIs:

class TerrainGroup
{
  ...
  void increaseLodLevel(long x, long y, bool synchronous = false); // Loads terrain's next LOD level.
  void decreaseLodLevel(long x, long y); // Removes terrain's highest LOD level.

  void setAutoUpdateLod(TerrainAutoUpdateLod* updater);
  void autoUpdateLod(long x, long y, bool synchronous, const Any &data); // Automatically update terrain's LOD.
  void autoUpdateLodAll(bool synchronous, const Any &data);
  ...
}

Their effects are all self-explanatory. You can call setAutoUpdateLod() to set or change strategy at anytime. The new strategy will take effect at the next autoUpdateLod(), which is usually called once every frame.
Run the EndlessWorld sample again to see how it works. Make sure the AutoLod box has been checked (default). Note that if you have tried the PAGE-UP/PAGE-DOWN, the sample will switch to manual-mode automatically. If so, just check the AutoLod box again.
Currently we supply a strategy which update LOD according to the distance between camera and the terrain. You can use id DefaultTerrainAutoUpdateLodStrategy::DEFAULT or DefaultTerrainAutoUpdateLodStrategy::BY_DISTANCE to reference it. To this strategy, the parameter "data" passed to autoUpdateLod() is the distance threshold, whose type is Ogre::Real. In the EndlessWorld sample it is set as 3000.

How to add my own strategy of Auto-Update-Lod

Call the API below to register your own strategy:

class TerrainAutoUpdateLodFactory
{
  ...
  static void registerAutoUpdateLodStrategy( uint32 strategy, TerrainAutoUpdateLod* updater )
  ...
}

Obviously, your implementation should inherit from TerrainAutoUpdateLod, which supplies a autoUpdateLod() function.
The strategy id should larger than 1, because 0 is occupied by NONE, and 1 is occupied by BY_DISTANCE (also called DEFAULT, see it above).
Attention, TerrainAutoUpdateLod is also a Singleton. It's your responsibility to initialize the instance properly. You should make sure the registered pointer can be successfully accessed whenever it's requested.

How to customize my own page loader, such as using a Terrain-Generator

Previously, you must have existed terrain data file to use the terrain paging feature, as the TerrainPagedWorldSection::loadPage() only calls TerrainGroup::define(x,y). Now things changed. The loadPage() will using definer to define terrains, whose base class is TerrainPagedWorldSection::TerrainDefiner. You can implement your own definer by inheriting from it and using TerrainPagedWorldSection::setDefiner() to make it in use.

class TerrainPagedWorldSection
{
public:
  ...
  class TerrainDefiner : public TerrainAlloc
  {
  public:
    virtual void define(TerrainGroup* terrainGroup, long x, long y)
    {
      terrainGroup->defineTerrain(x,y);
    }
  };

  void setDefiner(TerrainDefiner* terrainDefiner)
  {
    if(mTerrainDefiner)
      OGRE_DELETE mTerrainDefiner;
    mTerrainDefiner = terrainDefiner;
  }
  ...
}

For example, you can call TerrainGroup::define(x,y,0.0) to define a flat terrain when the exact terrain data file doesn't exist. And the EndlessWorld sample shows another definer, which generates heightmap at runtime.

Weekly Progress

2012-08-19

• Finish the EndlessWorld sample, resolve bugs.
• Merge with trunk default.
• Update wiki.

2012-08-13

• Almost finished the EndlessWorld sample.

2012-08-05

• Tried to make loading page smoother: queueing page-loadings, and divide material generation procedure.

2012-07-23

• Extend StreamSerialiser to support Part-by-Part compression, apply it to terrain file, saves 65% disk space.
• Fix some bugs, such as camera-jump-back.
• PG-UP/DOWN to inc/dec LOD manually (revert, and enhance).
• Update wiki.

2012-07-16

• Make the new terrain system work with paging ON
• Fix the init problem, the sample can be restarted, now
• Make AutoUpdateLodStrategy as singleton, and user can register his own strategy

2012-07-10

• After 2 big fixes, it works quite better, now.
• But if set paging ON, it still has some bugs.

2012-07-02

• Almost finished the Implementation Design section of the wiki.
• Use LodManager though no file bound.
• Fix an "Empty CPU Buffer" exception.

2012-06-25

• Write some wiki.

2012-06-10

• Alpha release.

2012-06-03

• Almost finish extracting the LodManager from Terrain.
• Going to read height/delta once for all.

2012-05-27

• Start reviewing the code, I think I have understood the idea deeply.
• Small refactoring, extract out a TerrainLodManager, which makes the Terrain component as clear as the original.(80% completed)

2012-05-20

• Finish the sequence diagram, and get to understand most of the previous implementation ideas.
• Have found some possible buggy point. See them above for detail.

2012-05-14

• Keep on warming up with the code.
• Start drawing a sequence diagram to show the work flow of LOD and paging.

2012-05-06

• Some code cleanup.

2012-04-29

• Create repository by importing from kuxv's, and create this wiki page.
• A small fix to make it compilable on my x64 system. Then it runs quite well.
• Merge new changesets from trunk.
• Confirm my development environment: Thinkpad T410s( Intel i5, Nvidia NVS 3100M, 8G memory ), Ubuntu 12.04 64bit. I also have a Win7 64bit, but only for testing.

THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED.

BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.

1. Definitions

• "Collective Work" means a work, such as a periodical issue, anthology or encyclopedia, in which the Work in its entirety in unmodified form, along with a number of other contributions, constituting separate and independent works in themselves, are assembled into a collective whole. A work that constitutes a Collective Work will not be considered a Derivative Work (as defined below) for the purposes of this License.
• "Derivative Work" means a work based upon the Work or upon the Work and other pre-existing works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which the Work may be recast, transformed, or adapted, except that a work that constitutes a Collective Work will not be considered a Derivative Work for the purpose of this License. For the avoidance of doubt, where the Work is a musical composition or sound recording, the synchronization of the Work in timed-relation with a moving image ("synching") will be considered a Derivative Work for the purpose of this License.
• "Licensor" means the individual or entity that offers the Work under the terms of this License.
• "Original Author" means the individual or entity who created the Work.
• "Work" means the copyrightable work of authorship offered under the terms of this License.
• "You" means an individual or entity exercising rights under this License who has not previously violated the terms of this License with respect to the Work, or who has received express permission from the Licensor to exercise rights under this License despite a previous violation.
• "License Elements" means the following high-level license attributes as selected by Licensor and indicated in the title of this License: Attribution, ShareAlike.

2. Fair Use Rights

Nothing in this license is intended to reduce, limit, or restrict any rights arising from fair use, first sale or other limitations on the exclusive rights of the copyright owner under copyright law or other applicable laws.

3. License Grant

Subject to the terms and conditions of this License, Licensor hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright) license to exercise the rights in the Work as stated below:

• to reproduce the Work, to incorporate the Work into one or more Collective Works, and to reproduce the Work as incorporated in the Collective Works;
• to create and reproduce Derivative Works;
• to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission the Work including as incorporated in Collective Works;
• to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission Derivative Works.
• For the avoidance of doubt, where the work is a musical composition:
  • Performance Royalties Under Blanket Licenses. Licensor waives the exclusive right to collect, whether individually or via a performance rights society (e.g. ASCAP, BMI, SESAC), royalties for the public performance or public digital performance (e.g. webcast) of the Work.
  • Mechanical Rights and Statutory Royalties. Licensor waives the exclusive right to collect, whether individually or via a music rights society or designated agent (e.g. Harry Fox Agency), royalties for any phonorecord You create from the Work ("cover version") and distribute, subject to the compulsory license created by 17 USC Section 115 of the US Copyright Act (or the equivalent in other jurisdictions).
  • Webcasting Rights and Statutory Royalties. For the avoidance of doubt, where the Work is a sound recording, Licensor waives the exclusive right to collect, whether individually or via a performance-rights society (e.g. SoundExchange), royalties for the public digital performance (e.g. webcast) of the Work, subject to the compulsory license created by 17 USC Section 114 of the US Copyright Act (or the equivalent in other jurisdictions).

The above rights may be exercised in all media and formats whether now known or hereafter devised. The above rights include the right to make such modifications as are technically necessary to exercise the rights in other media and formats. All rights not expressly granted by Licensor are hereby reserved.

4. Restrictions

The license granted in Section 3 above is expressly made subject to and limited by the following restrictions:

• You may distribute, publicly display, publicly perform, or publicly digitally perform the Work only under the terms of this License, and You must include a copy of, or the Uniform Resource Identifier for, this License with every copy or phonorecord of the Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Work that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder. You may not sublicense the Work. You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Work itself to be made subject to the terms of this License. If You create a Collective Work, upon notice from any Licensor You must, to the extent practicable, remove from the Collective Work any credit as required by clause 4(c), as requested. If You create a Derivative Work, upon notice from any Licensor You must, to the extent practicable, remove from the Derivative Work any credit as required by clause 4(c), as requested.
• You may distribute, publicly display, publicly perform, or publicly digitally perform a Derivative Work only under the terms of this License, a later version of this License with the same License Elements as this License, or a Creative Commons iCommons license that contains the same License Elements as this License (e.g. Attribution-ShareAlike 2.5 Japan). You must include a copy of, or the Uniform Resource Identifier for, this License or other license specified in the previous sentence with every copy or phonorecord of each Derivative Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Derivative Works that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder, and You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Derivative Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Derivative Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Derivative Work itself to be made subject to the terms of this License.
• If you distribute, publicly display, publicly perform, or publicly digitally perform the Work or any Derivative Works or Collective Works, You must keep intact all copyright notices for the Work and provide, reasonable to the medium or means You are utilizing: (i) the name of the Original Author (or pseudonym, if applicable) if supplied, and/or (ii) if the Original Author and/or Licensor designate another party or parties (e.g. a sponsor institute, publishing entity, journal) for attribution in Licensor's copyright notice, terms of service or by other reasonable means, the name of such party or parties; the title of the Work if supplied; to the extent reasonably practicable, the Uniform Resource Identifier, if any, that Licensor specifies to be associated with the Work, unless such URI does not refer to the copyright notice or licensing information for the Work; and in the case of a Derivative Work, a credit identifying the use of the Work in the Derivative Work (e.g., "French translation of the Work by Original Author," or "Screenplay based on original Work by Original Author"). Such credit may be implemented in any reasonable manner; provided, however, that in the case of a Derivative Work or Collective Work, at a minimum such credit will appear where any other comparable authorship credit appears and in a manner at least as prominent as such other comparable authorship credit.

5. Representations, Warranties and Disclaimer

UNLESS OTHERWISE AGREED TO BY THE PARTIES IN WRITING, LICENSOR OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE MATERIALS, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU.

6. Limitation on Liability.

EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

7. Termination

• This License and the rights granted hereunder will terminate automatically upon any breach by You of the terms of this License. Individuals or entities who have received Derivative Works or Collective Works from You under this License, however, will not have their licenses terminated provided such individuals or entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will survive any termination of this License.
• Subject to the above terms and conditions, the license granted here is perpetual (for the duration of the applicable copyright in the Work). Notwithstanding the above, Licensor reserves the right to release the Work under different license terms or to stop distributing the Work at any time; provided, however that any such election will not serve to withdraw this License (or any other license that has been, or is required to be, granted under the terms of this License), and this License will continue in full force and effect unless terminated as stated above.

8. Miscellaneous

• Each time You distribute or publicly digitally perform the Work or a Collective Work, the Licensor offers to the recipient a license to the Work on the same terms and conditions as the license granted to You under this License.
• Each time You distribute or publicly digitally perform a Derivative Work, Licensor offers to the recipient a license to the original Work on the same terms and conditions as the license granted to You under this License.
• If any provision of this License is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this License, and without further action by the parties to this agreement, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.
• No term or provision of this License shall be deemed waived and no breach consented to unless such waiver or consent shall be in writing and signed by the party to be charged with such waiver or consent.
• This License constitutes the entire agreement between the parties with respect to the Work licensed here. There are no understandings, agreements or representations with respect to the Work not specified here. Licensor shall not be bound by any additional provisions that may appear in any communication from You. This License may not be modified without the mutual written agreement of the Licensor and You.