History: SoC2012 Improve and Demo the Terrain System

Source of version: 84 (current)

{maketoc}

!Project Information
Proposal and discussion thread: [http://www.ogre3d.org/forums/viewtopic.php?f=13&t=69553|http://www.ogre3d.org/forums/viewtopic.php?f=13&t=69553]
Mercurial repository: [https://bitbucket.org/xiaoxq/ogre_soc_tpi/overview|https://bitbucket.org/xiaoxq/ogre_soc_tpi/overview]
The basement project: [http://www.ogre3d.org/tikiwiki/SoC2011+Terrain+Paging+Improvements|SoC2011 Terrain Paging Improvements]
!!Known Issues
Nothing currently.
!!How to Test
*Move around to see the terrain blocks loading and unloading.
*Move close and away to see the terrain LOD loading and unloading.
!!Screen Record
{FLASH(type="vimeo",vimeo="http://vimeo.com/45455776")}{FLASH}{FLASH(type="vimeo",vimeo="http://vimeo.com/47489080")}{FLASH}
!Implementation Design
!!New Terrain File Format
To be able to load LOD asynchronously, we should save height/delta data separately according to the lowest level it belongs to. So the terrain file format has been adjusted as follows:
{BOX(class="")}<pre>
&lt;TERRAIN_CHUNK>
  &lt;TERRAINGENERALINFO_CHUNK>
    uint8 align; uint16 Size; Real WorldSize; uint16 MaxBatchSize; uint16 MinBatchSize; Vector3 Pos;
  &lt;/TERRAINGENERALINFO_CHUNK>

  &lt;TERRAINLODDATA_CHUNK>zip( LOD{n-1} HeightData, DeltaData; )&lt;/TERRAINLODDATA_CHUNK>
  ...
  &lt;TERRAINLODDATA_CHUNK>zip( LOD{0} HeightData, DeltaData; )&lt;/TERRAINLODDATA_CHUNK>
zip(
  LayerInstanceList Layers;
  LayerBlendData;

  &lt;TERRAINDERIVEDDATA_CHUNK> NormalData; &lt;/TERRAINDERIVEDDATA_CHUNK>
  &lt;TERRAINDERIVEDDATA_CHUNK> ColourMap; &lt;/TERRAINDERIVEDDATA_CHUNK>
  &lt;TERRAINDERIVEDDATA_CHUNK> LightMap; &lt;/TERRAINDERIVEDDATA_CHUNK>
  &lt;TERRAINDERIVEDDATA_CHUNK> CompositeMap; &lt;/TERRAINDERIVEDDATA_CHUNK>
  QuadTree;
)
&lt;/TERRAIN_CHUNK></pre>
{BOX}
The new TERRAIN_CHUNK_ID has been upgraded to 2.
The LOD saving process happens at TerrainLodManager::saveLodData(); When a LOD is requested, the TerrainLodManager just open the file, skip the TERRAINGENERALINFO_CHUNK chunk, and skip the unmatched TERRAINLODDATA_CHUNK chunk, then read the height/delta data into memory. These are done at TerrainLodManager::readLodData();
!!Particial Compression
To support the new terrain file format, the StreamSerialiser is extended:
{CODE(wrap="1", colors="c++")}
class StreamSerialiser
{
  ...
  /** Start (un)compressing data
  @param avail_in Available bytes for uncompressing
  */
  virtual void startDeflate(size_t avail_in = 0);

  /** Stop (un)compressing data
  */
  virtual void stopDeflate();
  ...
}
{CODE}
It's very convenient, and may be also useful for other developers. You can call it like:
{CODE(wrap="1", colors="c++")}
...
//write
stream.write("abc",3); // "abc" won't be compressed
stream.startDeflate();
stream.write("def",3); // "def" will be compressed
stream.stopDeflate();
stream.write("ghi",3); // "ghi" won't be compressed
...
//read
stream.read(buffer,3); // buffer = "abc"
stream.startDeflate();
stream.read(buffer,x); // note that x is the length of compressed "def", buffer = "def"
stream.stopDeflate();
stream.read(buffer,3); // buffer = "ghi"
{CODE}
By using particial-compression, the new terrain file saves 65% disk space.
!!LOD State Machine
Previously, the terrain has only 3 states: Not Prepared, Fully Prepared, Fully Loaded. But now LODs are loaded asynchronously, so the terrain may be only partially prepared/loaded. Every LOD gets its own 3-states, as shown below:
{img fileId="2135" rel="box[g]"}

!!LOD Requests Interface and Implementation
The state machine looks somewhat complicated. Luckily we'll hide the details in TerrainLodManager. The interface for users is very simple:
{CODE(wrap="1", colors="c++")}
class Terrain
{
  ...
  void load(int lodLevel, bool synchronous); //request a specified LOD
  void increaseLodLevel(bool synchronous); //increase the current LOD by 1
  void decreaseLodLevel(); //decrease the current LOD by 1
  ...
}
{CODE}
In fact, they are just proxies for TerrainLodManager, which holds three LOD status: HighestLodPrepared, HighestLodLoaded, TargetLodLevel.
Every LOD request will finally call the updateToLodLevel(int lodLevel, bool synchronous). It acts like:
* Update the TargetLodLevel.
* If TargetLodLevel is higher than HighestLodPrepared, prepare it/them
* If TargetLodLevel is higher than HighestLodLoaded, load it/them
* If TargetLodLevel is lower than HighestLodLoaded, unload it/them
Currently we won't unprepare LODs. If wanted, the rule is:
* If TargetLodLevel is lower than HighestLodPrepared, unprepare it/them
!!Asynchronous LOD Loading
Asynchronously-loading of LOD data is the core feature of this project, the logic is very simple, as shown below.
{img fileId="2136" rel="box[g]"}
It's also the biggest source of bugs. My work is to make it safe and stable.
!!Divide Page Loading Procedure
Terrain page loading is a high cost task, as we should define and load the terrain data, and generate materials for rendering. Currently they all have to be done in main thread, so without any enhancement screen hiccup will occur.
To resolve this we add two strategies. Firstly, queue the page loading requests in a list. Fire another loading only after one is finished. As we should also leave some time for the derived loading, a time slice is inserted between two loadings, which is set as 900ms currently. See TerrainPagedWorldSection::mPagesInLoading for detail.

The most costly task before rendering a terrain is to generate materials, including composite map. It's done when the Terrain::getMaterial firstly called. The main call tree is like:
*Terrain::getMaterial
**generate
***addTechnique
****generateVertexProgram
****generateFragmentProgram
***addTechnique
****generateVertexProgram
****generateFragmentProgram
**generateForCompositeMap
***addTechnique
****generateVertexProgram
****generateFragmentProgram
Each generateXxxxxProgram will cost about 20ms on my PC. So the getMaterial() will cost more than 0.12s.
Finally, we queue the generate() and generateForCompositeMap() so that they execute separately. The result is acceptable. If we queue procedures at generateXxxxProgram level, the code becomes too tricky and unreadable. See Terrain::handleGenerateMaterialResponse() for detail.
!!Introduction to the EndlessWorld Sample
To show the terrain generation and paging feature, we add the EndlessWorld sample. See {ALINK(aname="How_to_customize_my_own_page_loader_such_as_using_a_Terrain-Generator")}How to customize my own page loader such as using a Terrain-Generator{ALINK} for the basic knowledge. It's based on the Terrain sample, but for fast terrain loading and unloading, we closed the lightmap, terrain-edition and terrain-saving.
When a new terrain page wanted, a definer will define it. The default one is PerlinNoiseTerrainGenerator. You can think it as an endless 2D perlin noise map, using the coordinates (x,y) as offset to get the value. As a result, the terrain gets ups and downs, and everywhere continuous. With some parameter adjustment, it looks natural, except that there are no sharp edges and corners.
Another EndlessWorld::SimpleTerrainDefiner also implements the interface, which only uses the terrain.png as import data for every terrain, just the same as Terrain sample.
!Guide for Users
!!How to make use of the Asynchronous-Loading feature
The simplest way is to manipulate the terrain LOD manually, I mean, calling the APIs above directly. As an example, you can run the EndlessWorld sample, and use PAGE-UP/PAGE-DOWN to increase/decrease terrains' LOD manually.
Another way is to ask an Auto-Update-Lod strategy to call those APIs for you. To support this, the TerrainGroup supplies APIs:
{CODE(wrap="1", colors="c++")}
class TerrainGroup
{
  ...
  void increaseLodLevel(long x, long y, bool synchronous = false); // Loads terrain's next LOD level.
  void decreaseLodLevel(long x, long y); // Removes terrain's highest LOD level.

  void setAutoUpdateLod(TerrainAutoUpdateLod* updater);
  void autoUpdateLod(long x, long y, bool synchronous, const Any &data); // Automatically update terrain's LOD.
  void autoUpdateLodAll(bool synchronous, const Any &data);
  ...
}
{CODE}
Their effects are all self-explanatory. You can call __setAutoUpdateLod()__ to set or change strategy at anytime. The new strategy will take effect at the next __autoUpdateLod()__, which is usually called once every frame.
Run the EndlessWorld sample again to see how it works. Make sure the AutoLod box has been checked (default). Note that if you have tried the PAGE-UP/PAGE-DOWN, the sample will switch to manual-mode automatically. If so, just check the AutoLod box again.
Currently we supply a strategy which update LOD according to the distance between camera and the terrain. You can use id __DefaultTerrainAutoUpdateLodStrategy::DEFAULT__ or __DefaultTerrainAutoUpdateLodStrategy::BY_DISTANCE__ to reference it. To this strategy, the parameter "data" passed to __autoUpdateLod()__ is the distance threshold, whose type is Ogre::Real. In the EndlessWorld sample it is set as 3000.
!!How to add my own strategy of Auto-Update-Lod
Call the API below to register your own strategy:
{CODE(wrap="1", colors="c++")}
class TerrainAutoUpdateLodFactory
{
  ...
  static void registerAutoUpdateLodStrategy( uint32 strategy, TerrainAutoUpdateLod* updater )
  ...
}
{CODE}
Obviously, your implementation should inherit from TerrainAutoUpdateLod, which supplies a autoUpdateLod() function.
The strategy id should larger than 1, because 0 is occupied by NONE, and 1 is occupied by BY_DISTANCE (also called DEFAULT, see it above).
Attention, TerrainAutoUpdateLod is also a Singleton. It's your responsibility to initialize the instance properly. You should make sure the registered pointer can be successfully accessed whenever it's requested.
!!How to customize my own page loader, such as using a Terrain-Generator
Previously, you must have existed terrain data file to use the terrain paging feature, as the TerrainPagedWorldSection::loadPage() only calls TerrainGroup::define(x,y). Now things changed. The loadPage() will using definer to define terrains, whose base class is TerrainPagedWorldSection::TerrainDefiner. You can implement your own definer by inheriting from it and using TerrainPagedWorldSection::setDefiner() to make it in use.
{CODE(wrap="1", colors="c++")}
class TerrainPagedWorldSection
{
public:
  ...
  class TerrainDefiner : public TerrainAlloc
  {
  public:
    virtual void define(TerrainGroup* terrainGroup, long x, long y)
    {
      terrainGroup->defineTerrain(x,y);
    }
  };

  void setDefiner(TerrainDefiner* terrainDefiner)
  {
    if(mTerrainDefiner)
      OGRE_DELETE mTerrainDefiner;
    mTerrainDefiner = terrainDefiner;
  }
  ...
}
{CODE}
For example, you can call TerrainGroup::define(x,y,0.0) to define a flat terrain when the exact terrain data file doesn't exist. And the EndlessWorld sample shows another definer, which generates heightmap at runtime.
!Weekly Progress
__2012-08-19__
*Finish the EndlessWorld sample, resolve bugs.
*Merge with trunk default.
*Update wiki.
__2012-08-13__
*Almost finished the EndlessWorld sample.
__2012-08-05__
*Tried to make loading page smoother: queueing page-loadings, and divide material generation procedure.
__2012-07-23__
*Extend StreamSerialiser to support Part-by-Part compression, apply it to terrain file, saves 65% disk space.
*Fix some bugs, such as camera-jump-back.
*PG-UP/DOWN to inc/dec LOD manually (revert, and enhance).
*Update wiki.
__2012-07-16__
*Make the new terrain system work with paging ON
*Fix the init problem, the sample can be restarted, now
*Make AutoUpdateLodStrategy as singleton, and user can register his own strategy
__2012-07-10__
*After 2 big fixes, it works quite better, now.
*But if set paging ON, it still has some bugs.
__2012-07-02__
*Almost finished the Implementation Design section of the wiki.
*Use LodManager though no file bound.
*Fix an "Empty CPU Buffer" exception.
__2012-06-25__
*Write some wiki.
__2012-06-10__
*Alpha release.
__2012-06-03__
*Almost finish extracting the LodManager from Terrain.
*Going to read height/delta once for all.
__2012-05-27__
*Start reviewing the code, I think I have understood the idea deeply.
*Small refactoring, extract out a TerrainLodManager, which makes the Terrain component as clear as the original.(80% completed)
__2012-05-20__
*Finish the sequence diagram, and get to understand most of the previous implementation ideas.
*Have found some possible buggy point. See them above for detail.
__2012-05-14__
*Keep on warming up with the code.
*Start drawing a sequence diagram to show the work flow of LOD and paging.
__2012-05-06__
*Some code cleanup.
__2012-04-29__
*Create repository by importing from kuxv's, and create this wiki page.
*A small fix to make it compilable on my x64 system. Then it runs quite well.
*Merge new changesets from trunk.
*Confirm my development environment: Thinkpad T410s( Intel i5, Nvidia NVS 3100M, 8G memory ), Ubuntu 12.04 64bit. I also have a Win7 64bit, but only for testing.
History: SoC2012 Improve and Demo the Terrain System

Source of version: 84 (current)

Search by Tags

Latest Changes

Search

Online Users
