History: SoC2012 Improve and Demo the Terrain System
Source of version: 65
Copy to clipboard
{maketoc}
!Project Information
Proposal and discussion thread: [http://www.ogre3d.org/forums/viewtopic.php?f=13&t=69553|http://www.ogre3d.org/forums/viewtopic.php?f=13&t=69553]
Mercurial repository: [https://bitbucket.org/xiaoxq/ogre_soc_tpi/overview|https://bitbucket.org/xiaoxq/ogre_soc_tpi/overview]
The basement project: [http://www.ogre3d.org/tikiwiki/SoC2011+Terrain+Paging+Improvements|SoC2011 Terrain Paging Improvements]
!!Known Issues
Hiccups when loading page
!Implementation Design
!!New Terrain File Format
To be able to load LOD asynchronously, we should save height/delta data separately according to the lowest level it belongs to. So the terrain file format has been adjusted as follows:
{BOX(class="")}<pre>
<TERRAIN_CHUNK>
<TERRAINGENERALINFO_CHUNK>
uint8 align; uint16 Size; Real WorldSize; uint16 MaxBatchSize; uint16 MinBatchSize; Vector3 Pos;
</TERRAINGENERALINFO_CHUNK>
<TERRAINLODDATA_CHUNK>zip( LOD{n-1} HeightData, DeltaData; )</TERRAINLODDATA_CHUNK>
...
<TERRAINLODDATA_CHUNK>zip( LOD{0} HeightData, DeltaData; )</TERRAINLODDATA_CHUNK>
zip(
LayerInstanceList Layers;
LayerBlendData;
<TERRAINDERIVEDDATA_CHUNK> NormalData; </TERRAINDERIVEDDATA_CHUNK>
<TERRAINDERIVEDDATA_CHUNK> ColourMap; </TERRAINDERIVEDDATA_CHUNK>
<TERRAINDERIVEDDATA_CHUNK> LightMap; </TERRAINDERIVEDDATA_CHUNK>
<TERRAINDERIVEDDATA_CHUNK> CompositeMap; </TERRAINDERIVEDDATA_CHUNK>
QuadTree;
)
</TERRAIN_CHUNK></pre>
{BOX}
The new TERRAIN_CHUNK_ID has been upgraded to 2.
The LOD saving process happens at TerrainLodManager::saveLodData(); When a LOD is requested, the TerrainLodManager just open the file, skip the TERRAINGENERALINFO_CHUNK chunk, and skip the unmatched TERRAINLODDATA_CHUNK chunk, then read the height/delta data into memory. These are done at TerrainLodManager::readLodData();
!!Particial Compression
To support the new terrain file format, the StreamSerialiser is extended:
{CODE(wrap="1", colors="c++")}
class StreamSerialiser
{
...
/** Start (un)compressing data
@param avail_in Available bytes for uncompressing
*/
virtual void startDeflate(size_t avail_in = 0);
/** Stop (un)compressing data
*/
virtual void stopDeflate();
...
}
{CODE}
It's very convenient, and may be also useful for other developers. You can call it like:
{CODE(wrap="1", colors="c++")}
...
//write
stream.write("abc",3); // "abc" won't be compressed
stream.startDeflate();
stream.write("def",3); // "def" will be compressed
stream.stopDeflate();
stream.write("ghi",3); // "ghi" won't be compressed
...
//read
stream.read(buffer,3); // buffer = "abc"
stream.startDeflate();
stream.read(buffer,x); // note that x is the length of compressed "def", buffer = "def"
stream.stopDeflate();
stream.read(buffer,3); // buffer = "ghi"
{CODE}
By using particial-compression, the new terrain file saves 65% disk space.
!!LOD State Machine
Previously, the terrain has only 3 states: Not Prepared, Fully Prepared, Fully Loaded. But now LODs are loaded asynchronously, so the terrain may be only partially prepared/loaded. Every LOD gets its own 3-states, as shown below:
{img fileId="2135" rel="box[g]"}
!!LOD Requests Interface and Implementation
The state machine looks somewhat complicated. Luckily we'll hide the details in TerrainLodManager. The interface for users is very simple:
{CODE(wrap="1", colors="c++")}
class Terrain
{
...
void load(int lodLevel, bool synchronous); //request a specified LOD
void increaseLodLevel(bool synchronous); //increase the current LOD by 1
void decreaseLodLevel(); //decrease the current LOD by 1
...
}
{CODE}
In fact, they are just proxies for TerrainLodManager, which holds three LOD status: HighestLodPrepared, HighestLodLoaded, TargetLodLevel.
Every LOD request will finally call the updateToLodLevel(int lodLevel, bool synchronous). It acts like:
* Update the TargetLodLevel.
* If TargetLodLevel is higher than HighestLodPrepared, prepare it/them
* If TargetLodLevel is higher than HighestLodLoaded, load it/them
* If TargetLodLevel is lower than HighestLodLoaded, unload it/them
Currently we won't unprepare LODs. If wanted, the rule is:
* If TargetLodLevel is lower than HighestLodPrepared, unprepare it/them
!!Asynchronous LOD Loading
Asynchronously-loading of LOD data is the core feature of this project, the logic is very simple, as shown below.
{img fileId="2136" rel="box[g]"}
It's also the biggest source of bugs. My work is to make it safe and stable.
!Guide for Users
!!How to make use of the Asynchronous-Loading feature
The simplest way is to manipulate the terrain LOD manually, I mean, calling the APIs above directly. As an example, you can run the EndlessWorld sample, and use PAGE-UP/PAGE-DOWN to increase/decrease terrains' LOD manually.
Another way is to ask an Auto-Update-Lod strategy to call those APIs for you. To support this, the TerrainGroup supplies APIs:
{CODE(wrap="1", colors="c++")}
class TerrainGroup
{
...
void increaseLodLevel(long x, long y, bool synchronous = false); // Loads terrain's next LOD level.
void decreaseLodLevel(long x, long y); // Removes terrain's highest LOD level.
void setAutoUpdateLodStrategy(uint32 strategy);
void autoUpdateLod(long x, long y, bool synchronous, const Any &data); // Automatically update terrain's LOD.
void autoUpdateLodAll(bool synchronous, const Any &data);
...
}
{CODE}
Their effects are all self-explanatory. You can call setAutoUpdateLodStrategy() to set or change strategy at anytime. The new strategy will take effect at the next autoUpdateLod call, which is usually called once every frame. Run the EndlessWorld sample again to see how it works. Make sure the AutoLod box has been checked (default). Note that if you have tried the PAGE-UP/PAGE-DOWN, the sample will switch to manual-mode automatically. If so, just check the AutoLod box again.
Currently we supply a strategy which update LOD according to the distance between camera and the terrain. You can use id DefaultTerrainAutoUpdateLodStrategy::DEFAULT or DefaultTerrainAutoUpdateLodStrategy::BY_DISTANCE to reference it. To this strategy, the parameter "data" passed to autoUpdateLod() is the distance threshold, whose type is Ogre::Real. In the EndlessWorld sample it is set as 3000.
!!How to add my own strategy of Auto-Update-Lod
Call the API below to register your own strategy:
{CODE(wrap="1", colors="c++")}
class TerrainAutoUpdateLodFactory
{
...
static void registerAutoUpdateLodStrategy( uint32 strategy, TerrainAutoUpdateLod* updater )
...
}
{CODE}
Obviously, your implementation should inherit from TerrainAutoUpdateLod, which supplies a autoUpdateLod() function.
The strategy id should larger than 1, because 0 is occupied by NONE, and 1 is occupied by BY_DISTANCE (also called DEFAULT, see it above).
Attention, TerrainAutoUpdateLod is also a Singleton. It's your responsibility to initialize the instance properly. You should make sure the registered pointer can be successfully accessed whenever it's requested.
!Weekly Progress
__2012-04-29__
*Create repository by importing from kuxv's, and create this wiki page.
*A small fix to make it compilable on my x64 system. Then it runs quite well.
*Merge new changesets from trunk.
*Confirm my development environment: Thinkpad T410s( Intel i5, Nvidia NVS 3100M, 8G memory ), Ubuntu 12.04 64bit. I also have a Win7 64bit, but only for testing.
__2012-05-06__
*Some code cleanup.
__2012-05-14__
*Keep on warming up with the code.
*Start drawing a sequence diagram to show the work flow of LOD and paging.
__2012-05-20__
*Finish the sequence diagram, and get to understand most of the previous implementation ideas.
*Have found some possible buggy point. See them above for detail.
__2012-05-27__
*Start reviewing the code, I think I have understood the idea deeply.
*Small refactoring, extract out a TerrainLodManager, which makes the Terrain component as clear as the original.(80% completed)
__2012-06-03__
*Almost finish extracting the LodManager from Terrain.
*Going to read height/delta once for all.
__2012-06-10__
*Alpha release.
__2012-06-25__
*Write some wiki.
__2012-07-02__
*Almost finished the Implementation Design section of the wiki.
*Use LodManager though no file bound.
*Fix an "Empty CPU Buffer" exception.
__2012-07-10__
*After 2 big fixes, it works quite better, now.
*But if set paging ON, it still has some bugs.