History: SoC2011 Terrain Paging Improvements

Source of version: 5 (current)

{maketoc}
!Project Proposal

!!Motivation
Current terrain component introduced in Ogre3D 1.7 uses chunked level of detail(LOD) together with geomorphing to achieve continuous LOD for terrain(heightmap) rendering. This terrain component performs very well on smaller sets of height data which is it's purpose however problems with limited size of memory come with larger sets of data. Paging of data is common solution for memory issues of this kind and paging component already exists in Ogre3D however it's performance is very poor when used together with terrain component and is not ready to be used in a real project. The current paging strategy which is used for paging terrain tiles in/out is also not optimal in some cases.

!!Project
There are basically two problems using terrain paging in Ogre3D. Poor performance of terrain paging and simple paging strategy used to select tiles for swapping in/out.

Poor terrain paging performance causes visible tearing/delay when it comes to load another page tile for rendering. This project will address this issue by modifying the way terrain component handles data. That involves:
*separate load of specific terrain data belonging to specific LOD instead of only possibility to load whole terrain data at once so more distant terrain tiles could be loaded only in low LOD
*load of terrain data in background or at users own decision
*notification system for various terrain component events like progress of data loading/preparation so user knows what about the progress

In some cases there is a problem with terrain tiles missing in the view or problem with actually rendering more tiles then needed. This issue is caused by default simple paging strategy used(Grid2DPageStrategy) which loads terrain tiles based on radius around camera. The project will bring new paging strategy which will try to be more accurate using actual terrain shape visibility from camera.

There will be introduced new demo to demonstrate above functionality:
*import and prepare terrain data
*load/unload specific terrain LOD
*use of/react on terrain load events
*switch between different LOD strategies

!!Proposal Timeline
!!!Bonding period (27 Apr – 24 May):
*Play and experiment with Ogre3D terrain and paging components and with its demo and test applications to further improve knowledge about it. Examine threading Ogre3D threading model and other needed core functionality as well.
*Actively interact with my mentor and Ogre3D community to discuss final details of my intended implementation. After this I will be absolutely sure about every bit of my future work.

!!!Official coding time starts (25 May – 12 Jul 2011):
First work to do is to implement separate LOD load mechanism for terrain component, notification system for it and test it. I divided whole project into stages so progress can be easily followed. Each stage also involves testing of added functionality.
*''Stage 1 (25 May – 14 Jun)'': Separate LOD loading for t''errain component. Modification of terrain component’s code to allow separate loading of height data.
*''Stage 2 (15 Jun – 21 Jun)'': Implement loading and preparation methods as background tasks.
*''Stage 3 (22 Jun – 30 Jun)'': User notification system for background tasks. Make this system configurable by user.
*''Stage 4 (1 Jul – 12 Jul)'': Make demo using new load mechanism. Option to display various debug statistics as well.

!!!Mid-term deadline (15 Jul):
At this point separate LOD loading should be fully working and integrated in Ogre3D. There should be test/demo showing it’s functionality as well.

!!!After mid-term (17 Jul – 16 Aug 2011):
In this phase implementation of new paging strategy takes place as well as creation of new terrain features showcase demo and its documentation.
*''Stage 5 (17 Jul – 30 Jul)'': Implement new paging strategy to replace current paging strategy (Grid2DPageStrategy).
*''Stage 6 (31 Jul – 10 Aug)'': Put together final application to show and demonstrate new features.
*''Stage 7 (11 Aug – 16 Aug)'': Final tests, bug fixes and documentation.

Any remaining time will be used for overall testing and bug fixing.

!!!Suggested ‘pencils down’ (16 Aug):
All the project’s targets will be fulfilled now and there will be application demonstrating them in a practical way.

!!Why I’m the one for the project
I’ve been interested in Computer Graphics for a few years now. I have studied CG at Masaryk University in Brno, Czech Republic. I have used Ogre3D in my little pet projects and I’m familiar with it’s design and concepts. I have experience with heightmap terrain rendering as I did that as part of my thesis and worked on game engine which involved heightmap rendering as well. I also have a sound knowledge of C++ coding and project development as I was working as a developer in software company. Now I’ve decided to continue university and so I have whole summer for this GSoC project. I would like to give something useful back to community as well.

!Project's Mercurial repository
Repository is hosted on BitBucket: [https://bitbucket.org/kuxv/ogre_soc_tpi/overview|https://bitbucket.org/kuxv/ogre_soc_tpi/overview]

!What was delivered
!!Terrain Component changes
!!!Terrain
At the beginning prepare() method is called which reads in terrain stuff like terrain alignment, position but textures as well(lightmap, composite, ...). Terrain's geometry(height and delta) data are not read in thou. That is done by increaseLodLevel() which you call with filename to read data from and tries to increase terrain's LOD level available to display/work with. It runs in background thread if not specified otherwise. It reads in geometry data of next higher LOD level first into temporary memory. After read is done data are distributed into terrain's memory(mHeightData/mDeltaData) in main thread. Then CPU vertex data are created in background and when finished copied to GPU in main thread. If user triggered another increaseLodLevel() in the meantime load of next LOD level is run. decreaseLodLevel() removes highest LOD level loaded or cancels load of one currently in progress. It removes just GPU data as removal of geometry from main memory is unnecessary thanks to organization of mHeightData/mDeltaData as flat array memory.

Terrain file format has been modified to make read of separate LOD data more efficient. Each LOD level is stored in separate chunk and only lowest LOD is saved vertex by vertex. Higher LOD levels are stored as a difference from lower one. File is not compressed as a whole anymore but each geometry chunk is compressed separately so it is possible to read in only specific LOD data. New format moves delta data immediately after height data for easier seeking. Although geometry data are saved separated terrain works with them internaly in a same way as before. Data split/join is done only when data are saved/loaded.

Internaly there are 3 states of terrain LOD:
{DL()}TargetLOD: Highest LOD level which user wants to be loaded.
LoadedLOD: Highest LOD level which is currently loaded.
StoredLOD: Highest LOD level which is stored in memory.
{DL}

To retrieve height at point there is bilinear interpolation of height data as it is possible that there is not full LOD level present all the time. Users should be aware that editing terrain heightmap when full data are not present is not possible as well.

There is new TextureAllocator for texture creation/destruction. It's main purpose is to save time when many terrain instances of same size are needed but just a few visible(i.e. paged out). Textures from terrain which is not visible can be reused by other terrain instance which is view.

It is possible to use increaseLodLevel()/decreaseLodLevel() to change LOD level manually as needed there is an interface class TerrainAutoUpdateLod to make that hard work for you. It has just one method: {CODE(wrap="1", colors="c++")}virtual void autoUpdateLod(Terrain *terrain, const String& filename, bool synchronous, const Any &data) = 0;{CODE}
{DL()}terrain: Instance of Terrain which LOD level is going to be changed.
filename: File from which geometry should be loaded
synchronous: Run this as part of main thread or in background
data: Any user specific data.{DL}

Idea is that user creates class implementing this method and registers it with terrain using Terrain::setAutoUpdateLod(). Then call Terrain::autoUpdateLod() in main loop to automatically change LOD level. There is one implementation of TerrainAutoUpdateLod as well. It is called TerrainAutoUpdateLodByDistance and changes LOD level according to distance from terrain to camera. It takes one argument holdDistance which tells how much ahead of terrain's actual LOD level change that level should be loaded.

!!!TerrainGroup
Each terrain's LOD level can be changed manually by increaseLodLevel/decreaseLodLevel but terrain has to be prepared first. Although terrain's prepare method runs in main thread TerrainGroup's prepare() runs in background if not specified otherwise.
For easier manipulation when used together with paging TerrainGroup::loadTerrain() behavior is changed. It runs prepare() and increaseLodLevel() in a chain in background to make sure that terrain's lowest LOD level is loaded. Then terrain's auto update LOD mechanism should takes care of load/unload of rest of data. For convenience there is TerrainGroup::autoUpdateLodAll() to update all terrains' LOD level. When new terrain is added shared texture allocator is assigned to it as well.

!!Terrain Paging Sample
To demonstrate functionality above new sample is added. It is very similar to Terrain Sample as there are not many differences in actual creation/destruction of terrains. Debug info to show status of terrain's load status was added. It shows what LOD level should be loaded(TargetLOD), what level is already loaded(LoadedLOD) and what level is stored in memory(StoredLOD). It allows users to switch on/off automatic LOD level change. It switched off you can increase LOD level using Page Up and decrease using Page Down. By default there is just one terrain but you can specify TerrainGroup's size at compile time using TERRAIN_PAGE_MIN_X/TERRAIN_PAGE_MAX_X/TERRAIN_PAGE_MIN_Y/TERRAIN_PAGE_MAX_Y in a same way as in Terrain Sample. To enable paging just uncomment #define PAGING. Just make sure that all terrains' dat files are created already. You can't use separate LOD loading mechanism when importing from image. So first specify dimensions of TerrainGroup then run sample. After all derived data are calculated and terrains saved close sample. Then #define PAGING, recompile and run sample again.
History: SoC2011 Terrain Paging Improvements

Source of version: 5 (current)

Search by Tags

Latest Changes

Search

Online Users
