History: SoC2012 Implementation of Off-Screen Particles

• «
• »

            !Aim of the project:

Thanks to this project Ogre users will benefit in faster rendering of particle systems. Particles hidden behind solid objects will not be painted.
Low frequency particle systems will be enabled to be rendered with downsampled resolution without observable loss of quality (i.a. thanks to blur at center of effects and full resolution in the vicinity of solid objects edges). That will enable saving more computational power for improvement of particle system granularity (for example).

!General Information:

Project covers implementation of "GPU gems 3. Chapter 23. High-Speed, Off-Screen Particles" from:
http://http.developer.nvidia.com/GPUGems3/gpugems3_ch23.html .

It is being realized during "Google Summer of Code 2012".

Student assigned to the project: Karol Badowski.
Promotor of the project: Assaf Raman.

Project repository is a fork of OGRE mercurial repository (revision 8d3b960cce3e), aviable at:
https://bitbucket.org/KarolBadowski/ogre-gsoc2012-offscreenparticles .
https://bitbucket.org/KarolBadowski/ogre-gsoc2012-offscreenparticles-tidyupcode (new fork)

Project discussion thread on official OGRE forum:
http://www.ogre3d.org/forums/viewtopic.php?f=13&t=69771 .

Project proposal on Google Summer of Code website:
http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/karolbadowski/1

!Milestones:
{FANCYLIST()}
__preparation before implementation__: - done
__depth acquisition__: - done
__downsampled rendering__: - done
__render target merging & shaders__: - done
__mixed resolution rendering__: - done{FANCYLIST} 


!Tasks for each milestone:

1:
- requirements specification
- UML diagrams
- reading tutorials
- understanding documentation

2:
- two pass depth acquisition
- multi render target depth acquisition
- z-buffer depth acquisition
- alpha channel depth acquisition
- automatisation of depth acquisition technique choice

3:
- dowsampling with variable scale
- point sampling depth renderer
- maximum of depth samples renderer

4:
- alpha-blending
- binary depth test
- soft particles

5:
- edges detection
- stenciling


!Initial proposition of classes diagram:

{IMG(src="http://www.bmm.yoyo.pl/ClassDiagram.JPG")}classes diagram version 1.0{IMG}

!Short description:

__OffScreenParticles__: (perhaps could be a CompositorChain)
{REMARKSBOX(type="information",highlight="y",width="400px")}Class controls appropriate order of operations.
In the constructor, it transparently makes a choice of appropriate subclasses (for example subclass of DepthAcquisitioner with appropriate parameters).
It acquires input data, calls methods of following classes and propagates the output.

Even if initial choice of some used subclasses and their parameters is transparent, there are still getters and setters.{REMARKSBOX}

__DepthAcquisitioner__:
{REMARKSBOX(type="information",highlight="y",width="400px")}Class could be abstract (or even interface), because methods will vary depending on version of used RenderSystem and GPU series.
It would be comfrtable if choice of the subclass was automatic, transparent to the developer. Decision process will need to "ask" classes providing intel about Render System. It also could check whether Alpha channel is beeing used (semi-transparent solid objects).

Single-Pass:
-ZBuff_DepthAcquisitioner: Z-buffer
(provided in directX10)
-MRT_DepthAcquisitioner: Multiple Rendering Target
(if older than directX10 and we do not use MSAA - multisample AntiAliasing)
-Alpha_DepthAcquisitioner: depth saved in channel alfa
(if tranparency is not used and (we do not need MSAA or GPU series different than GeForce6 or GeForce7)) 
Two-Pass:
-TwoPass_DepthAcquisitioner: writing to single rendering target in extra pass
(if we do not use MRT, still have possibility of MSAA even in directx9, still have possibility to use alfa channel)

Class should also provide a method to read acquired depth data transparently, no matter which subclass was used.{REMARKSBOX}

__DepthDownsampler__:
{REMARKSBOX(type="information",highlight="y",width="400px")}Class used when there is need for downsampling of the depth buffer.
It is not obligatory when we use Two-Pass Depth Acquisitioner.
Provides fields and setters allowing to change these fields:
Especially scale of downsampling.

Could be expanded and have a subclass providing downsampling that is not proportional to the powers of 2 or not square. It could provide rectangular fields, that requires additional field. Subclass: RectDepthDownsampler{REMARKSBOX}

__OffScreenRenderer__:
{REMARKSBOX(type="information",highlight="y",width="400px")}Abstract Class. 
Important abstract method: silhouette().
I will provide support at least for two overrides (suggested in the article), that use private methods:

-pointSamplingDepth(): creates halo
-maximumOfDepthSamples(): minor halo artifacts reduced after linear blending
(Game developers can provide own impementations of silhouette() method, for example: Median, Average, depending on derivative, how rapid depth changes, other...){REMARKSBOX}

__RenderTargetMerger__:
{REMARKSBOX(type="information",highlight="y",width="400px")}Abstract Class.
Class merges data from MRT, preparing final output.

Most important abstract method is concatenate().

Class should support the usage of some already implemented PixelShaders and their concatenations, user defined application-dependant PixelShaders.

Method alfaBlending() that uses zShade() abstract method.

I will provide the implementation of overrides for zShade(), using:
-binaryDepthTestShade(): 
-softParticlesShade(): 

If a specified particle effect does not support transparency, we can use only zShade() without alfaBlending().{REMARKSBOX}

__ResolutionMixer__:
{REMARKSBOX(type="information",highlight="y",width="400px")}It is class not mandatory to use. Game developer can define in the constructor of OffScreenParticles class whether to use it or not.

It provides edge detection method...
-detectEdges(): provides Laplace or Sobel edge detection matrix filter

...and second pass of depth acquisition with full resolution
-stencileEdgesPerticularisation(): fraction of pixels repeat the process without downsampling
(acquisition dependant on DepthAcquisitioner subclass, second pass of depth acquisition with full resolution){REMARKSBOX}


!Used strategies:

DEPTH ACQUISITION
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/Capture57.JPG",height="25%",width="25%")}{IMG}
TEXTURE DOWNSAMPLING (depth)
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/Downsample8x8.JPG",height="25%",width="25%")}{IMG}
SELECTIVE RENDERING
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen14.JPG",height="25%",width="25%")}{IMG}
OFF SCREEN RENDERING WITH DEPTH TEST (backbuffer)
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen22.JPG",height="25%",width="25%")}{IMG}
MAXIMUM OF DEPTH DOWNSAMPLING (displayed debug difference between max and min value) 
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen37.JPG",height="25%",width="25%")}{IMG}
EDGES DETECTION (Sobel filter X,Y,Xmirrored,Ymirrored)
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen31.JPG",height="25%",width="25%")}{IMG}
DEPTH TEST USING OFFSCREEN DEPTH MAP
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen50.JPG",height="25%",width="25%")}{IMG}
COPING WITH DiractX / OpenGL transformations
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen48.JPG",height="25%",width="25%")}{IMG}
HANDLING HALO EFFECTS (linear filter + DEPTH texture / MAXOFDEPTH texture)
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen56.JPG",height="25%",width="25%")}{IMG}{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen57.JPG",height="25%",width="25%")}{IMG}
DETECT INTERSECTIONS (depth test, Sobel filter)
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen53.JPG",height="50%",width="50%")}{IMG}
BLENDING OFFSCREEN TARGET PARTICLES (only additive/only blended)
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen41.JPG",height="25%",width="25%")}{IMG}
MIXED TECHNIQUE (my idea for 2-target offscreen rendering, coping with mixed alpha+additive at the same time)
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen61.JPG",height="25%",width="25%")}{IMG}{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen59.JPG",height="25%",width="25%")}{IMG}
DISCARDING EDGES ONLY BETWEEN ( SOLID OBJECTS <-> PARTICLES ) ( Sobel over alpha as a matrix )
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen74.JPG",height="50%",width="50%")}{IMG}{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen77.JPG",height="25%",width="25%")}{IMG}{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen80.JPG",height="25%",width="25%")}{IMG}
MIXED RESOLUTION
{IMG(src="http://www.bmm.yoyo.pl/everyday_screenshots/miniscreen83.JPG",height="75%",width="75%")}{IMG}



!Schedule:
"Off-Screen Particles" project timeline:

__Before May 21__ - Reading documentation of OGRE engine, especially particle systems

__Before Jun 4__ - communication with project supervisor about eventual difficulties or questions in understending of already implemented classes 

__Before Jun 16__ - continuation of Ogre Tutorials, Exam session on one university

__Jun 17__ - analysing OGRE Example Applications in search of useful solutions

__Jun 18-19__ - creating bacic scene for presentation and future tests

__Jun 20__ - defining animation for scene to distribute particles, creating fumes particle system

__Jun 21-25__ - reading tutorials and articles on depth testing + OGRE tutorials, EXam session on second university, tutorials about CG, HLSL, OGRE tutorials about shaders

__Jun 26__ - Exam DNA sequentioning, starting attemts to write depth acquisition

__Jun 27__ - trying to implement depth acquisition, examining code of "soft particles" effect implementation by  ahmedismaiel

__Jun 28-29__ - still trying to write depth acquisition, examining all forum topics about that.

__Jun 30__ - hosting family

__July 1__ - implementing passing achieved already depth texture between compositors in compositor Chain, 

__July 1-4__ - eliminating bugs in code (difference of displaying depth of semi-transparent objects in DirectX3D and OpenGL), downsampling is based on SSAO example - does not inherit texture, analysing possibility of handling semi-transparent objects together with particles.

__July 5-6__ - using own SolidMaterialsBuffer, entities are given the material manually. Achieved concatenation of compositors into chain. Deciding that there are no other solutions to preserve colour material, than using listeners on application level. Passing mocked texture instead created from depth in R,G,B,A channels. Using Post filters in compositor chain, reading script tutorials. fuguring out how to use compositors selectively? - only for solid materials. Modification of particle materials

__July 7__ - Figured out difference in execution of material between DirectX9 and OpenGL. Passing depth from vertex to pixel shader instead of calculating in pixel shader. Then texture is passed to next compositor that displays it. Using PF_FLOAT32_R. for passing depth.

__July 8__ - Writing downsampling in compositor, editing view from prepared movies to show summary, universal shader for downsamplingfor rectangular downsample,debugging of downsampler 

__July 9__ - [http://www.ogre3d.org/forums/viewtopic.php?f=13&t=69771&start=75|writing application-level proof of concept] basic version of off-screen particles, using backbuffer for depth testing. creting and displayin viewports with view of solid objects only, particles only, writing concatenation. Handling recognition of render targets (setting them to invisibility before rendering and visibility after, to avoid infinite mirrors.) screen target for downsampling, handling multiple miniscreens. rendering separate groups of objects with masks connected to viewports

__July 10__ - turned off antialiasing. Separated display of solid objects and particles in viewports, compare of depth of downsampled texture (done in backbuffer) and particles. contorling autoupdate level of render targets. Handling render targer in post/preRenderTargetListener, mocking setting of transparent colour. 

__July 11__ - looking for solution to handle colour transparency differently. Start implementing compositor for joining textures

__July 12__ - calculations for off-screen alpha blening algorythm (found mistake in N-vidia article ), continuation of writing concatenation compositor

__July 11-14__ - reading tutorials, figuring out how to writr, solved dynamic swap of texture

__July 14__ - finished texture merging compositor for proof of concept - finally visible halo effects (downsampled texture put on solid object background), taking part in charity event

__July 15-17__ - comming back to compositor version from 8 July. looking for reason of difference in Dx9 and OpengL display of depth (in one particles / objects without material SolidBuffer are omitted, in second one they generate white noise simmilar texture), learning ogre Wiki examples

__July 18-20__ - continue transformation to compositor chain, reading examples of post-filters, managed to implement downsampling and maxof depth downsampling, display downsampled depth texture, 

__July 21__ - saturday, spend with friends

__July 22-23__ - continue transformation to compositor chain, , Writing universal version of maxofdepth downsampling (controlling referenced texture address based on downsample level)- passed as size of pixel as automatic named parameter for transformations

__July 24__ - trying again to figure out how to preserve original material with rendering depth, reading forums and tutorials

__July 25-26__ - material of DownsampledDepth rendering stopped defining material by calling other comositor from chain. Transferred to SolidMaterialsBuffer compositor - everything works faster. Finding and removing mistakes in configuration. Slowly starting to transfer ready compositors into first one to merge them.

__July 26-27__ - finally figured out how first_render_queue and last_render_queue attributes work in selective rendering - rebuild earlier method of dividing (maskqes were used before). continue trying to figure out how to render one object with 2 materials in a comppositor: original colours generating pass and depthh texture generating one, how to merge them. 

__July 24-28__ - trying to ask on forum and find in forums, tutorials, other applications how to create new compositor pass inheriting all previous operations of original material (to preserve amount of textures, passess, depth maps, normmal maps) and just extending it without replacing (to generate scene depth map in the same pass). Did not find the answer yet. Seems like complicated issue (it is not achieved in SSAO example either, on which I am basing construction of compositor chain)

__till July 28__ - uploading archives of channges in scripts and application, to repository (forgot to upload older bakupped archives earlier). Figured out that program stopped working in DirectX9 - works only in OpenGL since some time - pproblem not strictly defined in *.log file. Colour texture is still mocked, not original, but one generated from depth texture. Particles red channel is still displayed (in their case material is different tha SolidMAterialBuffer - red channel of their colour is taken instead of "red channel only" depth texture.) More details of current state and changes, here: http://www.ogre3d.org/forums/viewtopic.php?f=13&t=69771&start=150#p464984
trying to merge compositor to concatenate rextures that way it would use difference in RenderQueueID
and here: http://www.ogre3d.org/forums/viewtopic.php?f=13&t=69771&start=150#p464999

__July 29__ - continue transferring shaders to recently created "OSP_AllInOne" compositor - trying in every trnsfer to expand shaders line by line from working version to previously created one, to see where is difference, mistake between OpenGL and DirectX work of the same script.  

__July 29-31__ - Still trying to figure out how to universally inherit original tactic of rendering texture and change only little fragment of shader / eventually whether transfer some of original definitionns of passes/textures from original material to new one, to preserve at least colour and alpha value. Ask on forum about that, trying to find the solution in examples simpler than DefferedShading, forums, code of applications. It is a bottle-neck problem. Reading script tutorial averyday to find solution.

__August 1-4__ - Still did not find the solution, asked on forum again, searched in documentation, script tutorials, it seems noone knows the advice how to solve that. Without this step next passes can't be displayed the original way (own vertex program an fragment program has to be used, so original material is replaced manually with material which uses these shaders).

__August 4__ - Still no advice on that problem found. Problem precisely described on forum. Splitting task to parallely continue searching for answer and complete missing passes of compositor to finish them before deadline (they will have to work on mocked data, not on original colour texture).

__August 5-7__ - Work without looking on project thread, hurrying up to complete missing compositor passes, shaders. Paralelly looking for solution of problem with inheriting original material.