History: SoC2009 Compositor

Source of version: 9 (current)

!Summer of Code 2009: Improving Compositor Framework
__Student:__ Noam Gat

__Mentor:__ Assaf Raman

__ Location:__ branches/soc09-compositor

__ Status:__ Merged into trunk, is in main Ogre branch from version 1.7

!!Motivation
Ogre's Post Processing (Compositors) framework, while being quite powerful, is still lacking some features. Numerous users have said that they found themselves implementing their own custom post processing solution because they couldn't do what they wanted with the compositor framework.
This project's aim is to address this problem, adding options for more advanced chain setups and more flexible composition options, enabling advanced rendering techniques like deferred rendering to be easier to set up, more flexible and more reusable.

!!Project Proposal
Ogre's Post Processing (Compositors) framework, while being quite powerful, is still lacking some features. Numerous users have said that they found themselves implementing their own custom post processing solution because they couldn't do what they wanted with the compositor framework.

Before I start suggesting the changes themselves, I will present two "wishful use cases" that could, theoretically, be solved by a post processing framework:

* Deffered shading post processing support. Consider this case : someone set up a deferred shading system using the compositor framework. Afterwards, they would like to add SSAO to the scene, like [http://www.ogre3d.org/forums/viewtopic.php?f=11&t=47927|nullsquared's demo]. SSAO requires the depth and normal passes, so the compositor has scene passes that calculate these themselves. But the deferred shading system already created those textures! We would like to be able to use them from the next compositors in the chain.
* Depth based effects. Even when not under deferred shading, many rendering techniques require simliar full-scene renders (like depth). We would like to define a "utility compositor" that does the depth pass of the scene, and then the other compositors would be able to use its output.

!!Gameplan
The features that will be added to the compositor framework, desgined to make it more modular are :

* [[Done] Inter-compositor communication : A compositor should have access to the other compositors' local textures. (It has been suggested to be able to use named texture units, but that is not chain-specific).
* [[Done, Validation only] Auto ordering of compositors : Slightly related to the previous addition, if a compositor depends on another one, they should be ordered accordingly. The chain can be auto-ordered to make sure that each compositor runs after its dependencies (can also find cyclic dependencies). UPDATE : I think that I won't auto-order compositors, but throw an exception if an illegal chain is attempted, as the coder is supposed to understand the chain that they are setting up, and there are many corner cases to figure out, not worth it IMO. I think it should be validation only.
* [[Skipped] Scriptable chains : A compositor chain should be configurable by script, rather than by hard coding it. Chains will be a bit longer (since utility compositors will be introduced), and I think its healthy to separate the chain definition from the code. (Applying the same chain to two viewports etc).
* [[Done] Allow auto-coupling of compositors and CompositorListener code - If someone created a compositor that needs a listener to set up matrices (example use case : nullsquared's SSAO), we would like to be able to automatically bind the listener whenever the compositor is added to the viewport, without code interference. I have already done something simliar with the [http://www.ogre3d.org/wiki/index.php/CompositorLogic_Framework|CompositorLogic framework], but if we are changing the way compositors work, it shouldn't be an external package. Maybe part of the compositor script will be defining a binded plugin.
* [[Done] Allow none-competing material scheme handlers - Some compositors need to render the scene differently. (This is already supported using schemes). Not all materials have that scheme defined, so the compositor can use code to prepare the correct material. Currently, the MaterialManager defines a MaterialListener class that can do just that (handleSchemeNotFound callback and work from there). However, there can currently only be one listener. So, if two different compositors render the scene with different schemes, they both need to be the single MaterialManager listener. In order to solve this, MaterialManager will be changed to allow single scheme binding, so that each compositor listener will be able to receive its relevant handleSchemeNotFound calls without competing with the others.
* [[Done] Allow custom target passes. This will allow rendering operations that arent scene/quad renders to take place in certain places. An example for this would be sphere/cone rendering for lights in the deferred pipeline. See target design section for more details.
* [[Done] Allow pausing frame rendering mid frame. This allows stopping frame rendering mid frame, rendering (for example, shadows) to an RTT, then resuming the rendering of the original frame, and continuing from there. This will allow a single texture to be reused for all shadows.

!!Target Design
This section will contain pseudo-code and pseudo-scripts that are not yet supported, but illustrate how the final API will look like.

!!!Inter-compositor communication.
The example will be deferred shading + SSAO. The deferred shading pass will declare the available textures and the SSAO compositor will use them.

DeferredPass.compositor would contain:
{CODE(wrap="1")}//Default is local scope, also global_scope is possible.
texture output_tex target_width target_height PF_A8R8G8B8 PF_A8R8G8B8 PF_A8R8G8B8 PF_A8R8G8B8 chain_scope {CODE}

SSAO.compositor would contain :
{CODE(wrap="1")}//syntax : texture_ref <local_name> <src_compositor> <src_name> 
texture_ref DeferredMRT DeferredPass output_tex 

target rt
{
    // Start with clear texture
    input none
    // SSAO pass
    pass render_quad
    {
        // Renders a fullscreen quad with a material
        material Ogre/SSAOQuad
        // The MRT target containing the normals
        input 0 DeferredMRT 0
        // The MRT target containing the depth
        input 1 DeferredMRT 1
    }
}{CODE}

!!!!Pooling
How will pooling (currently called 'sharing') be handled in these scopes?

Local scope (current texture mode, will be default) will be handled like it currently is. (As aggressive as possible)

Chain scope will allow viewports with parallel compositors to use the same texture instances. We can't be more aggressive than that, because the next compositors in the chain might be able to use it.

Global scope will not support the sharing flag, as there is already exactly one texture per compositor, which the user might want to have access to at any given time.


!!!!Validation
How will validation work?
I decided to implement validation only (rather than fixing) in this project, mainly for ease of implementation, but the user doesn't want his compositors re-ordered without knowing about it.

The main thing that needs checking is that when a compositor with a compositor_texture_ref directive is added to a chain, the compositor that it is referencing (and the texture inside) are defined and are before the compositor in the chain. If this is not the case, an exception will be thrown.

Global scoping can not be checked, as the user may use any strategy that they want as to when/how to run this compositor and update the texture. The texture itself will be initialized the first time it is about to be used. target_width/target_height params will either be illegal for this texture type, or initialized with the first viewport size that renders to it.


!!!Compositor Logics


Auto-coupling of compositors and compositor logics in scripts :
Gaussian Blur.compositor (currently has a compositor listener set up in code) :
{CODE(wrap="1")}// Gaussian blur effect
compositor "Gaussian Blur"
{
    technique
    {
        //This will automatically link the relevant CompositorLogic when the compositor is initialized
        //Theoretically, this can be expanded to have specific params in curly brackets, but overkill for now.
        compositor_logic GaussianBlurLogic

        // Temporary textures
        texture rt0 target_width target_height PF_A8R8G8B8 shared
        texture rt1 target_width target_height PF_A8R8G8B8 shared

        //...

        target rt1
        {
            // Blur horizontally
            input none
            pass render_quad
            {
                material Ogre/Compositor/HDR/GaussianBloom
                input 0 rt0
                //The compositor logic will listen to the identifer to know when to modify things
                identifier 700
            }
        }
        
        //...

    }
}{CODE}

* CompositorLogic interface. Still undecided on this one. The current debate : will it follow the RenderSystem design pattern (ie, a managing class will have a registerCompositorLogic API call, and plugins or application code will make the calls to register the CompositorLogic factories, or will it follow the Resource design pattern, meaning that CompositorLogic will be a new resource type. Problem with the resource pattern - how do we separate between release and debug without filename naming conventions?

!!!None-competing scheme handlers
This one is straightforward. We will change :
{CODE(wrap="1")}/** Add a listener to handle material events. */
virtual void addListener(Listener* l);

/** Remove a listener handling material events. */
virtual void removeListener(Listener* l);{CODE}

to something like:

{CODE(wrap="1")}/** 
    Add a listener to handle material events. 
    If schemeName is supplied, the listener will only receive events for that certain scheme.
*/
virtual void addListener(Listener* l, const Ogre::String& schemeName = ALL_SCHEMES);

/** 
    Remove a listener handling material events. 
    If the listener was added with a custom scheme name, it needs to be supplied here as well.
*/
virtual void removeListener(Listener* l, const Ogre::String& schemeName = ALL_SCHEMES);{CODE}

Thus enabling different scheme handlers to handle specific schemes without being aware of one another. Compositor Logics are likely to register such scheme handlers for the scheme that the compositors render the scene with.

!!!Custom passes
Another feature that is required is the ability to have custom compositor passes. Instead of rendering the scene (render_scene) or a full screen quad (render_quad), we need to be able to render custom geometry (for example, sphere for point lights, cones for spotlights). This will be handled in a new 'render_custom' directive, which will by default do nothing. However, it will give the binded compositor_logic code an option to render its custom geometry. The connection will be made by named custom renderers (which will be illustrated in the script).

The scripts will look like this :
{CODE(wrap="1")}target_output
{
    input none
    // lighting passes
    pass render_custom LightGeometry
    {
        input 0 gbuffer 0
        input 1 gbuffer 1
    }
}{CODE}

!!Schedule
The project will consist of many stages. Each part is designed not only to help the final goal of the project, but to be a self contained building block that Ogre users will be able to use. The final product will use them all, but like any infrastructure, they need to be modular.

__1) [[Done] (May 23 - June 5)Inter-compositor communication.__ Compositors will be able to define textures as publicly available to other compositors to use, and they will also able to reference textures that other compositors shared. This will include either sanity checking or auto fixing of chains (or both), so that only legal combinations are created.

__Testcase 1) [[Done, no SSAO] (June 5-10) Separation of the Deferred Shading compositors to two stages__ - one that generates the GBuffer, and one that uses it to create the final picture. Add another optional compositor that uses the GBuffer to create an SSAO buffer and apply that to the scene.

__2) [[Done] (June 11-18) Add auto-coupling of compositors and their related code in scripts__ - some compositors require code to setup their rendering properly. We want the compositors to be able to define a plugin name (or similar) that needs to be loaded for them to operate.

__3) [[Done] (June 18- July 25) Modify the MaterialManager::Listener interface__ to allow none-competing scheme handlers, so that different handlers for different schemes will be able to co-exist and receive their respective callbacks without using a 3rd manager.

__Testcase 2+3) [[In redesign] (June 25 - July 1) Create a depth-based composition technique__ that automatically generates the depth rendering setup through the "Depth" material scheme. This handler will be loaded automatically when the compositor's binded plugin is loaded.

__4) [[Skipped] Scriptable chains (July 2 - July 8):__ It should be possible to define compositor chains in scripts. This will make it easier to apply predefined sets of compositors to many viewports.

__5) [[Done] Add a render_custom composition pass (July 8 - July 29)__ (parallel to render_scene, render_quad etc). This will allow the compositor to render its own sequence per-light. Design is not complete yet, so extra time is added to decide on the details of this one.

__Testcase 5) [[Done] (July 29 - August 10) Add texture shadow support to the Deferred Shading demo.__ The shadow maps will probably (design is not yet final) be generated during the render_lights sequence (or they will be able to be pulled in somehow), and then casted on the scene.

__Uber-testcase) [[In Progress] (As much time as I have left) Complete the refactoring of the Deferred Shading demo__ by making it completely script based. The demo will just load the scene and apply a compositor chain script to it, and the rest of the job will be done by the compositor & associated plugin.

!Future work
The project has ended, and helpful features were added making the compositor framework more powerful than it was.
There were many design decisions during the process, and many discussions concerning the compositor framework. Many good ideas were raised but not addressed, mainly because of time constraints. This section will list those ideas, for future reference.


Some of the ideas are copy-pastes from discussions on the forums, but can still be understood. The full discussions behind them can be found in the forum thread (see links section).

!!Per-light compositor iteration
Compositors need to be able to render certain stages many times, usually once for each light. The common example would be the shadow map rendering stage in deferred shading.
So, we will take the 'iteration' directive from material passes and use the same one here. The new directives will be (all at the compositor pass level)

# iteration : This will be the same as the one from material passes - once_per_light to have one per light, including light type support etc.
# render_shadow_textures : If selected, this will trigger re-rendering of the relevant shadow textures. This will allow us to use Ogre's current shadowing framework for rendering shadows as part of the compositor pipeline.

Some additional directives might be needed, such as
# max_lights : This directive (to render_scene passes) tells the scene to disregard lights besides specific ones. For example, when rendering the gbuffer part of a scene with deferred shading, only the directional light should be taken into account.
# start_light : This directive will tell the compositor to start the light iteration from the non-first light index. Again, example would be skipping the directional light for the later passes.



I still don't know if these two are needed, or maybe they can be merged into the iteration directive.

!!Referencing compositor textures in materials
In some techniques, individual objects require the 'full scene preprocessing' textures, for example [http://graphics.cs.uiuc.edu/~kircher/inferred/inferred_lighting_paper.pdf|inferred lighting]. This means that a new content_type will be introduced (current ones are regular and shadow) for referencing compositor textures.
The referencing would either be done by name (meaning the material will be aware of the name of the texture its referencing) or by index (meaning this will only be possible if the material is rendered during a render_scene directive).

The latter would look like this in the compositor
{CODE(wrap="1")}compositor InferredLighting/MaterialPass
{
  texture_ref LBufferOutput InferredLighting/LBuffer mrt_output
  
  target_output
  {
    material_scheme InferredMaterial
    render_scene
    {
      input 0 LBufferOutput 0
    }
  }
}{CODE}

And the materials would look like
{CODE(wrap="1")}material InferredLighting/SomeObject
{
  technique
  {
     scheme InferredMaterial
     pass
     {
       ...
       texture
       {
          content_type compositor
       }
     }
  }
}{CODE}

!!Trigerring composition techniques for individual objects
The other part of the compositor<->material bridge is to be able to trigger compositors from individual objects. Some techniques (hair, fur etc) often have an RTT pass in the beginning and use the result in their next passes. This is what the compositor framework does, but for scenes instead of objects.



Many decisions need to be made, for example :
* Reuse textures to save memory or duplicate textures to pre-render all RTTs before the scene starts and reduce render target swapping
* Will these be standard compositors? Will it be possible to pass params to the compositor from the material?



But this feature could make quite a few techniques easier to setup in script.

!!Moving the material_scheme directive to pass scope
The motive behind this is to allow different render_scene directives to render using different material schemes.



In my opinion, the "material_scheme" directive belongs in the pass scope, and not in the target scope. The directive is usually applied to render_scene passes, and if someone puts render_scene and render_quad directives in the same target operation, do they expect that the quad's material will be affected by the material scheme? I don't think so.
The downsides of the solution are that they break backwards compatibility and might cause script duplication (if you really want the same custom scheme for multiple passes).
A possibility to solve these downsides is to allow the directive at both scopes, but I really don't like that option. I think that material_scheme really belongs in the pass scope.



This isn't currently possible either. Technique resolution (which is where missing techniques are handled) happens during updateRenderQueue, which means that you cannot resolve techniques based on the scheme that will be active during their render queue.
I'm going to have to find a different solution, or change more stuff in Ogre. Ideas?

!Further Reading
!!Deliverables
((Deferred Shading|The deferred shading demo that complements the work done in this project))

!!Relevant links
[http://www.ogre3d.org/forums/viewtopic.php?f=13&t=48709|Forum thread for this project]

[http://www.guerrilla-games.com/publications/dr_kz2_rsx_dev07.pdf|KillZone 2 Deferred Shading overview]

[http://www.gamedev.net/community/forums/mod/journal/journal.asp?jn=263350|Journal of Ysaneya - another deferred shading implementation overview]

[http://www.ogre3d.org/forums/viewtopic.php?f=11&t=49505|THP Project - deferred shading project using ogre's current pipeline]
History: SoC2009 Compositor

Source of version: 9 (current)

Search by Tags

Latest Changes

Search

Online Users
