SoC2006 RmExporter         RmOgreExporter [v2], FxOgreExporter - GSoC 2006

Summer of Code 2006: RmOgreExporter (v2), FxOgreExporter

Student: Vladimir Sukhoy (buddy)

Mentor: Michael Reimpell (reimpell)

Status: Ongoing development in ogreaddons/RmOgreExporter

The project has two major parts:

  • Extend and reimplement existing RenderMonkey -> Ogre material exporter (RmOgreExporter (v2)).
  • Create separate .fx -> ogre material converter (the initial proposal was modified on OGRE community request, original plan was to create an exporter tool for FXComposer) (FxOgreExporter).

RmOgreExporter (v2)

Existing RmOgreExporter codebase will be used to enhance the tool to a newer level of usability. A better level of integration between existing major
shader/material designer software and OGRE engine is a major goal.

Latest version installer:
http://sukhoy.public.iastate.edu/soc2006/RmOgreInstaller-202-rm162-py24.exe
this is RM1.62 Python 2.4 compatible version, as you can probably guess from the filename.
You will need to install Python and pywin32 separately!.

Features

Major

  • [pylogic] rewrite exporter logic in Python and bundle interpreter code with the software, so that users will be able to contribute and tweak the exporter for their tasks (this one is really time-consuming, since rmsdk python wrapper must be created).
  • [import] add import capabilities to the plugin, so that people can edit their existing OGRE materials in shader designers (RenderMonkey) and then export the result back into OGRE.
  • [scripting] add scripting capabilities for "batch" export.
  • [previewer] add optional preview capability to the tool.
  • [doc] document the code being developed and come up with "User's Manual".

Minor

  • [rmogl] support OpenGL branch of RenderMonkey (it is not supported in RmOgreExporter v1).
  • [exflex] - more flexibility to export scripts (different directories, different material names...)
  • [persistence] - make export/import options persistent in the workspace file.
  • [zipbackend] - implement zip backend for export/import.
  • [improv] - Improve tool intelligence to deal with inherent differences in shader designer's material model and OGRE material framework.
  • [idefiles] - More makefiles/project files/solutions for more IDEs (e.g. MSVS .NET 2005, and MinGW support)
  • [meshimport] - Import OGRE's mesh geometry files (.mesh) into Shader Designers.
  • [varsup] - More variables supported by OGRE: (camera in world space, ambient light color, ...)
  • [installer] - Package the tool and provide installer.

What is done

  • rmsdk was wrapped into python (via boost::python). I estimate "integration" level of rmsdk (e.g. what you can do in Python over what you can do in C++) at 90-95% for "Core" functionality (some minor stuff like XML manager or Undo/Redo stuff is missing). That means that one can freely manipulate workspace nodes, use appliсation singleton etc. It seems that rmsdk wrapper may (and probably should) be released as a standalone library, as it may be useful for anyone who wants to develop python plugins for RenderMonkey. As the amount of source in rmsdk was hmm... serious, I had to develop some kind of toolset on top of boost::python to simplify a lot of things. But still there are many kilobytes involved, no magic here :-(.
  • As RenderMonkey uses .dll binary interface to talk with plugins, a gateway plugin was required that "translates" rendermonkey calls to python module and invokes appropriate python functionality. This gateway plugin is essential to use rmsdk wrapper.
  • Native Ogre material serializer framework is used to store materials. There is one minor issue though: there is no easy way to store "useful" comments inside the material script with this approach, however it is still preferred to doing everything manually as it was in RmOgreExporter (v1).
  • pyogre folks forgot to wrap MaterialSerializer class, so a tiny wrapper "ogre_mini" was created via boost::python. The wrapper exports some functionality to support native serializing from python. Nevertheless I plan to use their code for preview.
  • Python source which implements (partially, for now) exporting plugin functionality similar to RmOgreExporter (v1), and uses rmsdk wrapper is developed.


In version 2.01

  • Compatibility with RenderMonkey 1.62, thanks to sebarnolds!
  • Ability to export glsl shaders (preview non-functional), again thanks to sebarnolds!



FxOgreExporter

Brief

The work seems to be mostly complete, so the version number is now 1.0 :-). The name of the executable is "fxogretool".

Note that the tool is no silver bullet - so expect some polishing to be done on the output. Sometimes matrices have to be transposed, normals flipped etc. It usually depends on what you are trying to visualize with exported material and what was expected to be visualized by FX file designer. Also do not expect it to handle everything you can do with FX _semantically_. But usually it produces output that is close.

By the way, usually .fx file can be fed to HLSL compiler (and to OGRE) with no problems, so no need to extract HLSL pieces from .fx code. FX is actually HLSL or vice versa).

Current version (in addons/RmOgreExporter see fxogretool project) supports following:

  • Direct3D 9 .fx BNF parser (tested against most of examples in DirectX9 SDK directory - no parse errors, total of 101 file tested).
  • Can build parse and abstract syntax trees from .fx script.
  • Can dump these trees to xml, so that they can be used for debugging or for any other purpose (syntax representation of .fx file in XML can be useful for anyone messing with .fx files). Parse tree is more appropriate for something comprehensive, AST is smaller, however often more convenient to analyze.
  • The source code has an FX grammar parser and lexical analyzer grammars exposed in boost::spirit format (C++ code which looks like (and is used to build) BNF grammar).
  • Regular expressions are used to setup automatic variables (which can be supplied via command line and alternatively via config file).
  • Variable definitions from FX file are used to setup variables in material script.
  • Techniques, passes and states from FX file are translated to OGRE techniques, passes and states (if possible).



Detailed

The executable depends on OGRE (material serializer stuff), boost (spirit, regex etc.). There is no dependency on DirectX - everything related to FX is homegrown.
The basic usage:

fxogretool source.fx result.material

A number of options are available. To request the list of them use

fxogretool --help

Important options are:

--fx=file.fx

Specifies fx file to parse (Uses first argument if not supplied explicitly, like above).

--mat=file.material

Specifies material file to create (Uses second argument if not supplied explicitly).

--ptxml=file.xml

Store parse tree in XML.

--astxml=file.xml

Store AST tree in XML.

--autovarfile=config.txt

Use config.txt to get regular expressions for automatic variables. regexps may be also supplied via command line. The config file is expected to look like (like anything that ends with "World" will be param_named_auto name world_matrix etc.)

world_matrix=^.+World$  
 view_matrix=^.+View$
 ....

Problems


  • Tree building is relatively slow (several seconds, up to 10-15 on my machine for fat enough .fx file).
  • Relatively slow compilation.

Future

  • To keep ogre from complaining (it is DirectX parser who complains, actually) the tool should walk over respective vertex/pixel shader code and eliminate unused variable references.
  • Support DirectX 10 FX files (they actually extended the grammar and semantics quite a bit).
  • Support #define
  • Support #include (although OGRE does not support those).
  • Much of the above actualy requires some sort of semantics to be implemented. Nice semantics is time consuming and is almost equivalent to the actual compiler (to some extent). But in the end given semantics support it is actually possible to map HLSL to GLSL or Cg, as those are loosely close. On the contrary there is no good docs (at least I could not find those) on HLSL semantics, no standard document or anything close. Even grammar itself is given pretty much AS IS. So no cakewalk there.


See the builtin help on command line options.