Skip to main content

History: All-purpose script parser

View published page Collapse Into Edit Sessions

Preview of version: 8

Custom Script Parser?

In many games, object data is stored in external files for easy adjustment and management. For example, in a racing game, each car might have it's own configuration file that specifies it's top speed, turning radius, etc. Many programmers use XML or a binary format to store this information.

If you've ever written a .material script for Ogre, you probably know that Ogre's C-style script syntax is very intuitive and compact. Now you can store any kind of data you want in Ogre C-style scripts, and and quickly and easily load them with this parser.

Like Ogre's .material scripts, this will automatically parse all your scripts when a resource group is loaded. Since all scripts are pre-parsed, the system can access scripts not by filename, but by script name; in other words, you can include multiple scripts per file and still access them individually.

This script parser was designed to be very lightweight and fast; it's only a few KBs of code, and can parse around 10 MB per second on an average PC (if your average script is ~1 KB, you can load around 10,000 scripts in one second).

Using the Parser

Using this script parser is very easy. Simply initialize the system by creating a ConfigScriptLoader instance. You don't even have to keep a pointer to the instance because ConfigScriptLoader is a singleton class:

Copy to clipboard
new ConfigScriptLoader();

Make sure you create a ConfigScriptLoader class before loading any resource groups, otherwise it will miss them and your scripts won't get loaded.

Now, after loading your resource groups, the ConfigScriptLoader will have automatically parsed your scripts for you! All you need to do now is request the desired script, and process it's contents. Getting a script is simple:

Copy to clipboard
ConfigNode *rootNode;
 rootNode = ConfigScriptLoader::getSingleton().getConfigScript("entity", "Crate");

ConfigScriptLoader::getConfigScript() will retrieve the root node of the specified script, which you can use to access any of it's data.

You may notice that the getConfigScript() requires two names; one for the script type, and another for the actual name. For example, the above code would load this script:

Copy to clipboard
entity Crate{
   position 100 50 200
   rotation 0 0 0
   scale 1 1 1
 }

You may be wondering what file this script is coming from. The fact is, it doesn't matter! As long as your scripts have a .object extension (and this can be changed if you modify line 22 of ConfigScript.cpp), their filename is completely irrelevant. Since you access scripts by script name rather than file name, it makes no difference where the script is located. This is a big advantage since this allows you to organize your scripts any way you want without worrying about your game not being able to find it.

Now that you have a pointer to the root ConfigNode, you can access any of the data it contains. For example, to access the Y position coordinate (50) given in the above script example, you first find the "position" node, then access it's 2nd value (as you can see it contains 3 values in all: 100, 50, and 200).

Copy to clipboard
float positionY = rootNode->findChild("position")->getValueF(1);

1 is used to access the 2nd position value since std::vector is 0-based (like any other C array). getValueF() is a variation of getValue() that automatically converts the value to a float. Other variations are included for doubles, ints, etc.

Note: When you shut down your application, don't forget to delete the ConfigScriptLoader instance - Ogre won't do this for you, so failing to do so will result in a memory leak. Since it's a singleton class, you can always delete it using this code:

Copy to clipboard
delete ConfigScriptLoader::getSingletonPtr();

Source Files



ConfigScript.h

Copy to clipboard
//This code is public domain - you can do whatever you want with it
//Original author: John Judnich
#ifndef _CONFIGSCRIPT_H__
#define _CONFIGSCRIPT_H__

#include <OgreScriptLoader.h>
#include <OgreStringConverter.h>
#include <hash_map>
#include <vector>

class ConfigNode;

class ConfigScriptLoader: public Ogre::ScriptLoader
{
public:
    ConfigScriptLoader();
    ~ConfigScriptLoader();

    inline static ConfigScriptLoader &getSingleton() { return *singletonPtr; }
    inline static ConfigScriptLoader *getSingletonPtr() { return singletonPtr; }

    Ogre::Real getLoadingOrder() const;
    const Ogre::StringVector &getScriptPatterns() const;

    ConfigNode *getConfigScript(const Ogre::String &type, const Ogre::String &name);

    void parseScript(Ogre::DataStreamPtr &stream, const Ogre::String &groupName);
    
private:
    static ConfigScriptLoader *singletonPtr;

    Ogre::Real mLoadOrder;
    Ogre::StringVector mScriptPatterns;

    stdext::hash_map<Ogre::String, ConfigNode*> scriptList;
    
    //Parsing
    char *parseBuff, *parseBuffEnd, *buffPtr;
    size_t parseBuffLen;

    enum Token
    {
        TOKEN_Text,
        TOKEN_NewLine,
        TOKEN_OpenBrace,
        TOKEN_CloseBrace,
        TOKEN_EOF,
    };

    Token tok, lastTok;
    Ogre::String tokVal, lastTokVal;
    char *lastTokPos;

    void _parseNodes(ConfigNode *parent);
    void _nextToken();
    void _prevToken();
};

class ConfigNode
{
public:
    ConfigNode(ConfigNode *parent, const Ogre::String &name = "untitled");
    ~ConfigNode();

    inline void setName(const Ogre::String &name)
    {
        this->name = name;
    }

    inline Ogre::String &getName()
    {
        return name;
    }

    inline void addValue(const Ogre::String &value)
    {
        values.push_back(value);
    }

    inline void clearValues()
    {
        values.clear();
    }

    inline std::vector<Ogre::String> &getValues()
    {
        return values;
    }

    inline const Ogre::String &getValue(unsigned int index = 0)
    {
        assert(index < values.size());
        return values[index];
    }

    inline float getValueF(unsigned int index = 0)
    {
        assert(index < values.size());
        return Ogre::StringConverter::parseReal(values[index]);
    }

    inline double getValueD(unsigned int index = 0)
    {
        assert(index < values.size());

        std::istringstream str(values[index]);
        double ret = 0;
        str >> ret;
        return ret;
    }

    inline int getValueI(unsigned int index = 0)
    {
        assert(index < values.size());
        return Ogre::StringConverter::parseInt(values[index]);
    }
    
    ConfigNode *addChild(const Ogre::String &name = "untitled", bool replaceExisting = false);
    ConfigNode *findChild(const Ogre::String &name, bool recursive = false);

    inline std::vector<ConfigNode*> &getChildren()
    {
        return children;
    }

    inline ConfigNode *getChild(unsigned int index = 0)
    {
        assert(index < children.size());
        return children[index];
    }
    
    void setParent(ConfigNode *newParent);

    inline ConfigNode *getParent()
    {
        return parent;
    }

private:
    Ogre::String name;
    std::vector<Ogre::String> values;
    std::vector<ConfigNode*> children;
    ConfigNode *parent;

    int lastChildFound;  //The last child node's index found with a call to findChild()

    std::vector<ConfigNode*>::iterator _iter;
    bool _removeSelf;
};

#endif


ConfigScript.cpp

Copy to clipboard
#include "ConfigScript.h"
#include "Exception.h"

#include <OgreScriptLoader.h>
#include <OgreScriptLoader.h>
#include <OgreResourceGroupManager.h>
using namespace Ogre;

#include <vector>
#include <hash_map>
using namespace std;
using namespace stdext;

ConfigScriptLoader *ConfigScriptLoader::singletonPtr = NULL;

ConfigScriptLoader::ConfigScriptLoader()
{
    //Init singleton
    if (singletonPtr)
        EXCEPTION("Multiple ConfigScriptManager objects are not allowed", "ConfigScriptManager::ConfigScriptManager()");
    singletonPtr = this;

    //Register as a ScriptLoader
    mLoadOrder = 100.0f;
    mScriptPatterns.push_back("*.object");
    ResourceGroupManager::getSingleton()._registerScriptLoader(this);
}

ConfigScriptLoader::~ConfigScriptLoader()
{
    singletonPtr = NULL;

    //Delete all scripts
    stdext::hash_map<String, ConfigNode*>::iterator i;
    for (i = scriptList.begin(); i != scriptList.end(); i++){
        delete i->second;
    }
    scriptList.clear();

    //Unregister with resource group manager
    if (ResourceGroupManager::getSingletonPtr())
        ResourceGroupManager::getSingleton()._unregisterScriptLoader(this);
}

Real ConfigScriptLoader::getLoadingOrder() const
{
    return mLoadOrder;
}

const StringVector &ConfigScriptLoader::getScriptPatterns() const
{
    return mScriptPatterns;
}

ConfigNode *ConfigScriptLoader::getConfigScript(const String &type, const String &name)
{
    stdext::hash_map<String, ConfigNode*>::iterator i;

    String key = type + ' ' + name;
    i = scriptList.find(key);

    //If found..
    if (i != scriptList.end())
        return i->second;
    else
        return NULL;
}

void ConfigScriptLoader::parseScript(DataStreamPtr &stream, const String &groupName)
{
    //Copy the entire file into a buffer for fast access
    parseBuffLen = stream->size();
    parseBuff = new char[parseBuffLen];
    buffPtr = parseBuff;
    stream->read(parseBuff, parseBuffLen);
    parseBuffEnd = parseBuff + parseBuffLen;

    //Close the stream (it's no longer needed since everything is in parseBuff)
    stream->close();

    //Get first token
    _nextToken();
    if (tok == TOKEN_EOF)
        return;
    
    //Parse the script
    _parseNodes(0);

    if (tok == TOKEN_CloseBrace)
        EXCEPTION("Parse Error: Closing brace out of place", "ConfigScript::load()");
    
    //Delete the buffer
    delete[] parseBuff;
}

void ConfigScriptLoader::_nextToken()
{
    lastTok = tok;
    lastTokVal = tokVal;
    lastTokPos = buffPtr;

    //EOF token
    if (buffPtr >= parseBuffEnd){
        tok = TOKEN_EOF;
        return;
    }

    //(Get next character)
    int ch = *buffPtr++;
    while (ch == ' ' || ch == 9){    //Skip leading spaces / tabs
        ch = *buffPtr++;
    }

    //Newline token
    if (ch == '\r' || ch == '\n'){
        do {
            ch = *buffPtr++;
        } while ((ch == '\r' || ch == '\n') && buffPtr < parseBuffEnd);
        buffPtr--;

        tok = TOKEN_NewLine;
        return;
    }

    //Open brace token
    else if (ch == '{'){
        tok = TOKEN_OpenBrace;
        return;
    }

    //Close brace token
    else if (ch == '}'){
        tok = TOKEN_CloseBrace;
        return;
    }

    //Text token
    if (ch < 32 || ch > 122)    //Verify valid char
        EXCEPTION("Parse Error: Invalid character", "ConfigScript::load()");

    tokVal = "";
    tok = TOKEN_Text;
    do {
        //Skip comments
        if (ch == '/'){
            int ch2 = *buffPtr;
            
            //C++ style comment (//)
            if (ch2 == '/'){
                buffPtr++;
                do {
                    ch = *buffPtr++;
                } while (ch != '\r' && ch != '\n' && buffPtr < parseBuffEnd);
                
                tok = TOKEN_NewLine;
                return;
            }
        }

        //Add valid char to tokVal
        tokVal += ch;

        //Next char
        ch = *buffPtr++;
    } while (ch > 32 && ch <= 122 && buffPtr < parseBuffEnd);
    buffPtr--;

    return;
}

void ConfigScriptLoader::_prevToken()
{
    tok = lastTok;
    tokVal = lastTokVal;
    buffPtr = lastTokPos;
}

void ConfigScriptLoader::_parseNodes(ConfigNode *parent)
{
    typedef std::pair<String, ConfigNode*> ScriptItem;

    while (1) {
        switch (tok){
            //Node
            case TOKEN_Text:
                //Add the new node
                ConfigNode *newNode;
                if (parent)
                    newNode = parent->addChild(tokVal);
                else
                    newNode = new ConfigNode(0, tokVal);

                //Get values
                _nextToken();
                while (tok == TOKEN_Text){
                    newNode->addValue(tokVal);
                    _nextToken();
                }

                //Add root nodes to scriptList
                if (!parent){
                    String key;

                    if (newNode->getValues().empty())
                        key = newNode->getName() + ' ';
                    else
                        key = newNode->getName() + ' ' + newNode->getValues().front();

                    scriptList.insert(ScriptItem(key, newNode));
                }

                //Skip any blank spaces
                while (tok == TOKEN_NewLine)
                    _nextToken();
                
                //Add any sub-nodes
                if (tok == TOKEN_OpenBrace){
                    //Parse nodes
                    _nextToken();
                    _parseNodes(newNode);
                    
                    //Skip blank spaces
                    while (tok == TOKEN_NewLine)
                        _nextToken();

                    //Check for matching closing brace
                    if (tok != TOKEN_CloseBrace)
                        EXCEPTION("Parse Error: Expecting closing brace", "ConfigScript::load()");
                } else {
                    //If it's not a opening brace, back up so the system will parse it properly
                    _prevToken();
                }
                                
                break;

            //Out of place brace
            case TOKEN_OpenBrace:
                EXCEPTION("Parse Error: Opening brace out of plane", "ConfigScript::load()");
                break;

            //Return if end of nodes have been reached
            case TOKEN_CloseBrace:
                return;

            //Return if reached end of file
            case TOKEN_EOF:
                return;
        }
    
        //Next token
        _nextToken();
    };
}

ConfigNode::ConfigNode(ConfigNode *parent, const String &name)
{
    ConfigNode::name = name;
    ConfigNode::parent = parent;
    _removeSelf = true;    //For proper destruction
    lastChildFound = -1;

    //Add self to parent's child list (unless this is the root node being created)
    if (parent != NULL){
        parent->children.push_back(this);
        _iter = --(parent->children.end());
    }
}

ConfigNode::~ConfigNode()
{
    //Delete all children
    std::vector<ConfigNode*>::iterator i;
    for (i = children.begin(); i != children.end(); i++){
        ConfigNode *node = *i;
        node->_removeSelf = false;
        delete node;
    }
    children.clear();

    //Remove self from parent's child list
    if (_removeSelf && parent != NULL)
        parent->children.erase(_iter);
}

ConfigNode *ConfigNode::addChild(const String &name, bool replaceExisting)
{
    if (replaceExisting) {
        ConfigNode *node = findChild(name, false);
        if (node)
            return node;
    }
    return new ConfigNode(this, name);
}

ConfigNode *ConfigNode::findChild(const String &name, bool recursive)
{
    int indx, prevC, nextC;
    int childCount = (int)children.size();

    if (lastChildFound != -1){
        //If possible, try checking the nodes neighboring the last successful search
        //(often nodes searched for in sequence, so this will boost search speeds).
        prevC = lastChildFound-1; if (prevC < 0) prevC = 0; else if (prevC >= childCount) prevC = childCount-1;
        nextC = lastChildFound+1; if (nextC < 0) nextC = 0; else if (nextC >= childCount) nextC = childCount-1;
        for (indx = prevC; indx <= nextC; ++indx){
            ConfigNode *node = children[indx];
            if (node->name == name) {
                lastChildFound = indx;
                return node;
            }
        }

        //If not found that way, search for the node from start to finish, avoiding the
        //already searched area above.
        for (indx = nextC + 1; indx < childCount; ++indx){
            ConfigNode *node = children[indx];
            if (node->name == name) {
                lastChildFound = indx;
                return node;
            }
        }
        for (indx = 0; indx < prevC; ++indx){
            ConfigNode *node = children[indx];
            if (node->name == name) {
                lastChildFound = indx;
                return node;
            }
        }
    } else {
        //Search for the node from start to finish
        for (indx = 0; indx < childCount; ++indx){
            ConfigNode *node = children[indx];
            if (node->name == name) {
                lastChildFound = indx;
                return node;
            }
        }
    }

    //If not found, search child nodes (if recursive == true)
    if (recursive){
        for (indx = 0; indx < childCount; ++indx){
            children[indx]->findChild(name, recursive);
        }
    }

    //Not found anywhere
    return NULL;
}

void ConfigNode::setParent(ConfigNode *newParent)
{
    //Remove self from current parent
    parent->children.erase(_iter);

    //Set new parent
    parent = newParent;

    //Add self to new parent
    parent->children.push_back(this);
    _iter = --(parent->children.end());
}

Porting Code

Our project uses Ogre on the client but not on the server. If you need to fetch resource files on the server side, you may try to port this code. The above code assumes the input reader does onto convert CR-LF to CR, which happens on Windows systems. It also make the assumption, "we read the whole file and then process the resulting buffer." This ignores the issue of CR-LF and also ignores the normal buffered input stream available in C++. So if any interest I can publish the code for the non-Ogre Ogre config file reader.

There is also a mistake related to the last character in the file being a }
_parseNodes. Before the "//Check for matching closing brace" it skips white space, but then the check for TOKEN_NewLine fails. The order there should be reversed.

Alias: All-purpose_script_parser

History

Information Version
Mon 11 of Jun, 2012 09:50 GMT-0000 scrawl add changelog of my version 11
Mon 11 of Jun, 2012 09:47 GMT-0000 scrawl fixed a bug in the non-ogre dependant parser 10
Wed 25 of Apr, 2012 20:04 GMT-0000 rkeene Ogre free version with some corrections. 9
Wed 25 of Apr, 2012 19:32 GMT-0000 rkeene 8
Wed 25 of Apr, 2012 19:05 GMT-0000 rkeene Warning about non-portable code. 7
Wed 25 of Apr, 2012 18:59 GMT-0000 rkeene 6
Sun 31 of Jul, 2011 22:49 GMT-0000 jacmoe 5
Wed 30 of Jun, 2010 01:10 GMT-0000 JustBoo 4
Sun 03 of Jan, 2010 00:30 GMT-0000 jacmoe 3
Fri 25 of Dec, 2009 21:44 GMT-0000 jacmoe 2
Sat 11 of Oct, 2008 17:56 GMT-0000 OgreWikiBot Robot: Cosmetic changes 1