Text pre-processing

Text pre-processing is a useful technique for adding features to systems that use text files as input. Probably the most common example of pre-processing is the C pre-processor. This is the mechanism that handles comments, #includes and #defines before C/C++ code is feed into the compiler. These are features that could be used in many different types of text files and Boost Wave is one C pre-processor implementation that’s readily available.

In addition to using a “generic” pre-processor we can also create very domain specific pre-processing functions. Here are two examples.

D3DX FX Shader Techinques

We use the D3DX FX system for our shaders. One problem you can run into with shaders is requiring all sorts of different combinations of parameters, like does this shader use normal mapping, skinning, etc. Writing all of these combinations out by hand can be tedious and the FX system doesn’t directly have anything to support the concept of shader permutations.

technique WriteDeferred[AlphaTest][NormalMap][Skinned]
{
  pass p0
  {
    VertexShader    = compile vs_2_0 WriteDeferredVS(Skinned);
    PixelShader     = compile ps_2_0 WriteDeferredPS(AlphaTest, NormalMap);
#if AlphaTest
    AlphaTestEnable = True;
    AlphaFunc       = Greater;
    AlphaRef        = 128;
#endif
  }
}

With this enhanced syntax, we instruct the pre-processor to generate all combinations of the WriteDeferred technique with the AlphaTest, NormalMap, and Skinned options. This results in 2^3 = 8 combinations:

WriteDeferred
WriteDeferredAlphaTest
WriteDeferredNormalMap
WriteDeferredAlphaTestNormalMap
WriteDeferredSkinned
WriteDeferredAlphaTestSkinned
WriteDeferredNormalMapSkinned
etc.

Here’s what it expands to after our pre-processor runs on it:

technique WriteDeferredNormalMap
#define AlphaTest 0
#define NormalMap 1
#define Skinned 0
{
  pass p0
  {
    VertexShader    = compile vs_2_0 WriteDeferredVS(Skinned);
    PixelShader     = compile ps_2_0 WriteDeferredPS(AlphaTest, NormalMap);
#if AlphaTest
    AlphaTestEnable = True;
    AlphaFunc       = Greater;
    AlphaRef        = 128;
#endif
  }
}
#undef AlphaTest
#undef NormalMap
#undef Skinned

Note that our custom pre-processor generates instructions for the standard C pre-processor that the D3DX FX system understands. Generating pre-processor #defines based on the permutation allows us to customize the body of the technique for each configuration. We also use to it alter the compilation of vertex and pixel shaders using compile-time constants.

This is a simple example, where each of the “parameters” is either on or off, but since we have complete control in our pre-processor, we can make things as complex as we like. Here’s another example from our system, which allows us to specify a set of values to choose from (parenthesis mean the parameter always has one of the values, brackets mean the parameter is optional).

technique Fill(Color,Textured)[Add,Multiply][UseStencil]
{
  // ...
}

One caveat to modifying the input to the FX compiler this way is that the line numbers will be incorrect if it reports an error. This easily solved by maintaining a separate mapping from old line numbers to new ones. Using this, any error messages that reference line numbers can be fixed up before logging them.

Lua Profiling

Here’s a piece of Lua code that’s marked up for profiling:

function Actor:UpdateBoneCoords()
 
    PROFILE("Actor:UpdateBoneCoords")
 
    local model = Shared.GetModel(self.modelIndex)
 
    if (model ~= nil) then
        local poses = PosesArray()
        self:BuildPose(poses)
        return
    end
 
    self:UpdatePhysicsModelCoords()
 
end

We use this same method for instrumenting our C++ code for profiling. In C++, PROFILE is a #define which creates a C++ object on the stack. When this object is constructed profiling begins, and when it’s destroyed (because it goes out of scope) profiling ends. In a ship build, the PROFILEs are #defined to nothing to eliminate any overhead from the final game.

In Lua, there’s no way to directly implement this because we don’t have control over an object’s lifetime. There’s also no built-in pre-processor, so there’s no way to completely eliminate the PROFILE markers. Both of these problems can be solved with a custom pre-processing step. Our pre-processor searches for the PROFILE token and inserts function calls at it’s location, and at every location where we exit the Lua block that contains it.

Here’s what the pre-processed output looks like:

function Actor:UpdateBoneCoords()
 
    __profile_start(10)
 
    local model = Shared.GetModel(self.modelIndex)
 
    if (model ~= nil) then
        local poses = PosesArray()
        self:BuildPose(poses)
        __profile_end(10); return
    end
 
    self:UpdatePhysicsModelCoords()
 
__profile_end(10); end

In addition to inserting the __profile_begin and __profile_end markers, the string “Actor:UpdateBoneCoords” was also inserted into a string table and the index was substituted. This allows the profiler to run with as little overhead as possible. Handling all of the different syntax cases in Lua is a little complex, but the code is only around 150 lines of code.

With the Lua pre-processor, it was possible to maintain the line numbers using Lua’s ability to have multiple statements on a single line. Keeping the line numbers identical eases error reporting and Decoda integration.

6 Responses to “Text pre-processing”

  1. Reg said:

    Nov 14, 10 at 11:31 am

    When talking about preprocessor in C, C++ or HLSL, I suppose #line macro could be useful in solving the line numbering problem.

  2. Mads said:

    Nov 15, 10 at 6:30 pm

    I am so stealing this ;)

  3. Kevin Bjorke said:

    Nov 18, 10 at 2:33 pm

    I’ve used the m4 pre-processor off and on for a long time — it’s usually part of stock linux/cygwin releases but almost no one knows it’s there, much less how to use it. It has the features of a cpp-style processor but also recursion, loops, and functions.

    At Square, I used it to generate lots of shader permutations so that if an artist needed varying numbers of resources like shadow maps, they could just get the right shader with a few clicks (and I only needed to write a single “core” file).

    When I was at NVIDIA I was able to write shaders that could be re-emitted as DX9, DX10, CgFX, and bits of Cg (for translation into GLSL, which officially lacks an #include in its preprocessor). That was an extreme case, but you probably get my drift.

  4. Ari Timonen said:

    Nov 20, 10 at 3:03 pm

    Interesting, thanks Max.

  5. mens weight loss tips said:

    May 02, 13 at 10:20 pm

    I all the time emailed this weblog post page to all my associates, for the reason that if like to read it then my contacts will too.

  6. telecharger snapchat said:

    Apr 21, 14 at 3:34 am

    Currently it sounds like Expression Engine is the preferred blogging platform out there right now.

    (from what I’ve read) Is that what you’re using on your blog?


Leave a Reply