Eliminating Exporting in an Asset Pipeline

An asset (like a model) undergoes a number of steps from in the journey from an artist’s tool to real-time display in a game engine. These steps are known collectively as the asset pipeline. The pipeline for a typical game engine looks something like this:

The proprietary format on the left is the format that the content creation tool natively saves. For 3D Studio MAX this is the .MAX format. This is an undocumented binary format that relies on installed plug-ins to load, so it’s not really possible or practical for our own tools to read it directly. In order for our tools to load the asset it first must be converted into an intermediate format.

The intermediate format is either a standard format like COLLADA or a custom defined format that is easy for our tools to read. This file is exported from the content creation tool and stores all of the information we could possibly desire about the asset, both now and in the future. Normally this export process is done by selecting Export for the tool’s menu and then choosing the name of the file to write. Since the intermediate format is designed to be general and easy to read, this intermediate format isn’t optimized for space or fast loading in-game. To create the game-ready format, we compile the intermediate format into a game format.

The game format is an optimized format that is loaded by the engine at run-time. The game format file is generated by our tools from the intermediate format. The main reason we don’t directly export from 3D Studio MAX into our game format is that we may want to change it in the future. If the game format was created by our exporter then any change to the format or creation process would require re-exporting all of our assets. Having an intermediate format also allows us to have different versions of the game format that are targeted at different release platforms (PC, console, etc.).

One annoyance with this setup is that the proprietary file and the intermediate file have to be kept in sync — that is whenever a change is made to the .MAX file, the artist needs to be re-export the intermediate format for the change to appear in game. Typically the proprietary file and the intermediate file would both be kept in some sort of revision control system such as Perforce. In addition to needing to be re-exported, the files need to be checked in as well. While these both seem like they would be easy, on a large team with tens of thousands of assets, mistakes will happen.

The ExportObject in 3D Studio MAX

A solution that we used at Iron Lore was to automatically export the intermediate format when the file was saved in 3D Studio MAX. The way this was done was with a special plug-in. This plug-in was an object called an ExportObject that the artists would place in the scene. The object looked like a floppy disk (how quaint!) and the only thing it really did was implement a custom save handler. As mentioned earlier, the proprietary .MAX format requires plug-ins to load it, and the reason for this is that each plug-in is responsible for saving its own data. In the case of the ExportObject, during the “save” it would export the entire scene to the proprietary file format.

There are a few options for how to save this data. The first is to save directly to a new file on disk. The second is to write the data in-place in the .MAX file (i.e. where the plug-in is supposed to save its data). The third is to append the data to the end of the .MAX file.

The first option is unattractive because we still have two files that would need to be kept in sync in Perforce. The third option is the one that we used at Iron Lore and is possible because 3D Studio MAX will ignore data at the end of the file when loading a .MAX file. Implement this system again today, I would chose the second option because it doesn’t rely on this quirk of the loader.

Once the data is auto-exported, it’s a simple matter to load it in our own tools. In the case of appending the data to the end of the .MAX file, I included the length of the intermediate data as the final 4 bytes. This makes it very simple to seek backwards and read only the required data.

If the data is written in-place, then a special token needs to be used to locate the appropriate block of data. This token is simply a string of bytes that is unlikely to appear anywhere else in the file and signifies the beginning of the data. If this string is long enough, the chance of it appearing elsewhere in the file is astronomically small. If this doesn’t seem robust, consider that there is roughly a 1 in 108 chance that you will be struck by lighting on any particular day; there is a 1 in 1068 chance that an arbitrary 8-byte string will appear in a typical sized .MAX file.

Since the ExportObject is a persistent member of the scene, it can also store options that might be necessary for exporting. Since the export options are stored along with the .MAX file, the file can be re-saved/exported without the operator having to know or re-input the correct options.

This system worked very well at Iron Lore — in the future I’ll describe our first approach which was much more problematic!. The only thing to keep in mind is that the export must be fast so that it doesn’t noticeably slow down the saving process for the artists.

6 Responses to “Eliminating Exporting in an Asset Pipeline”

  1. Dalin Seivewright said:

    May 20, 10 at 9:41 pm

    Did UWE consider other pipeline options before choosing to do things this way (assuming it is the process you’re using?)
    Verse (http://www.uni-verse.org/) has always seemed like a great idea to me – real-time communication between applications (so changes to a model can be instantly updated on another application) and you can store the assets in a single file. Applications like Blender already have support for this type of thing, which lets it connect to a Verse server and modify any assets on it. This means you don’t need any proprietary formats (except the one that verse supports).

    Even Verse isn’t for you (as I understand it, there aren’t many programs that support Verse, so you might have to write your own applications built around it), a few plugins that allowed for custom import/export would allow you to store a single copy of the model in its intermediate form (rather than a proprietary and intermediate form).

    Just some thoughts.

  2. Peter Thierolf said:

    Sep 09, 10 at 9:08 am

    We tried to store our export data inside of the .max file for all the same reasons.

    However, we failed reading it from the .max file again.

    The reason is that 3dstudio stores some own data in the middle of your data whenever it gets longer than a few kilobytes. It seems to store some strange tables and investing a few days, we have not been able to re-engineer enough of it to understand it.

    I personally would very much appreciate a solution that allows to scan nodes and read data of some builtin object types like tri-objects from the original file.

    True, you can’t access any weird plugins and also you have to update your reader in order to keep up with autodesk.

    However, if you ask artists to put a respective supported modifier on top of the modifier stack for all objects that should end up in the game would be acceptable I think. That is not so much different from asking for a special exporter objects presence.

    That way we wouldn’t increase the files size, there would in no case be a requirement to re-save files because we had to fix the exporter etc. *and* it would be really fast.

    If there is anyone out there who wants to achieve the same, team up ?

    Cheers,

    Peter

  3. Peter Sedov said:

    Dec 30, 10 at 2:31 pm

    >during the “save” it would export the entire scene to the proprietary file format.

    Did you mean ‘intermediate format’?

    >chance that you will be struck by lighting

    You probably meant ‘lightning’ (thunderbolt). Being struck by lighting (illumination) is really sad :).

  4. Roscoe said:

    Aug 14, 14 at 7:19 pm

    There are specific inquiries facing contractors and subs in Bexar County to apply proper brush strokes and give you several ways, failed wells,
    or if the project. Check that your search
    for appropriate interior consumers repairs and refurbishing an old house hidden unexpected problems.
    These components include: the furnace and air conditioning contractor.

  5. Trevor said:

    Sep 03, 14 at 10:10 am

    Thanks , I have recently been ssearching for information approximately this subject for ages and yours is
    the best I have came upon so far. However, what iin regards to the bottom
    line? Are you positive about the source?

  6. 3d architectural visualization said:

    Sep 20, 14 at 12:19 am

    ӏ likе the valuable information yߋu provide in your artiϲles.
    I will Ьookmark your blog and take a look at once
    more here frequently. I am fairly sure Ӏ’ll be informed many new stuff proper right Һere!

    Best οf luϲk for the following!


Leave a Reply