Eliminating Exporting in an Asset Pipeline
An asset (like a model) undergoes a number of steps from in the journey from an artist’s tool to real-time display in a game engine. These steps are known collectively as the asset pipeline. The pipeline for a typical game engine looks something like this:
The proprietary format on the left is the format that the content creation tool natively saves. For 3D Studio MAX this is the .MAX format. This is an undocumented binary format that relies on installed plug-ins to load, so it’s not really possible or practical for our own tools to read it directly. In order for our tools to load the asset it first must be converted into an intermediate format.
The intermediate format is either a standard format like COLLADA or a custom defined format that is easy for our tools to read. This file is exported from the content creation tool and stores all of the information we could possibly desire about the asset, both now and in the future. Normally this export process is done by selecting Export for the tool’s menu and then choosing the name of the file to write. Since the intermediate format is designed to be general and easy to read, this intermediate format isn’t optimized for space or fast loading in-game. To create the game-ready format, we compile the intermediate format into a game format.
The game format is an optimized format that is loaded by the engine at run-time. The game format file is generated by our tools from the intermediate format. The main reason we don’t directly export from 3D Studio MAX into our game format is that we may want to change it in the future. If the game format was created by our exporter then any change to the format or creation process would require re-exporting all of our assets. Having an intermediate format also allows us to have different versions of the game format that are targeted at different release platforms (PC, console, etc.).
One annoyance with this setup is that the proprietary file and the intermediate file have to be kept in sync — that is whenever a change is made to the .MAX file, the artist needs to be re-export the intermediate format for the change to appear in game. Typically the proprietary file and the intermediate file would both be kept in some sort of revision control system such as Perforce. In addition to needing to be re-exported, the files need to be checked in as well. While these both seem like they would be easy, on a large team with tens of thousands of assets, mistakes will happen.
A solution that we used at Iron Lore was to automatically export the intermediate format when the file was saved in 3D Studio MAX. The way this was done was with a special plug-in. This plug-in was an object called an ExportObject that the artists would place in the scene. The object looked like a floppy disk (how quaint!) and the only thing it really did was implement a custom save handler. As mentioned earlier, the proprietary .MAX format requires plug-ins to load it, and the reason for this is that each plug-in is responsible for saving its own data. In the case of the ExportObject, during the “save” it would export the entire scene to the proprietary file format.
There are a few options for how to save this data. The first is to save directly to a new file on disk. The second is to write the data in-place in the .MAX file (i.e. where the plug-in is supposed to save its data). The third is to append the data to the end of the .MAX file.
The first option is unattractive because we still have two files that would need to be kept in sync in Perforce. The third option is the one that we used at Iron Lore and is possible because 3D Studio MAX will ignore data at the end of the file when loading a .MAX file. Implement this system again today, I would chose the second option because it doesn’t rely on this quirk of the loader.
Once the data is auto-exported, it’s a simple matter to load it in our own tools. In the case of appending the data to the end of the .MAX file, I included the length of the intermediate data as the final 4 bytes. This makes it very simple to seek backwards and read only the required data.
If the data is written in-place, then a special token needs to be used to locate the appropriate block of data. This token is simply a string of bytes that is unlikely to appear anywhere else in the file and signifies the beginning of the data. If this string is long enough, the chance of it appearing elsewhere in the file is astronomically small. If this doesn’t seem robust, consider that there is roughly a 1 in 108 chance that you will be struck by lighting on any particular day; there is a 1 in 1068 chance that an arbitrary 8-byte string will appear in a typical sized .MAX file.
Since the ExportObject is a persistent member of the scene, it can also store options that might be necessary for exporting. Since the export options are stored along with the .MAX file, the file can be re-saved/exported without the operator having to know or re-input the correct options.
This system worked very well at Iron Lore — in the future I’ll describe our first approach which was much more problematic!. The only thing to keep in mind is that the export must be fast so that it doesn’t noticeably slow down the saving process for the artists.