Effective 3D Exporter Design

There's a common feeling in the game industry that many programmers don't 'get' artists, and visa versa. This often results in artists making fun of programmers for speaking geek, and programmers making tools that force artists to do more work than they should have to.

This article is about how you can make your 3D graphics exporter tools better, in terms of helping your artists get their job done. If you're lucky, this will get your project better art, which will mean bigger sales, and more money, fame and fortune. Plus, maybe your artists will actually like you for helping them do their best work!

Philosophy

It may sound namby-pamby, but I think it's a good idea to come up with philosophical guidelines when you are designing your tools, to give you something to judge your choices by. Here are some of mine.

- Make your tools "bullet-proof"

Obviously, the artists don't try to break your 3D exporter, but if you make them juggle eggs, then the yolk will be on you occasionally. If your tools are designed such that it's easy to make mistakes, people will make mistakes. If it's possible to design them so mistakes are harder or impossible, then you'll save a lot of time in the long run. How many times have you spent hours tracking down a crash bug, only to find was bad data, data you could have prevented from being bad in the first place if your tools checked for it?

To give you a concrete example, I once received a skinning library and exporter from another team. I immediately made a simple test and tried to run it, and it crashed with no explanation. After asking one of the artists familiar with the exporter, he messed with it for about 2 hours and finally gave up, not able to get it to work. So, it then got passed on to one of the programmers more familiar with the code.

After an hour or so of passing the files over to him, he told me 'Oh! You need to have 4 or less influences per vertex.' That's fair enough, but why does the tool crash? First, the library could check for too many influences - at least the debug version. Second, the exporter could do a similar job. The exporter might be more useful if it's flexible enough to handle more than 4 influences - maybe it should be designed to take some kind of project-based configuration file, so that you can specify the limits for a particular system, and it can check for them and warn you. I'm sure that the same scenario that I went through happened several times during the project: artist adds new art, game suddenly crashes, hours are spent tracking it down to the problem of too many vertex influences.

In fact, some commercial engines that I've seen can export data that crashes the engine. What were they thinking? The exporter should never export data the engine can't handle. As well as this, the engine or tool, at least in debug mode, should check that the data is correct. Otherwise, you're just asking for hours of debugging.

So, again, make your tools bullet proof!

- Don't force your artists to speak to a programmer in order to work out why something doesn't work.

If your artists have to come to you every time something doesn't work, then you're going to be chasing down issues for your artists all the time.

Anything you can do to design your tools so that artists can figure out the issues on their own means you can work on other things, and everyone can get their work done faster and with less frustration.

- Make the tool, not the artist, do the non-artistic work.

This is pretty simple. Basically, don't force the artists to do things a trained monkey could do. An extremely simple example would be - don't force them to break up their quads into triangles. If you are passed a quad and you need triangles, then silently make triangles. Of course, you should probably check if it's planar or not and mark it as an error or warning (see below). But the point is - wherever possible, do the obvious thing.

- Make it possible for your tools to work as part of a one-step build process.

You've probably heard about a one-step build being a good idea. The one-step build avoids human error, by eliminating all manual steps in the conversion of human-made creative assets to game-ready data. In particular, this means that artists should not be required to manually re-export their 3D models. They just create the models, textures, level maps, and so on. The build tools will export them as necessary (when they are newer than the game-ready versions of the same data.) An added benefit of the one-step build is that your artists spend most of their time creating art, instead of repeating non-creative tasks that automated systems could do a lot faster and cheaper.

- Make your tools to support the way that artists work.

There's a point to this piece of philosophy. Never forget that the goal of your tools is to help the artists make great art. You do that by designing your tools to support all the features and techniques that artists typically use. We'll go on to discuss some examples of how this philosophy comes into play.

______________________________________________________

Designing a tool: 3D Exporters

It seems like most exporters I come across are designed to export everything in the currently loaded scene in 3D Studio Max, Maya, Lightwave, and so on, but within the limits of the system the programmer designed. If anything in the scene is hidden, they don't export it, and for animation, they export based on the current setting of the time line in the user interface. Alternatively, they export the currently selected items. Some people refer to this export style as WYSIWYG exporting. It sounds like a good idea at first, and it might be great for testing your new engine. But with a little thought, it soon becomes clear that a different design would make things go smoother for everybody involved.

Let's see what some of the problems are, and the solutions that come up by applying the philosophies above.

- Make your tools show errors that can be noticed, understood and useful.

Asking an artist to look through ten thousand lines of diagnostic output, in order to find the three warnings that flew by while the data is converted is not useful. Following the "Don't force your artists to speak to a programmer" philosophy, the best way to deal with problems is to show them visually. If certain polygons are obviously wrong, then somehow mark them and show the artist.

In my tools, if certain polygons are bad, I'll still output the polygons or the rest of the model, but my file formats have an optional semi-generic error encoding system where I can put in error messages, error lines, error points, and so on. When the 3D objects are loaded up into the viewer or the game, any object that has error information can be displayed or marked in obvious colors such as flashing red.

Those objects can then be selected, and the errors specific to that object can be brought up. If the errors are visually representable, then the polygons associated with that error will also be displayed if the object is selected and a particular error highlighted. For example, an error selected in the debug menu that says "polygon has too many weights" will highlight the polygon in question in the game or previewer.

Another good idea is to have the programmer that wrote the error message put their name in the message. For example, it could state: "WARNING: found vertices with more than 4 influences". That way, the artist will immediately know who to talk to if they have a question about the message.

- Never let your tools fail.

I know some of you will disagree with this idea, but hear me out. This is a goal I tried to reach on my last game. No matter what the problem, try to output something into the game or previewer. This goes along with the philosophy of "Don't force your artists to speak to a programmer in order to work out why something doesn't work".

It's very frustrating when you can't build a level because something is broken, and even more so when the problem is unrelated to what you are currently working on. It's best to at least get the object or level working as soon as possible, even with errors or missing items. For example, if a texture path is incorrect and you can't find the source file for the texture, then use a special 'ERROR' texture and still write out the file. This will make it visually clear to the artist what is wrong, and they will most likely know how to fix it immediately, but they will also get to see their model now instead of later.

Another example from my level building tools - if one object in the level has errors the level still gets built, but that object is just tagged as bad. There are often thousands of items in a level made by different people. If all it takes is for one of them to be bad to stop your level from loading, you are not going to be getting much work done.

You can go into more detail still, using both warnings and errors in exported data. A warning means that the data needs to be fixed, but the object should still run. In other words, the game should feel safe to execute the A. I. code for that object without fear of crashing. An error means that the particular object is not in a state the game can use. After loading a level, all objects with warnings would be flashing and all objects with errors will just have a name displayed in their place. At that point, the artist or designer can select objects and find out the specific errors. This is much easier to fix than sifting through hundreds of lines of building diagnostics.

However, in some cases it might be appropriate to fail immediately, especially if building a level or object takes a long time. If there are things you can check for quickly that you know will cause a problem, you might want to bring those to the attention of the artist immediately. But if possible, visual evidence is almost always better than just an error message.

- Don't force your artists to remove helper objects.

Many exporters require your artists to have the data they need, and only that data, in their 3D editor. Artists are not allowed extra lights, extra models, extra cameras, and so on. But have you ever actually seen how your artists work? They need lights to light their models. They need extra models for constraints, for checking sizes, or for other references. Asking them to have to delete those every time they export is asking for trouble.

Firstly, you are making artists do this work manually every time they export. They may even have to rebuild all their lights and constraints if they accidentally save after they've deleted them for exporting. Alternatively, they may forget to remove them, export and thereby clutter the game with un-needed items. Even worse, they may accidentally make the game crash and you've got another hour or two as you try to figure out why.

Following the "make your tools support the way artists work" philosophy, it would be better to find a way to let artists specify what needs to be exported, so that you can ignore all the rest. That way your artists are unlikely to make as many mistakes. They don't need to remember to delete objects. They don't have to worry about accidentally saving after having deleted objects. They can keep all the objects, lights and cameras that make their job easier.

I'd also recommend that the artists should specify what to export, as opposed to specifying what not to export. If you choose the "not" route, then if they add something and forget to mark it to "not" be exported , you're going to end up with stuff you don't want in your data. This is worse than finding something is missing because it was not specified, since once it is specified it will never be missing again. It is usually easier to see what is missing in the game than to notice something extra that isn't supposed to be present.

- Don't force your artists to check export settings more than once.

Many exports have a gigantic amount of settings. For example, they include elements choosing whether to export vertex colors or animations; whether to save textures as filenames or re-write them; whether to save as an object or a character; what internal name to use for a character, and so on.

While these settings might be important, it's asking for trouble to ask your artists to check them every time they export the same file. Following both the "Make the tool, not the artist, do the non-artistic work" and the "make it bulletproof" philosophy, these options should be set once and saved somewhere for exporting that particular file. So, if the camera is not supposed to be exported in this file, then the next time the file is loaded and exported it will automatically not export the camera. If another file is supposed to export animations with a base node called "orcbase", then if it is loaded and exported again, it should automatically export animations as "orcbase".

Following the "make it bullet proof" rule, the artist should only have to look at settings if he or she needs to change them, and never once they are set. If they have to check them every time they export, odds are some percentage of those times they will miss checking or un-checking an option, and the coder and the artist will lose more time working things out.

- Don't use the time line as an exporter setting.

Lots of exporters export animation by looking at the current time line in the user interface. If it's set from frame 10 to frame 150, then they export 141 frames from frame 10 to frame 150. As above, this is also error-prone. Every time an artist exports, they have to remember to put the editor's UI in a certain state. If they forget to do so, then certain in-game animations will no longer work properly.

Following the "make your tools bullet proof" philosophy, it's better to have a way to specify once which frames to export. That way, artists don't have to remember to reset the time line. They know when they export, no matter what state the UI is in, they are going to get the frames they want.

An even better option would be to write a flexible specification system that allows your artists to put more than one animation in the same file. For example, your walk cycle might be frames 10 to 40, and your run cycle might be frames 100 to 130. Some artists find that method of animating much easier than forcing them to do every animation in a separate file. It also means global changes to a character, like adding a gun attachment point to the hand, will be reflected without changes needed to all the other animations.

- Don't force your artists to collapse hierarchies.

Another issue I see in lots of exporters is that they export the hierarchy in the 3D tool directly to the game hierarchy. In order to have the best speed in today's hardware, it's important to have as few separate models as possible.

Unfortunately exporters that export the hierarchy directly require the artists to take separated models, and merge them into one model. A simple example might be a static 2-wheeled cart. The artist might build it using 2 tubes for wheels, a cylinder for an axel, 6 cylinders per wheel for the spokes, and then a box minus 2 faces for the cart area. In some exporters, if those objects are not all merged into 1 model, they will be exported as 16 separate models. Most programmers don't give that a second thought; they just assume the artists will merge everything.

Once the artists merge the models, it becomes much more difficult to edit. For example, if an artist wanted to widen the cart, they would want to stretch the axel and the cart, but not the wheels or the spokes. If you forced them to collapse the model they can no longer do that easily.

"Making your tools support the way that artists work" means letting the artists leave their models in the state best suited for them to edit it to perfection, not the easiest state for a tools coder to export. If your exporter has a way to specify: "From this node down in the hierarchy, merge all children into one model", you'll allow your artists to leave their art in the best state for editing, and still get optimal models for the game.

- Don't use the hidden or frozen attribute to filter the export.

Going back to "Make your tools bullet proof", your exporter shouldn't care if something is marked as hidden or frozen. That's because your specification says what to export, regardless of whether a particular item is hidden or frozen. Like above, your artists may have hidden something to make the model easier to edit, and then saved the file. The next time the file is exported, the code should still be exporting everything required and not exporting extraneous items. Otherwise, you are going to run into time-consuming debug problems, next time you load up your level.

- Don't export function curves!

I've seen many exporters that pull function curves directly out of the 3D package. The documentation for the exporter then goes on to tell the user that, because of this *feature* the artists are only allowed to use certain kinds of animations. Does anybody see a problem here? All the interesting animation functions require features not supported by function curves. Probably the most important one is constraints. Constraints are incompatible with exporting function curves, as are expressions.

Following the "Make your tools support the way artists work" philosophy, a much better solution is to sample the output of the package in question. You can then write some curve fitting functions, if you happen to want to use function curves to compress your data. This way, your artists are free to use every tool at their disposal to make great animation, and you'll still be able to compress their results to something reasonable.

- Don't ever consider editing a middle file format.

Middle file formats are a great idea. They allow you to separate your processing tools from your exporters, and also make it possible to use different 3D packages on the same project.

However, middle file formats should really be treated just like .obj or .o files in a compiled language. They are files that the build process makes that are temporary files, never to be touched by human hands. The reason? Obviously, if any changes need to be made to the 3D data, it's going to be made by the artists back in the original 3D package. Any changes applied to the middle format are going to be lost when the artist re-exports the new model.

This goes back to the goal of your tools being to help your artists make great art. If your processes involve editing the middle file formats, and that editing is lost every time your artists edit an original 3D source file, you are forcing someone to waste time manually re-tweaking the new middle file. You've just given them an incentive to not want to edit things to be better.

- Don't needlessly limit the possible applications of your exporter.

Design your exporter to handle as many cases as possible. To give you a concrete example, I once worked with an exporter that required the artists to manually export a "bind pose" for skinned characters. That one idea removed any possibility of using that exporter to export full real-time 3D cut-scenes, since a cut-scene would never have all the characters in a bind pose state. Instead, the exporter should lookup the bind pose automatically.

Another example of this would be limiting your exporter to only exporting one item at a time. That limit would provent you from exporting multiple characters or multiple animations. If you removed that limit, your artists would be able to make real time cut-scenes with a single click export process, as well as make it easy for artists to animate throws and other multiple-character animations.

- Don't create or load bad data.

Your exporters should never create bad data, and your engine, previewer and tools should never load bad data. Obviously, if you have bugs in either, you might get bad data, but often there are other issues at play.

I put version numbers in all my data files to support the idea of making your tools bullet proof. This is done so that the art previewer and game code can check if they understand a certain type of file, and print an error on the screen if the version is out of date. The alternative is that your game loads the data, then crashes without showing a specific error, and you spend hours trying to figure out why.

There are other various ways to handle this issue. For example, at one company I worked at, the version numbers were used as part of the load path. So, for example, a version 2.1 character might come from "\gamedata\2.1\characters", where as code that needed 2.3 version characters would load them from "\gamedata\2.3\characters". Based on their version, all the tools knew where to put their results.

This method had the advantage that people using older versions of the code still had access to data that worked with their version. In turn, people using or working on newer tools and newer versions could safely build new assets, and not worry about breaking everyone. Personally I'd probably use both ideas just to be safe. But, however you do it, a similar approach will save everybody time not tracking down version issues.

In my version numbers, I have option bits - for example, a bit that says if the file is the kind of file with error information or not. I mentioned above that putting error info in the file is a good idea for development. But in the shipping product, that code and that data is a waste, so I compile it out. But I still have the code check for a version number, and that includes whether the file is a development file or a release file. Otherwise, during final bug testing it's likely that a development file is going to get accidentally used during testing. Without the check and corresponding error, there are going to be hours spent tracking down the cause of the crash.

- Don't require manual exporting.

Manual exporting breaks the advice of the one-step build. I can't believe how many teams put up with manually exporting things - it should be possible to type "make", and have all your 3D assets reprocessed from original Max/Maya/Lightwave files into the data for your game.

What happens when you don't do this? Every time you make a change to your 3D format, your artists have to manually re-export everything. When you start a new platform again, everything will have to be manually re-exported.

Yes, of course, for a previewer function you'll want a "preview" option that will export the data and pop it into your previewer. But when the data actually exports into the game, you want exporting to happen automatically every time you ask it to rebuild your data.

______________________________________________________

Automatic batching for 3D Exporters

Following the "Make the tool, not the artist, do the non-artistic work", once you've solved the items above, it should be a relatively simple matter to make automatic exporting part of your build process. For Maya, you can use Mayabatch. You can pass Mayabatch a script that your build process creates to load and export a file. For 3D Studio Max, you can do the same with Maxscript by having your build process create a script to load a file, export it and exit Max.

Another added benefit of not requiring manual exporting is that your artists will never wonder which Maya/Max/Lightwave file they need to load up, in order to edit an asset. The file they need to load will be the file that's in the build.

However, I'm not suggesting there should be no way to manually export. I'm merely suggesting that your tools should be able to build in one step. Your process should facilitate matters, so that if you need to rebuild something, there are no manual steps required. If you need the ogre model rebuilt, you should be able to say something like "rebuild ogre". The alternative is to hunt down the one artist in your company that worked on the ogre, knows which of 45 files named ogre_23.max, ogre26.max, ogre25b.max, ogre-test3.max, and so on need to be loaded, and then where and how to export it. This is not a very appealing process.

For example, in a recent project, we used some 3rd party tools that didn't follow this advice. We had a character with eight animations. As I was programming, there were several times I had to ask my artist to add a few nodes to the character's hierarchy. Because the system didn't support a one-time build solution, each time he edited the character, he had to hand export the model, and hand export each of the eight animations. This included editing several exporter settings before each export. If the exporter had been designed better, he might need to only press one button to get all nine files exported, after having done it the first time.

In this case, the multi-stage export is still not the end of the story. The artist had to then take each of the nine exported files and copy them to the correct places in the build. If the tool truly supported a one-step build, that entire ten minute manual labor error prone process would have been reduced to saving the file and saying "done!".

There are levels or stages of one-step building. Sometimes, you want an immediate export-like function to iteratively edit an object being used in the current level. Other times, you want to click one button or type one string, and have a particular object or level and all its updated assets re-exported and reprocessed. Other times, you want all of it re-exported and reprocessed from scratch. Or possibly, you might want to rebuild the entire game. Your tools should support all of the above. If your exporter just supports manual exporting, only the first option is available to you. If your exporter is based initially on automated exporting, all the other options are just simple interface issues.

Separate Exporting from Specification

Some people will read the above ideas and believe what I'm suggesting is a system that uses a specification file, as opposed to a WYSIWYG system, but please don't make that mistake. In other words, some imagine some ugly non artist-friendly text file that specifies what and how to export stuff, versus the common method of exporting that you currently see in the 3D package. Although the ideas above require a specification, that doesn't mean making the spec couldn't be WYSIWYG.

For example you could make a plug-in for your favorite 3D software that looked at the current scene, what's visible and what's not, and generated a specification file for the exporter system. This would essentially give you the same functionality as a normal no-specification file system, but it would have all the advantages above plus more.

For example, with a specification-centered file system, you can have multiple characters per 3D file, and extract each character separately. That would be useful for genres like fighting games, where there are animations and moves that combine two or more players and complex movies such as throws. The artists would find it hugely beneficial to be able to keep both characters in one file, and animate them together.

Another advantage is that using a specification file reduces the chance for human error in the build. Sure, the artist might forget to add something to the spec file if they add a new object in the scene. But the chance for that mistake happens only once, when they add something new or delete something. With a good preview system that will be known almost instantly.

In contrast, relying on a manual exporter system creates an opportunity for error every time a file is saved - i.e. every time an animation is tweaked, a texture remapped, a property edited, etc. So, not only is the spec file system more powerful ( allowing multiple animations, characters and other actions per file), but it drastically reduces the chances for errors. All the while, this doesn't restrict the artist from having other scaffolding in the file to help them do their job.

Separating the exporter from the specification file generator also gives you unlimited options regarding how to generate the spec. You could:

If you make your specifications text or XML-based, you'll have the option to hand-create them when needed, on top of generating them like mentioned above.

In fact, although it can be a cop out, if you make a good exporters that export based on a specification, you can start with hand written specifications. Then, as your team has time and finds new issues to solve, you can make more and better specification generation and editing tools. Those tools could be any of the spec-making plug-ins mentioned above, or they could be Microsoft Infopath-based XML or a web-based editor with a list of each animation, starting and ending frames, and joints to include or exclude. The sky's the limit.

There's another advantage in solving all of the above issues - the ability to allow the artist to create 3D cut-scenes completely in their favorite 3D package. There's no more having to edit the camera in-game, or assemble all the parts (characters, backgrounds, camera motion, character animations) separately, They can just make the perfect cut-scene using all the features of the 3D package. Then using a spec, either automatically or hand generated, all the required parts can be automatically exported and run through your tool-chain to get into the game.

You can even merge specs as well. You could pass the tools multiple specs, and they would load all the items referenced by the various specs, export them, and process them into a single bundle of data for loading. This is a great way to bundle together related data that is best edited separately.

Conclusion: happy artists make better games!

I hope this article has pointed out why it's important to design your tools with a philosophy in mind. My impression is that most tools are designed by a programmer, for a programmer, to supply data for other programmers. Therefore, the work flow of the person who is going to use the tools is not carefully considered. Nor are possible ways to avoid errors. Game library code writers are often some of the worst offenders, not because they are not smart, but because their sole goal in writing an exporter is generally to make some test data for their library. No thought is given to using the tool outside of that one small case. I think most tools would benefit from taking these kinds of ideas and philosophies into consideration during their design.

While some of these ideas might be more work than the simpler solution you are living with now, it only takes one programmer to implement a better solution. The solution will effect, help and benefit 10 to 150 artists, which will more than pay for itself over the course of the project.

In the end, the more you help your artists, the better job they will be able to do. That's not only going to mean better art and more sales, but it will mean less stress for you and maybe even some compliments from your fellow artists. What, compliments to a programmer from an artist? I bet you thought it was impossible!

Acknowledgements

I wanted to thank all the people that helped me with this article. This includes personnel from Atari, Inxile, Humongous, Naughty Dog, Electronic Arts, Capcom, Sony, and Activision.

______________________________________________________