Manager In A Strange Land: Content Turnaround

I discussed the importance of turnaround time in my last column, and pointed out that coder turnaround may not be the most important area to hit.

One of the things that can really slow game developers down is long level-load times. This is bad not only because the user will probably have to suffer the same long load time as well, but also because of how much it retards the development effort.

This is a lesson I've learned the hard and bitter way. I am currently working on the first project in years where our load times are tolerable.

Here is how I was sucked into this trauma.

Way back when, I liked text files. The game would read in text files for the world data, parse the text, and there would be the world. Almost everything was text, including the meshes. At first, everything seemed okay, because the amount of data was small, and so were load times. So optimization was not a high priority. Eventually optimization becomes a priority, but the amount of optimization you have to do to make levels load fast is enormous. Our solution, which I discussed in our Draconus postmortem, solved the problem of load times for the end player, but actually made our workflow even worse.

I like to think we were in the wrong place at the wrong time. The old-school PC developers and the console developers knew better than to have text file formats; we were fairly new, and PC hard-drives had gotten fairly powerful, so it seemed like text files were the way to go. In fact, text files were a fairly workable solution for Die By The Sword. So we didn't know any better.

I thought I was following C. A. Hoare's advice: "Premature optimization is the root of all evil." Which implies that you should write your code, and only when you discover it is too slow, optimize it.

I now know that optimizing your level load times is never premature. As with rendering, you should be thinking from day one about how to make it fast.

But text files have their advantages. Humans can read them, for one. (A disadvantage is that humans can edit them; if they're files you export from 3ds max, this can create problems for your tool chain, when someone tweaks a file to create an effect, and then people want to go back and edit the original 3ds max file.) More importantly, text files are independent of the final optimized in-game data structure, so when the code changes it can remain compatible with the old format, and the content people don't have to go into max and re-export everything. This is huge, because you really want to be able to tweak your data formats at will without requiring weeks of artist overhead.

Here's our current solution.

We use intermediate file formats that are text-based. We write tools that convert these text files into a format that is as close as possible to the binary format that the target platforms and development systems actually use. (Avoid pointers; when you need pointers, store offsets in the file that the program later fixes up.) These tools must run on the PC; having to run a console development system to build your data is no fun.

This is a best-of-both-worlds solution: you can still read the text files. But loading the binary files takes as close to the theoretical minimum of the media as possible. And when the format changes, a batch file can convert everything.

Of course, doing this is nontrivial. It took two of our best engineers working for around four months to convert all of our various objects and file formats into these fast binary representations. And there are still one or two files that never got converted.

You have to worry about the in-game data and the tools staying in sync. The way we did this was to have it be one and the same code-base: it builds a tool if you set one compiler #define but builds the game if you set another. How this system works is beyond the scope of the article, and I hope one day Game Developer will pay Greg Taylor, the guy behind it, to give it the treatment it deserves.

This just offloads the text file parsing from the game to the tool. The parsing still has to be done. A lot of time still has to be spent. But what we've done is move the time from every single run of the game to a single setup period after the user gets new data. Flow can be achieved because the turnaround on small changes is fast.

Our load times are way down. For example, when editing a game script, I can compile the script, run the game, and test the script, in a matter of seconds.

Another trick for fast loads: wad files.

Wad files are more straightforward. Seeks can take a third of a second, especially if you're reading off a DVD. If you combine all the files a level needs into one big file, you can save (number of files / 3) seconds. Videogames often have hundreds and hundreds of meshes and textures, so we're talking a saving of minutes. Again, it's a good idea to have a command line tool to build the wads.

Another thing that brings the content team down is large gets from the asset control system. If you're one of the few developers out there who still uses SourceSafe, stop: use Perforce. (We haven't shipped a product using Perforce yet, so caveat emptor.) The speed is incredible; we used to spend fifteen to thirty minutes a day just getting everybody's latest data to our local machines. Now it usually takes less than one minute. Even though Perforce costs about $800 a person, and it has to be somebody's part-time job to administrate (eventually Perforce will get as slow as SourceSafe if you don't give it love), it doesn't take long to pay for itself in employee time. Coupled with its reliability, the atomic changes in the source, the synchronization of code and data, the fact it deletes files on our local machines when we're done -- I can sync to anywhere in the history of our current project -- it's just fantastic. And their customer support! I send them an e-mail and I get a response in 15 minutes. Try doing that with SourceSafe or CVS.

I do miss one feature CVS had. If somebody submitted an evil change, I could easily undo it by typing:

cvs update -j (date+time after the evil change) -j (date+time before the evil change) *.*

and cvs would merge the reverse of that change in with our current sources. It sounds scary but every time I tried it, it worked.

______________________________________________________