3D Engines for Games: A Broader Perspective

Author: by Søren Hannibal

You've developed an object viewer, a Quake level viewer, a radiosity renderer and a patch tessellator. You know 3D math, OpenGL and the 3DS file format by heart. Now you want to go further -- you want to use this knowledge to develop an entire engine.

Does this seem like a large task for you? Well let me tell you, it is. Anyone who has developed a complete game will agree with me. Many people give up or start over along the way. But there are a number of things you can do that will increase your chances of reaching your goal.

I will not try to tell you how to make a good, playable game. Nor will I tell you how to program any specific parts of an engine. In fact there won't even be a single line of code in this article. Rather, I want to encourage you to think beyond shadow volumes and dotproduct3. Think about what it will all be used for, and work with this goal in mind.

Even though this article talks about the development of a 3D engine, the same techniques should be applicable to, for example, 2D or playing-card game engines. The main goal is to enable you to quickly and (relatively) painlessly develop an engine and a game that uses it.

What to Consider

There are three steps that I undertake before beginning work on an engine:

These three steps might require a few days at the beginning of the project (when you're eager to get busy with your compiler), but I guarantee you that the time will be well spent. It is entirely possible to develop a game without any planning at all, but the further you get in development, the more you will wish you had devoted some time to planning in the beginning.

There are a number of reasons why you should clearly define your goals before you begin.

The specs of the engine should be fitted to the team making it, and to the game for which it is being made. Programmers often aim too high, and start building features that are not completely necessary. Or they may not have the time, the manpower, or the skills to finish off a highly polished game.

Setting the goals help you clarify your focus, and give you a direction to go. With a clear purpose, you will likely keep on the right track through the end of the project. One small completed project, even with only a few levels, is better than ten great projects abandoned when they are 90% finished.

The third reason for establishing clear goals is that you will remember that developing the engine is not an end in itself; it is just a tool to make the game. Gameplay often takes a back seat in game development, both in marketing and among developers, but remember that a neat engine never made a good game. Gamers have known this for years, and we developers should keep this in mind.

Note that the goals are quite broad, and should help everyone to get a quick overview of the project. More specific details are considered later on in the process.

The Team

The goals should be suitable and feasible. Therefore they should be based on the team, the deadlines, and the project. A number of useful questions to ask yourself are:

Who is going to work on the project?
How many people are available?
When will they be available?

If you work alone it is easier to keep an overview, but if there are more people, it makes it easier to keep up the spirit. Knowing that more people will join as the project goes along can also help to push you over the worst hurdles.

Is it a full-time or spare time project?
How much time will each member be willing to spend?

If you are a single programmer, and have a full-time job apart from programming, writing the next Unreal entirely from scratch does not sound that feasible.

What are the skills and the experiences of the team members?

If you have few artists and many programmers, writing a game with procedural graphics would make more sense than trying to make a new Final Fantasy. And if none of the team members have ever written any large programs, it might be better to aim for a small project.

The key thing is that you have to be realistic or, even better, pessimistic about your team's skills and available time. It is definitely better to make a simple game quickly than trying to do an epic tale with too few or too inexperienced people. Even highly professional development houses have sometimes managed to screw up here, causing years of delays and ending up releasing boring, uninteresting games.

Deadlines

The deadlines include the final version and the feature-complete alpha as well as demos or possible milestone requirements. Even if you don't work for a company, you should still set deadlines. With no clear deadlines it is easy to keep working on nitty-gritty details that have only a marginal impact on the project, but if you know when to be finished with each part, you can always come back to do fine polishing later.

Oh, by the way - learn to keep the deadlines! That would make a lot of people happy - including yourself.

The Project

What is the real goal of the project?
Is it just for fun, or do you want to sell it commercially?
Do you do it just to learn or to get a job in the business? Or do you just have an idea to try out?

If you do a project just to learn, there is no point in making 100 levels, when it is just the same tasks you will do over and over. And if you have an innovative gameplay idea that you want to try out, you should not have to worry about writing menu systems or cinematic routines.

Is the target format PC, Mac or a console? Is it cross-platform?

A simple way to avoid problems when working across multiple platforms is to design for the smallest target and simply scale up and add more detail on the more powerful machines. However, different machines have different weaknesses, and it is important to know these (some machines, for example, excel at polygon-pushing power, others have more memory, etc.).

What style of game will the engine be used for? How complex is the game? Is it very innovative, or a clone of an existing game?

If you're working on a very innovative game, it is very important to be able to get it to a playable state, but using simple boxes might work fine as graphics. A first-person shooter, on the other hand, follows a standard recipe, and in the beginning it is more important to see the level design in-engine than to tweak character control and gameplay. Similarly, for a complex RPG, there should be much more focus on setting up systems for the level designers to enter texts, weapon stats, enemy properties and so on than in a puzzle game.

Do you want to allow the mod community to add new stuff to it? In that case, you are limited from using any commercial package like 3D Studio Max, and you better have easy-to-use, bug-tested editors to go with the game. It is important to remember that the mod community consists of gameplayers, not professional game designers (yet.) Also try to make sure that the file formats are simple and documented, in case people want to write their own tools. And if you have a script system that the mod community has access to, make sure that it is very stable and works properly, even if it gets weird user input.

How many levels should be in the game?

The more levels, and the more complex they are, the more important it is to have a good set of tools to use. A 30-level Quake game will require more advanced tools than an outdoor game with three levels based on procedural geometry.

Still, do not underestimate the amount of time that is wasted due to inadequate tools. In my first game, Banshee (a 1942 clone), we used a terrible in-house level-editor on the Commodore Amiga for the enemies. However, since it was not possible to set any properties, the levels had to be converted to source code, copied to the PC, add data by hand within the source code, and then compile and test it. We estimated that we would probably have saved a month or two (out of 11 months in total) if I had written my own level editor.

Should the engine be used for only one game, multiple games in the same genre, or completely different types of games?

If you want to target your engine for use in multiple games, there should be as little gameplay-specific code in the engine as possible. If it will be used for just a single game, it might be possible to do some hardwired functions. As an example, in Banshee the player-characters used hard-coded functions, while my script language controlled the enemies. Still, I cannot recommend this practice, as it gets too messy.

An Example

What I've explained thus far should help you determine where your project is going. The next phase is to share this vision with all of the people in the project.

As an example, let's imagine a team consisting of four people: an artist/3D modeler, a level designer, an engine programmer, and a gameplay/editor programmer. They have limited experience, and they are all working in their spare time. Their goal is to make a simple game that can give them a job at a game company, and to release the game as freeware. Therefore there are no requirements for demo versions or milestones (except internally). The target machine is a 450 MHz PC with a TNT2 graphics card.

One realistic goal for our little team would be to create a simple single-player Terminal Velocity-style shoot-'em-up, with about 3-5 levels. The engine should not be developed with reuse in mind, and the levels should not be editable by players.

Now the goal is defined, and the features required by the engine can be determined. Again, it is very important to keep the skills of the team and the time constraints of the project in mind.

Some things to consider for your feature list are:

The Wish List

It is important to distinguish between what you need and what you would like. So I suggest making two lists: a feature list and a wish list. Even though it is often more fun to work on items in the wish list -- usually the eye candy -- wish list items should wait until the required features are implemented, or when you're waiting for other people to finish their parts.

Things that fundamentally change how the game is made, such as implementing skeletal animation or developing a scene editor, should not be on the wish list. These things change how the animators create game content, and their previous work will be wasted. However, a lot of the eye and ear candy, such as lens flares, 3D sound, rigid-body dynamics, and advanced particle systems can go into the wish list, as the game can still work fine without these items.

Network play is often put on the wish list. However, implementation of network playing will influence a large amount of the code, and greatly change the direction of the game. This might introduce a ton of new bugs and might require reworking the levels. In my opinion it's a bad idea to leave this question open for later discussion. Either networking is built-in from the start and all structures and routines are prepared for its implementation, or it is not there at all.

Content Path

It is a good exercise to consider how the game content is created and gets into the game. Preferably the artists, modelers, level designers, and gameplay programmers should have as much freedom to create the game with as little work for you as possible.

A typical dilemma is whether it is worth it to make a level editor, or if a tool such as 3D Studio Max should be used. It is much faster to write an exporter or a converter than to write a full editor, but in the editor it can be possible to see the level as it will end up in the game, and it will be easier to edit object properties, apply scripts and create new tests. Whether to build or buy again depends on the project, the team and the deadline at hand.

It's common for developers to have to wait for compilation, conversion or rendering tasks, and frequently developers have to perform tedious tasks which can introduce errors if not performed correctly. If possible, try to eliminate these tasks, or at least optimize them. For example, if your engine requires BSP-tree compilation, it should have an option to run an un-compiled version as well.

Another problem I have encountered is when artists depend on programmers to test their content in the game. This is annoying for the both artists and programmers, so try to design a system so that artists can test content in the game themselves as soon as possible. During the development of my first 3D game, Amok, the productivity increased by a factor of 3 to 4 when we got an extra development kit for testing the graphics and the levels.

The Example Returns

Let's return to our imaginary development team. Our little team has decided what the engine should do, and it has kept things simple to keep the deadline short and the goal easily reachable.

Their engine will require a landscape with subdivision. The landscape is based on a height image from a paint package, and the engine will select the textures procedurally. It will also contain 3D object drawings, with the objects modeled in Milkshape. All objects and textures are loaded into memory at the beginning, and no texture swapping will be needed.

The elements should be able to move and rotate in full 3D (six degrees of freedom). Collision will be sphere-sphere for checks against other elements, and sphere-polygon for checks against the landscape.

The game elements will be placed on the map in a simple top-down 2D editor. In this editor it will also be possible to set properties on each type of element, such as hit-points, sound effect and particle effects. The engine will be able to be started from the editor, to speed up testing of gameplay.

All gamecode will be written in C++, and the code for each element will be derived from a virtual class that is used to get information from the editor. The main debug functions will be a log-file and an overlaying text window for run-time information such as framerates and status flags.

The game will contain a particle system for explosions and other effects, and it will use MP3 for music and .WAV files played in simple stereo for sound effects. It will use a joystick for input. On the wish list are lens coronas, multiple sky layers, shadows on the landscape from the elements and a moving sun.

By now you know what your final goal is, and what your engine should do in order to reach this goal. Now it is time to get into the technical details, and define which subsystems and classes are needed, how they should work, and in what order they need to be written. Not everything needs to be described in full detail, but the more integral a subsystem is to the engine, the more important it is to lock down its functionality, or at least the interface.

There are three good reasons to spend time defining the system in detail before beginning.First, by scheduling the different tasks properly, it is possible to avoid people having to wait for each other to finish their tasks before being able to complete their own tasks. Bottlenecks can be discovered faster and circumvented, and you can see if the deadlines are achievable.

Second, a good schedule covering all tasks will also help the team to focus on what is actually important -the game - and not be fixated on working on the engine.

Third, there is a psychological factor: It seems much more manageable if there are a lot of small tasks rather than a few large ones, and it is easier to see progress occur.

How?

What I prefer to do is write down in detail which subsystems will be required, what their functionality is. Designing a directory structure and defining file types and their content is also useful.

Drawing charts showing how the different subsystems interact, and how the different classes and files are related can greatly help in giving an overview.

Recycle or Reinvent

One question to ask yourself is how much you want to use other people's work and how much you want to write from scratch. A lot of programmers take pride in doing as much as possible themselves, but again I think that it is more important to see what your goal is. If you are tring to impress a potential employer, I am sure he would be more impressed by seeing a programmer that gets the job done by any means possible than a programmer that can do everything himself but who takes much longer to do it. Sure, you learn a lot by doing everything yourself, but being able to work with other people's code is also an extremely valuable skill for today's developers.

For example, doing your own image loader will not help you to write a game faster, or improve the quality of the game. And unless you have some revolutionary ideas that no one has ever thought about before, a publicly available script system will perform almost as well as your hand-written one. True, you might get a little more performance and a little more user-friendliness by writing your own, but often the time is better spent elsewhere in the project.

Of course there are problems by using other people's libraries. Even if you can get hold of a preview version, the authors might want money if you use it commercially. This should not be a problem in our example case, as the team primarily wants to show its skills to potential employers.

Another problem is that the more the library is tied in with your engine, the more important it is to make sure that the library fits to your exact requirements, and the more important it is for you to tailor your engine to fit to the library. An MP3-player has very loose ties, and will easily fit into any engine, no matter how it works, but a physics library like Math Engine should be planned for from the beginning of the project. Once I spent days implementing a commercial library into my engine, just to find out that the way it did one certain thing was completely incompatible with my system

Debugging might also be a problem. Even though most of these libraries are well documented and thoroughly debugged, you might have crashes either caused by your own code or the library. But often the library source code is not released, and debugging gets practically impossible, and therefore it might be easier to implement your own system.

The question about what to do yourself and what to borrow from other people, like many other questions, depends on the team, the project and the deadlines. Again there is no clear answer.

The Schedule

Schedule important things to be done first. It sounds logical, but that is not always how it happens during a project, because different people have different ideas about what is important. The criterion that I like to use is whether people's productivity will go up when a subsystem is being worked on. I always have much of the ear- and eye candy at the very bottom of the list.

A good example is the rendering system. It is very important to get the objects on the screen quickly, both for the programmers to see if all of the code is working, but also for the artists to test the graphics and level designs. But on the other hand a feature like texture mapping is not very important at all. The gameplay programmer, the level designer and the modeller can still work fine without textures. Sure, it helps, but the productivity gains are marginal.

Another example is the collision. Sure, collision is necessary, but getting the scripting system up and running is much more important, in order for the gameplay programmer to start working on player control, weapons, enemies and levels. This is especially important if it is a new, innovative type of gameplay, where lots of tweaking is expected.

Your overall goal should be to get as much performance out of other people as possible, and to spot bottlenecks before they occur. For example, if it looks like it will take a while before your indoor routines will be ready, consider letting your artists start to create the indoor sections in a Quake editor, and test in Quake, while you work on the code.

Modularity

If you are not used to writing large programs, you might not know how you should split up the different subsections, as they naturally all work together in some way or other. But it is a good exercise to do it, even for simple projects, and it helps you a lot as the project grows.

To give an example from my current engine, I have an input class, which contains joystick, keyboard and mouse, but which also contains my random generator and frame rate measuring. My reason for doing this is that this is practically all I need to have a complete record and playback functionality, and I do not need to access many subsystems to save a demo file.

Of course what you decide to have in each subsystem is up to you. But if you are aiming for later converting to other platforms or APIs, you should not try to shape the subsystem after the features or structure of the primary API or platform.

Another example of modularity is a complete plug-in structure. In my current engine, drawing functionality has been separated from the core, and a new effect, such as lens flares, can be added simply by writing a DLL, and adding the new effect to the elements in the editor.

Keep it Simple, Stupid (KISS)

A simple reminder: the less code you write, the less bugs there will be. Before you add a feature, make sure that it will be used thoroughly. Writing lots of highly specific code usually means more bugs than a little, more general, code that requires some parameters to be set in the level editor. And since the general code is run more often, and with different input, it will be much more thoroughly tested than the specific code.

Once again, this means less work for yourself, but more work (and more freedom) to your creative people.

Keep it Stable, Stupid (KISS 2.0)

My experience is that it is much easier to fix bugs as soon as they occur. If many bugs are left, you never know if it is a known issue or something new that causes a crash. And never work around a bug that you can't find and hope that it will disappear over time. This kind of bug tends to settle down deeply nested in your code, marry and have lots of little baby bugs all over the place.

Keeping your code more bug-free is also a blessing when it comes to milestones and demos. The bugs always resurface five minutes before the fat man with the wallet enters the door.

If I had to sum up what I have been trying to say so far in four words, they would be: Think before you code!

Spend one or two days before you start to write a lot of code, and you will not get stuck as easily later on. And make sure that you get everything down on paper instead of keeping it in your head. This will you to keep focused and goal oriented. Always consider everything you do before you do it.

Be realistic! Know your own and your team's strengths and weaknesses. And do not aim too high.

Stick to the plan! Don't give up! Focus! Keep on! Be Persistent! Quitting is such an easy decision to make, but quitters are not winners. When Duke Nukem 3D was completed its technology was inferior to Quake, but the Duke Nukem crew still managed to make a better game because they kept working on and on. I know that you learn a lot and find better methods as you go through a project, but learning to accept slightly inferior code is a very important skill that will make you much more productive.

Don't just follow John Carmack and Tim Sweeney's technology plans. Learn from others, but don't copy blindly. They work 80 hours a week, with twice the efficiency of the rest of us, and have a few extra programmers to assist them. If the rest of us had to follow them we would never finish anything. Try to do your own things instead - seek out a project that is right for you.