Teaching an Old Dog New Bits: How Console Developers are Able to Improve Performance When the Hardware Hasn't Changed

Editor's note: This paper was originally published in the 1999 Game Developer's Conference proceedings.

Personal computers and video game consoles have both made tremendous strides in graphics and audio performance; however, despite these similarities there is a tremendous benefit in understanding some important differences between these two platforms.

Evolution is a good thing, right?

The ability to evolve is the cornerstone behind the long-term success of the IBM PC. Tremendous effort has been taken on the PC so that individual components of the hardware could be replaced as they became inefficient or obsolete, while still maintaining compatibility with existing software. This modularity of the various PC components allows the user to custom build a PC to fit specific needs. While this is a big advantage in general, this flexibility can be a tremendous disadvantage for developing video games. It is the lack of evolution; the virtual immutability of the console hardware that is the greatest advantage to developing high quality, easy to use video game software.

You can choose any flavor, as long as it's vanilla

The price of the PC's evolutionary ability comes at the cost of dealing with incompatibility issues through customized drivers and standardization. In the past, it was up to the video game developer to try to write custom code to support as many of the PC configurations as possible. This was a time consuming and expensive process, and regardless of how thorough the developer tried to be, there were also some PC configurations that still had compatibility problems. With the popularity of Microsoft's window based operating systems, video game developers have been given the more palatable option of allowing other companies to develop the drivers and deal with the bulk of the incompatibility issues; however, this is hardly a panacea, since this necessitates a reliance on "unknown" and difficult to benchmark code, as well as API's that are designed more for compatibility than optimal performance. The inherit cost of compatibility is compromise. The API code must compromise to support the largest amount of hardware configurations, and likewise, hardware manufacturers make compromises in their hardware design in order to adapt well to the current standards of the API. Also, both the API and the hardware manufacturers have to compromise because of the physical limitations of the PC's hardware itself, such as bus speed issues.

Who's in charge here?

The operating system of a PC is quite large and complicated, and is designed to be a powerful and extensively featured multi-tasking environment. In order to support a wide variety of software applications over a wide range of computer configurations, the operating system is designed as a series of layers that distance the software application from the hardware. These layers of abstraction are useful for allowing a software application to function without concerning itself with the specifics of the hardware. This is an exceptionally useful way of maintaining compatibility between hardware and software, but is unfortunately not very efficient with respect to performance. The hardware of a computer is simply a set of interconnected electronic devices. To theoretically maximize the performance of a computer's hardware, the software application should write directly to the computer's hardware, and should not share the resources of the hardware, including the CPU, with any other applications. This would maximize the performance of a video game, but would be in direct conflict with the implementations of today's modern PC operating systems. Even if the operating system could be circumvented, it would then fall upon the video game to be able to support the enormous variety of hardware devices and possible configurations, and would therefore be impractical.

It looked much better on my friend's PC

Another problem with having a large variety of hardware is that the video game developer cannot reliably predict a user's personal set-up. This lack of information means that a game can not be easily tailored to exploit the strengths and circumvent the weaknesses of a particular system. For example, if all PC's had hard-drives that were all equally very fast, then a game could be created that relied on having a fast hard-drive. Similarly, if all PC's had equally slow hard-drives, but had a lot of memory, then a game could compensate for the lack of hard-drive speed through various techniques, such as caching data in RAM or pre-loading data into RAM. Likewise, if all PC's had fast hard-drives, and not much memory, then the hard-drive could compensate for the lack of much memory by keeping most of the game on the hard-drive, and only spooling in data as needed.

Another good example is the difference between polygon rendering capabilities. There is an enormous variation in both performance and effects between hardware assisted polygonal rendering, such that both the look of rendered polygons and the amount of polygons that can be rendered in a given amount of time can vary greatly between different machines. The look of polygons could be made consistent by rendering the polygons purely through software, however, the rendering of polygons is very CPU intensive, so may be impractical since less polygons can be drawn, and the CPU has less bandwidth to perform other functions, such as game logic and collision detection. Other bottlenecks include CD drives, CPU speeds, co-processors, memory access speeds, CPU caches, sound effect capabilities, music capabilities, game controllers, and modem speeds to name a few.

Although many PC video game programmers have made valiant attempts to make their games adapt at run-time to the computers that they are run on, it is difficult for a developer to offer much more than simple cosmetic enhancements, audio additions, or speed improvements. Even if the developer had the game perform various benchmark tests before entering the actual game code, it would be very difficult, and not to mention limiting to the design of a game, for the developer to write code that could efficiently structurally adapt itself to the results of the benchmark.

Which button fires?

A subtle, yet important problem is the large variety of video game controllers that have to be supported by the PC. Having a wide variety of game controllers to choose from may seem at first to be a positive feature since having more seems like it should be better than having less, yet this variety actually has several negative and pervasive repercussions on game design. One problem is that the game designer can not be certain that the user will have a controller with more than a couple of buttons. Keys on the keyboard can be used as additional "buttons", but this can be impractical or awkward for the user, and also may require that the user configure which operations are mapped to the buttons and keys. Another problem is that the placement of the buttons with respect to each other is not known, so the designer doesn't know what button arrangement is going to give the user the best gameplay experience. This problem can be somewhat circumvented by allowing the user to remap the actions of the buttons, but this isn't a perfect solution since the user doesn't start out with an inherent knowledge of the best way to configure the buttons, so may choose and remain using an awkward button configuration. Also, similar to the button layout, the designer doesn't know the shape of the controller, so can't be certain what types of button or controller actions might be uncomfortable to the user.

An additional problem associated with game controllers on the PC is that most PC's that are sold are not bundled with a game controller. This lack of having a standard, bundled controller means that video game on the PC should either be designed to be controlled exclusively by the keyboard, or at the very least should allow the user to optionally use a keyboard rather than a game controller. Not allowing the use of the keyboard reduces the base of users that may be interested in buying your game, but allowing the game to be played fully using the keyboard will potentially limit the game's controls, and therefore limit the game's overall design.

Of course, even if every PC did come bundled with a standard game controller, there would still be users who would want to use their own non-standard game controllers. The difference, however, is that the non-standard game controllers would either be specific types of controllers, such as a steering wheel controller, or would be variations of the standard game controller, and would therefore include all of the functionality of the original controller. The decision to use the non-standard controller over the standard controller would be conscious decision made by the user, rather than an arbitrary decision made because there is no standard.

Chasing a moving target

Another problem associated with the PC's evolutionary ability is that it is difficult to predict the performance of the final target platform. The development of video games has become an expensive and time consuming endeavor, with budgets in the millions, and multi year schedules that are often unpredictable. The PC video game developer has to predict the performance of the target machine far in advance of the release of the game, which is difficult indeed considering the volatility of schedules and the rapid advancements in technology. Underestimating the target can cause the game to seem dated or under-powered, and overestimating the target could limit the installed base of potential consumers. Both could be costly mistakes.

Extinction vs. evolution

While PC's have become more powerful through continual evolution, video game consoles advance suddenly with the appearance of an entirely new console onto the market. As new consoles flourish, older consoles eventually lose popularity and fade away. The life cycle of a console has a clearly defined beginning: the launch of the console into the market. The predicted date of the launch is normally announced well in advance of the launch, and video game development is begun early enough before the launch so that at least a handful of video game titles will be available when the console reaches the market. The end of a console's life cycle is far less clearly defined, and is sometimes defined to be the time when the hardware developer of the console announces that there will no longer be any internal support for that console. A more practical definition is that the end of a console's life cycle is when the public quits buying much software for that console. Of course, the hardware developer would want to extend the life cycle of a console for as long as possible, but stiff competition in the market has caused hardware developers to often follow up the launch of a console by immediately working on the design of the next console.

Each and every one is exactly the same

Unlike PC's which can vary wildly from computer to computer, consoles of a particular model are designed to be exactly the same. Okay, so not exactly the same, but close enough that different revisions between the hardware generally only vary in minor ways that are usually pretty minor from the perspective of the video game developer, and are normally transparent to the user. Also, the console comes with at least one standard game controller, and has standardized peripheral connections.

The general premise is that game software can be written with an understanding that the base hardware will remain consistent throughout the life-span of the console; therefore, a game can be tailored to both exploit the strengths of the hardware, and to circumvent the weaknesses.

The consistency of the hardware components allows a console to have a very small, low level operating system, and the video game developer is often given the ability to either talk to the hardware components directly, or to an extremely low hardware abstraction layer.

The performance of the components of the hardware is virtually identical for all consoles of a given model, such that the game will look the same and play the same on any console. This allow the video game developer to design, implement, and test a video game on a small number of consoles, and be assured that the game will play virtually the same for all consoles.

And finally to the point!

Both the rapid lifecycle of a video game console and the consistency of the hardware promote video game development strategies that are often very different from the strategies used to make PC video games. A side-effect of these strategies and the console development environment is that video games released later in the life of a console tend to be incrementally more impressive than earlier titles, even though the hardware hasn't changed. Theoretically, since the hardware doesn't change, first generation software can be as equally impressive as later generation titles, but in reality this is seldom the case. It may seem obvious that a developer should try to make a first generation title as impressive as a last generation title, but actually this strategy has been the downfall of many talented developers. There are many good and valid reasons why software improves over time, and the understanding and strategizing about these reasons can greatly improve the chances for a developer to be successful in the marketplace.

Difficulties of Console Video Game Development

There are many difficulties that are encountered when developing a console video game, but the following is a list of several major issues:

Learning curve

The learning curve may be the most obvious of all difficulties, and is often one of the most disruptive elements of a video game's development schedule. In the past, video games were often developed by small groups of one or more people, had small budgets, ran in a small amount of memory, and had short schedules. The graphics were almost always 2D, and the mathematics of the game were rarely more than simple algebra. Today, video games have become much more complicated, and often require extremely sophisticated algorithms and mathematics. Also, the pure size of the data within a game has made both the run-time code and the tool pipeline require extremely sophisticated solutions for data management issues. Furthermore, 3D mathematics and renderings can be very CPU intensive, so new tricks and techniques are constantly being created. Also, the developer will often have to use complex commercial tools, such as 3D modeling packages, to generate the game's graphics and data. Add into this the fact that Operating Systems, API's, and hardware components are continually changing, and it should be obvious that just staying current with the latest technology requires an incredible amount of time and can have a big impact on the schedule of a game.

The console video game developer has the additional burden that, unlike the PC where the hardware evolves more on a component or API level, new console hardware is normally drastically different and more powerful than the preceding hardware. The console developer has to learn many new things, such as new CPU's, new operating systems, new libraries, new graphics devices, new audio devices, new peripherals, new storage devices, new DMA techniques, new co-processors, as well as various other hardware components. Also, the console developer usually has to learn a new development environment, including a new C compiler, a new assembler, a new debugger, and slew of new support tools. To complicate matters, new consoles normally have many bugs in such things as the hardware, the operating system, the software libraries, and in the various components of the development environment.

The learning curve of the console hardware is logarithmic in that it is very steep at first, but tends to drop off dramatically by the end of the console life-span. This initial steep learning curve is why often the first generation software isn't usually as good as later software.

Hardware availability and reliability

Hardware isn't very useful without software, and software takes a long time to develop, so it is important to hardware developers to try to encourage software developers to begin software development well in advance of the launch date of the hardware. It is not uncommon for developers to begin working on a title even before the hardware development kits are available. To do this, developers will start working on things that don't depend on the hardware, such as some common tools, and they may also resort to emulating the hardware through software emulation. Obviously, this technique is not likely to produce software that maximizes the performance of the hardware, but it is done nevertheless because of the time constraints of finishing a product as close as possible to the launch of the console into the market. The finished first generation game's performance is not going to be as good as later generations of games, but this compromise is deemed acceptable in order to achieve the desired schedule.

When the hardware does become available for developers, it is usually only available in limited quantity, is normally very expensive, and eventually ends up being replaced by cheaper and more reliable versions of the hardware at some later time. Early revisions of the hardware may not be fully functional, or may have components that run at a reduced speed, so are difficult to fully assess and are quite scarce since the hardware developer doesn't want to make very many of them. Even when more dependable hardware development kits become available, they are usually difficult to get, since production of these kits is slow and expensive, quantities are low, and software developers are in competition to get them.

The development kits, especially the initial hardware, tend to have bugs that have to be worked around or avoided. Also, the hardware tends to have contact connection problems so that it is susceptible to vibrations, oxidation, and overheating. These problems generally improve with new revisions of the development hardware.

All of these reasons will contribute to both a significant initial learning curve, and a physical bottleneck of having an insufficient number of development kits. This will have a negative impact on a game's schedule, and the quality of first generation software often suffers as a consequence.

Bottlenecks

An extremely important aspect to console game development is the analysis of the console's bottlenecks, strengths, weaknesses, and overall performance. This is critical for developing high performance games, since each component of the console has a fixed theoretical maximum performance, and undershooting that performance may cause your game to appear under-powered, while overshooting may cause you to have to do major reworking of the game's programming and/or design. Also, overshooting performance may cause the game to run at an undesirable frame rate, which could compromise the look and feel of the game.

The clever developer will try to design the game to exploit the strengths of the machine, and circumvent the weaknesses. To do this, the developer must be as familiar as possible with the limitations of the machine. First, the developer will look at the schematic of the hardware to find out the documented sizes, speeds, connections, caches, and transfer rates of the hardware. Next, the developer should do hands-on analysis of the machine to look for common weaknesses such as: slow CPU's, limited main memory, limited video memory, limited sound memory, slow BUS speeds, slow RAM access, small data caches, small instruction caches, small texture caches, slow storage devices, slow 3D math support, slow interrupt handling, slow game controller reading, slow system routines, and slow polygon rendering speeds. Some of these things are easy to analyze, such as the size of video memory, but some of these things are much trickier, such as polygon rendering speeds, because the speed will vary based on many factors, such as source size, destination size, texture bit depth, caching, translucency, and z-buffering, to name just a few. The developer will need to write several pieces of test code to study the performance of the various hardware components, and should not necessarily trust the statistics found in the documentation, since these are often wrong or misleading.

A developer should use a profiler to analyze where speed losses are occurring in the run-time code. Most programmers will spend time optimizing code because the programmer suspects that code is slow, but doesn't have any empirical proof. This lack of empirical data means that the programmer will invariable waste a lot of time optimizing things that don't really need to be optimized, and will not optimize things that would have greatly benefited form optimization. Unfortunately, a decent profiler is almost never included in the development software, so it is usually up to the individual developer to write his own profiling software.

The testing of performance is an extremely important tool to use in order to maximize performance. Often the reason why software improves between generations is that the developers slowly learn over time how to fully understand the bottlenecks, how to circumvent the bottlenecks, and how to identify what actually constitutes a bottleneck.

Operating System / Libraries

Although the consoles tend to have very small operating system and libraries when compared to the operating systems found on the PC, they are still an important factor of console video game development.

Operating systems and support libraries on video game consoles are used to fill many needs. One such need is that the hardware developer will often attempt to save money on the production of console hardware by switching to cheaper components, or by integrating various components together. It is up to the operating system to enable these changes, while having the effects of these changes be transparent to both the consumer and the developer. The more that the operating system abstracts the hardware, the easier it is for the hardware developer to make changes to the hardware. However, remember that this abstraction of the hardware comes at the price of reduced potential performance. Also, the operating system and support libraries will commonly provide code for using the various components of the console. This has the advantage that developers don't have to know the low-level details of the hardware, and also potentially saves time since different developers won't have to spend time creating their own versions of these libraries. The advantage of not having to write this low level code is important in early generation projects, because the learning curve for the hardware is already quite high, and there may not be time in the schedule for doing very much of this kind of low-level optimization. Clever developers will slowly replace the system libraries over time, especially with the speed critical subroutines, such as 3D vector math and polygonal set-up. Also, the hardware developer will occasionally improve upon poorly written libraries, so that even the less clever developers will eventually benefit from these optimizations. Improvements to the system libraries are a big reason why later generation games can increase dramatically in performance.

Development tools

On the PC, development tools have evolved over the years, and have become quite sophisticated. Commercial companies have focused years of efforts on making powerful, optimal, polished, and easy to use development tools. In contrast, the development tools provided for console video game development are generally provided by the hardware manufacturers, and are usually poorly constructed, have many bugs, are difficult to use, and do not produce optimal results. For example, the C compiler usually doesn't optimize very well; the debugger is often crude and, ironically, has many bugs; and there usually isn't a decent software profiler.

Initially developers will rely on these tools, and the first few generations of software will be adversely effected by their poor quality. Over time, clever programmers will become less reliant on the tools that are provided, or will develop techniques to work around the weaknesses of the tools.

In-house tools

In-house tools are one of the most important aspects of producing high performance console video game software. Efficient tools have always been important, but as the data content in video games has grown exponentially over the last few years, in-house tools have become increasingly more important to the overall development process. In the not too distant future, the focus on tool programming techniques may even exceed the focus on run-time programming issues. It is not unreasonable that the most impressive video games in the future may end up being the ones that have the best support tools.

In-house tools tend to evolve to fill the needs of a desired level of technology. Since new consoles tend to have dramatic changes in technology over the predecessor consoles, in-house tools often have to be drastically rewritten or completely replaced to support the new level of technology. For example, a predecessor console may not have had any 3D support, so the tools developed for that console most likely would not have been written to support 3D. When a new console is released that can draw 100,000 polygons per second, then it is generally inefficient to try to graft support for this new technology onto the existing tools, so the original tools are discarded. To continue the previous example, let's say that the new tool needs to be able to handle environments in the game that average about 500,000 polygons, and have a maximum worst case of 1 million polygons. Most likely the tool will evolve to the point where it runs pretty well for environments of the average case, but will most likely run just fast enough that the slowest case of a 1 million polygons is processed in a tolerable, albeit painful, amount of time. The reasons for this are that tools tend to grow in size and complexity over time, and tools tend to only be optimized to the point that they are not so slow as to be intolerable. Now let's say that a newer console is released that can now draw 1 million polygons a second, and now our worst case environment is a whopping 1 billion polygons! Although theprevious in-house tool could support a lot of polygons, the tool will still end up being either extensively rewritten or discarded, since the tool will not be able to be easily modified to be efficient enough to deal with this much larger amount of polygons.

The ability of a tool to function efficiently as the data content processed by the tool increases is referred to as the ability of the tool to "scale." In video game programming, tools are seldom written to scale much beyond the needs of the current technology; therefore, when technology changes dramatically, old tools are commonly discarded, and new tools have to be developed.

The in-house tools can consume a large amount of the programming time of a first generation title, since not only are the tools complicated, but they evolve over time as the run-time game code is implemented. Initial generations of games are created using initial generations of tools. Likewise, later generations of games are created using later generations of tools. As the tools become more flexible and powerful, the developer gains the ability to create more impressive games. This is a big reason why successive generations of console games often make dramatic improvements in performance and quality over their predecessors.

Reuse of code

A problem that stems from the giant gaps in technology between console generations is that it makes it difficult to reuse code that was written for a previous generation of console hardware. Assembly programming is especially difficult to reuse since the CPU usually changes between consoles, but the C programming language isn't much of a solution either, since the biggest problem is that the hardware configurations and capabilities are so different. Any code dealing directly with the hardware or hardware influenced data structures will have to be discarded. Even code that does something universal in nature, such as mathematical calculations, will most likely need to be rewritten since the new hardware will most likely have some sort of different mathematical model.

Also, just as the in-house tool code becomes outdated, so does game code that is written for less powerful technology. Animation, modeling, character, environment, and particle code will all need to be discarded.

In practice, very little code can be reused between technological leaps in hardware platforms. This means that earlier generation games will not have much code reuse, but each new generation of games for a console will be able to reuse code from its predecessors, and therefore games will tend to improve with each generation.

Optimization

By definition, having optimal code is preferable to having bulky or less efficient code. It would therefore seem logical to say that to achieve maximum performance form the hardware, all code should be completely optimal. Unfortunately, this is not an easy or even practical thing to achieve, since the writing of completely optimal code has many nuances, and can be very time-consuming. The programmer must be intimately familiar with the details of the hardware. He must fully understand how to implement the code, such as possibly using assembly language since C compilers will often generate inefficient code. The programmer must make certain to best utilize the CPU caches. Also, the programmer should understand how the code may effect other pieces of code, such as the effects of the code on the instruction cache, or the amount of resources that are tied up by his code. The programmer has to know how to effectively use co-processors or other devices. He must develop an algorithm that is maximally efficient when implemented. Also, the programmer will need to measure the code against the theoretical maximum optimal performance to be certain that the code can indeed be considered to be fully optimal.

Writing even highly optimized code for specific hardware is time-consuming, and requires a detailed knowledge of both the hardware and the algorithm to be optimized. It is therefore commonly impractical to attempt to highly optimize even a majority of the code. This is especially true when writing a first generation game, since the developer is not familiar enough with the intricacies of the hardware to be very productive at writing optimal code. Instead, it is more productive to only spend time optimizing the code that most profoundly affects the efficiency of the overall game. Unfortunately, identifying what code should be optimized can also be a difficult task. As a general rule, the code to be optimized is often the code that is executed most frequently, but this is not always the case. Performance analyzing, testing, and profiling can help identify inefficient code, but these are also not perfect solutions and the experience of the programmer becomes an important factor in making smart decisions concerning what code should be optimized.

As a programmer gets more familiar with the intricacies of the hardware, he will be able to perform a greater amount of optimizations. Also, when developing later generation games, the programmer will often be able to reuse previously written optimized code. Plus there is often more time in the schedule of later generation titles in which to perform optimizations. This accumulation of optimal code is a big reason why games often improve in performance in successive generations.

Other Considerations

There are many other reasons to explain the improvement in performance of next generation software that are not directly related to programming for a video game console. For example, developers will often copy or improve upon the accomplishments of other developers. Likewise, developers will avoid the mistakes made by others. Also, developers acquire and lose employees fairly frequently, which creates a lot of cross-pollination of ideas and techniques between the various development houses. These and many other reasons are important, but since they are not specific to console video game development, they have not been specifically discussed.

The Crash Bandicoot Trilogy: A Practical Example

The three Crash Bandicoot games represent a clear example of the process of technology and gameplay refinement on a single platform. Crash Bandicoot was Naughty Dog's first game on the Sony Playstation game console, and its first fully 3D game. With Crash Bandicoot 2: Cortex Strikes Back and Crash Bandicoot: Warped, we were able to improve the technology and offer a slicker, more detailed game experience in successively less development time. With the exception of added support for the Analog Joystick, Dual Shock Controller, and Sony Pocketstation the hardware platforms for the three titles are identical.

Timely and reasonably orderly development of a video game title is about risk management. Given that you have a certain amount of time to develop the title, you can only allow for a certain quantity of gameplay and technology risks during the course of development. One of the principle ways in which successive games improve is by the reuse of these risks. Most solutions which worked for the earlier game will work again, if desired, in the new game. In addition, many techniques can be gleaned from other games on the same machine that have been released during the elapsed time.

In the case of sequels such as the later Crash games there is even more reduction of risk. Most gameplay risks, as well as significant art, code, and sound can be reused. This allows the development team to concentrate on adding new features, while at the same time retaining all the good things about the old game. The result is that sequels are empirically better games.

Crash Bandicoot - how do we do character action in 3D?

Development: September 1994 - September 1996
Staff: 9 people: 3 programmers, 4 artists, 1 designer, 1 support

Premise: Do for the ultra popular platform action game genre what Virtua Fighter had done for fighting games: bring it into 3D. Design a very likeable broad market character and place him in a fun and fast paced action game. Attempt to occupy the "official character" niche on the then empty Playstation market. Remember, that by the fall of 1994 no one had yet produced an effective 3D platform action game.

Gameplay risk: How do you design and control an action character in 3D such that the feel is as natural and intuitive as in 2D?

When we first asked ourselves, "what do you get if you put Sonic the Hedgehog (or any other character action game for that matter) in 3D," the answer that came to mind was: "a game where you always see Sonic's Ass." The entire question of how to make a platform game in 3D was the single largest design risk on the project. We spent 9 months struggling with this before there was a single fun level. However, by the time this happened we had formulated many of the basic concepts of the Crash gameplay.

We were trying to preserve all of the good elements of classic platform games. To us this meant really good control, fast paced action, and progressively ramping challenges. In order to maintain a very solid control feel we opted to keep the camera relatively stable, and to orient the control axis with respect to the camera. Basically this means that Crash moves into the screen when you push up on the joypad. This may seem obvious, but it was not at the time, and there are many 3D games which use different (and usually inferior) schemes.

Technical risk: How do you get the Playstation CPU and GPU to draw complex organic scenes with a high degree of texture and color complexity, good sorting, and a solid high resolution look?

It took quite a while, a few clever tricks, and not a little bit of assembly writing and rewriting of the polygon engines. One of our major realizations was that on a CD based game system with a 33mhz processor, it is favorable to pre-compute many kinds of data in non real-time on the faster workstations, and then use a lean fast game engine to deliver high performance.

Technical risk: How do the artists build an maintain roughly 1 million polygon levels with per poly and per vertex texture and color assignment?

The challenge of constructing large detailed levels turned out to be one of the biggest challenges of the whole project. We didn't want to duplicate the huge amount of work that has gone into making the commercial 3D modeling packages, so we chose to integrate with one of them. We tried Softimage at first, but a number of factors caused us to switch to Alias Power Animator. When we began the project it was not possible to load and view a one million polygon level on a 200mhz R4400 Indigo II Extreme. We spent several months creating a system and tools by which smaller chunks of the level could be hierarchically assembled into a larger whole.

In addition, the commercial packages were not aware that anyone would desire per polygon and per vertex control over texture, color, and shading information. They used a projective texture model preferred by the film and effects industry. In order to maximize the limited amount of memory on the Playstation we knew we would need to have very detailed control. So we created a suite of custom tools to aid in the assignment of surface details to Power Animator models. Many of these features have since folded into the commercial programs, but at the time we were among the first to make use of this style of model construction.

Technical risk: How do you get a 200mhz R4400 Indigo II to process a 1 million polygon level?

For the first time in our experience, it became necessary to put some real thought into the design of the offline data processing pipeline. When we first wrote the level processing tool it took 20 hours to run a small test case. A crisis ensued and we were forced to both seriously optimize the performance of the tool and multithread it so that the process could be distributed across a number of workstations.

Conventional wisdom says that game tools are child's play. Historically speaking, this is a fair judgment - 2D games almost never involve either sophisticated preprocessing or huge data sets. But now that game consoles house dedicated polygon rendering hardware, the kid gloves are off.

In Crash Bandicoot, players explore levels composed of over a million polygons. Quick and dirty techniques that work for smaller data sets (e.g., repeated linear searches instead of binary searches of has table lookups) no longer suffice. Data structures now matter - choosing one that doesn't scale well as the problem size increases leads to level processing tasks that take hours instead of seconds.

The problems have gotten correspondingly harder, too. Building an optimal BSP tree, finding ideal polygon strips, determining the best way to pack data into fixed-size pages for CD streaming - these are all tough problems by any metric, academic or practical.

To make matters worse, game tools undergo constant revisions as the run-time engine evolves towards the bleeding edge of available technology. Unlike many jobs, where programmers write functional units according to a rigid a priori specification, games begin with a vague "what-if" technical spec - one that inevitably changes as the team learns how to best exploit the target machine for graphics and gameplay.

The Crash tools became a test bed for developing techniques for large database management, parallel execution, data flexibility, and complicated compression and bin packing techniques.

Art / Technical risk: How do you make low poly 3D characters that don't look like the "Money for Nothing" video?

From the beginning, the Crash art design was very cartoon in style. We wanted to back up our organic stylize environments with highly animated cartoon characters that looked 3D, but not polygonal. By using a single skinned polygonal mesh model similar to the kind used in cutting edge special effects shots (except with a lot less polygons), we were able to create a three dimensional cartoon look. Unlike the traditional "chain of sausages" style of modeling, the single skin allows interesting "squash and stretch" style animation like that in traditional cartoons.

By very careful hand modeling and judicious use of both textured and shaded polygons, we were able to keep these models within reasonable polygon limits. In addition, it was our belief that because Crash was the most important thing in the game, he deserved a substantial percentage of the game's resources. Our animation system allows Crash to have unique facial expressions for each animation, helping to convey his personality.

Technical risk: How do you fit a million polygons, tons of textures, thousands of frames of animation, and lots of creatures into a couple megs of memory?

Perhaps the single largest technical risk of the entire project was the memory issue. Although there was a plan from the beginning, this issue was not tackled until February of 1996. At this point we had over 20 levels in various stages of completion, all of which consumed between 2 and 5 megabytes of memory. They had to fit into about 1.2 megabytes of active area.

At the beginning of the project we had decided that the CD was the system resource least likely to be fully utilized, and that system memory (of various sorts) was going to be one of the greatest constraints. We planned to trade CD bandwidth and space for increased level size.

The Crash series employs an extremely complicated virtual memory scheme which dynamically swaps into memory any kind of game component: geometry, animation, texture, code, sound, collision data, camera data, etc. A workstation based tool called NPT implements an expert system for laying out the disk. This tool belongs to the class of formal Artificially Intelligence programs. Its job is to figure out how the 500 to 1000 resources that make up a Crash level can be arranged so as to never have more than 1.2 megabytes needed in memory at any time. A multithreaded virtual memory implementation follows the instructions produced by the tool in order to achieve this effect at run time. Together they manage and optimize the essential resources of main, texture, and sound RAM based on a larger CD based database.

Technical/Design risk: What to do with the camera?

With the 32 bit generation of games, cameras have become a first class character in any 3D game. However, we did not realize this until considerably into the project. Crash represents our first tentative stab at how to do an aesthetic job of controlling the camera without detracting from gameplay. Although it was rewritten perhaps five times during the project, the final camera is fairly straightforward from the perspective of the user. None of the crop of 1995 and 1996 3D action games played very well until Mario 64 and Crash. These two games, while very different, were released within two months of each other and we were essentially finished with Crash when we first saw Mario. Earlier games had featured some inducement of motion sickness and a difficulty for the players in quickly judging the layout of the scene. In order to enhance the tight, high impact feel of Crash's gameplay, we were fairly conservative with the camera. As a result, Crash retains the quick action feel of the traditional 2D platform game more faithfully than other 3D games.

Technical risk: How do you make a character collide in a reasonable fashion with an arbitrary 3D world… at 30 frames a second?

Another of the game's more difficult challenges was in the area of collision detection. From the beginning we believed this would be difficult, and indeed it was. For action games, collision is a critical part of the overall feel of the game. Since the player is looking down on a character in the 3rd person he is intimately aware when the collision does not react reasonably.

Crash can often be within a meter or two of several hundred polygons. This means that the game has to store and process a great deal of data in order to calculate the collision reactions. We had to comb through the computer science literature for innovative new ways of compressing and storing this database. One of our programmers spent better than six months on the collision detection part of the game, writing and rewriting the entire system half a dozen times. Finally, with some very clever ideas and a lot of hacks, it ended up working reasonably well.

Technical risk: How do you program, coordinate, and maintain the code for several hundred different game objects?

Object control code, which the gaming world euphemistically calls AI, typically runs only a couple of times per frame. For this kind of code, speed of implementation, flexibility, and ease of later modification are the most important requirements. This is because games are all about gameplay, and good gameplay only comes from constant experimentation with and extensive reworking of the code that controls the game's objects. The constructs and abstractions of standard programming languages are not well suited to object authoring, particularly when it comes to flow of control and state.

For Crash Bandicoot we implemented GOOL (Game Oriented Object LISP), a compiled language designed specifically for object control code that addresses the limitations of traditional languages.

Having a custom language whose primitives and constructs both lend themselves to the general task (object programming), and are customizable to the specific task (a particular object) makes it much easier to write clean descriptive code very quickly. GOOL makes it possible to prototype a new creature or object in as little as 10 minutes. New things can be tried and quickly elaborated or discarded. If the object doesn't work out it can be pulled from the game in seconds without leaving any hard to find and wasteful traces behind in the source. In addition, since GOOL is a compiled language produced by an advanced register coloring compiler with reductions, flow analysis, and simple continuations, it is at least as efficient as C, more so in many cases because of its more specific knowledge of the task at hand. The use of a custom compiler allowed us to escape many of the classic problems of C.

Crash Bandicoot 2: Cortex Strikes Back - Bigger and Badder!

Development: October 1996 - November 1997
Staff: 14 people: 4 programmers, 6 artists, 1 designer, 3 support

Premise: Make a sequel to the best selling Crash Bandicoot that delivered on all the good elements of the first game, was well as correcting many of our mistakes. Increasing the technical muscle of the game, and improving upon the gameplay, all without "been there done that…" in one year.

For Crash 2 we rewrote approximately 80% of the game engine and tool code. We did so module by module in order to allow continuous development of game levels. Having learned during Crash 1 about what we really needed out of each module we proceeded to rewrite them rapidly so that they offered greater speed and flexibility.

Technical risk: A fancy new tools pipeline designed to deal with a constantly changing game engine?

The workstation based tools pipeline was a crucial part of Crash 1. However, at the time of its original conception, it was not clear that this was going to be the case. The new Crash 2 tools pipe was built around a consistent database structure designed to allow the evolution of level databases, automatic I/O for complex data types, data browsing and searching, and a number of other features. The pipe was modularized and various built-in restrictions were removed. The new pipe was able to support the easy addition of arbitrary new types of data and information to various objects without outdating old information.

We could never have designed such a clean tool program that would be able to handle the changes and additions of Crash 2 and Warped at the beginning of the first game. Being aware of what was needed at the start of the rewrite allowed us to design a general infrastructure that could support all of the features we had in mind. This infrastructure was then flexible enough to support the new features added to both sequels.

Technical/process risk: The process of making and refining levels took too long during the first game. Can we improve it?

The most significant bottleneck in making Crash 1 was the overall time it took to build and tune a level. So for Crash 2 we took a serious look at this process and attempted to improve it. For the artists, the task of surfacing polygons (applying texture and color) was very time consuming. Therefore, we made improvements to our surfacing tools.

For both the artists and designers, the specification of different resources in the level was exceedingly tedious. So we added a number of modules to the tools pipeline designed to automatically balance and distribute many of these resources, as well as to auto calculate the active ranges of objects and other resources that had to be controlled manually in the first game. In addition, we moved the specification of camera, camera info, game objects, and game object info into new text based configuration files. These files allowed programmers and designers to edit and add information more easily, and it also allowed the programmers to add new kinds of information quickly and easily.

The result of this process was not really that levels took any less time to make, but that the complexity allowed was several times that of the first game. Crash 2 levels are about twice as large, have integrated bonus levels, multiple branches, "hard paths," and three or four times as many creatures, with an order of magnitude more settable parameters. The overall turn around time for changing tunable level information was brought down significantly.

Technical/Design risk: Can we make a better, more flexible camera?

The camera was one the of the things in Crash 1 with which we were least satisfied. So in order to pen up the game and make it feel more lifelike, we allowed the camera to look around much more and supported a much wider set of branching and transition cameras. In addition, arbitrary parameterized information was added to the camera system so that at any location the camera had more than 100 possible settable options.

If the two games are compared side by side, it can be seen that the overall layouts of Crash 2 levels are much larger and more complicated. The camera is more natural and fluid, and there are numerous dynamic camera transitions and effects which were not present in the first game. Even though the Crash 2 camera was written entirely from scratch, the lessons learned during the course of Crash 1 allowed it to be more sophisticated and aggressive, and it executed faster than its predecessor.

Optimization risk: Can we put more on screen?

Crash 1 was one of the fastest games of its generation, delivering high detail images at 30 frames per second. Nevertheless, for Crash 2 we wanted to put twice as much on screen, yet still maintain that frame-rate. In order to achieve this goal we had one programmer doing nothing but re-coding areas of the engine into better assembly for the entire length of the project. Dramatically increasing performance does not just mean moving instructions around; it is a complex and involved process. First we study the performance of all relevant areas of the hardware in a scientific and systematic fashion. Profiles are made of cache latencies, coprocessor parallel processing constraints, etc. Game data structures are then carefully rearranged to aid the engine in loading and processing them in the most efficient way. Complicated compression and caching schemes are explored to both reduce storage size (often linked to performance due to bus bandwidth) and to speed up the code.

Simultaneously we modularized the game engine to add more flexibility and features. Crash 2 has more effects, such as Z-buffer-like water effects, weather, reflections, particles, talking hologram heads, etc. Many annoying limitations of the Crash 1 drawing pipeline were removed, and most importantly, the overall speed was increased by more than two-fold. In order to further improve performance and allow more simultaneous creatures on screen, we re-coded the GOOL interpreter into assembly, and also modified the compiler to produce native MIPS assembly for even better performance.

Technical risk: If we can put more on screen, can we fit it in memory?

We firmly believe that all three Crash games make use of the CD in a more aggressive fashion than most Playstation games. So in order to fit the even larger Crash 2 levels into memory (often up to 12 megabytes a level) we had to increase the efficiency of the virtual memory scheme even more. To do so we rewrote the Al that lays out the CD, employing several new algorithms. Since different levels need different solutions we created a system by which the program could automatically try different approaches with different parameters, and then pick the best one.

Inaddition, since Crash 2 has about 8 times the animation of the first game, we needed to really reduce the size of the data without sacrificing the quality of the animation. After numerous rewrites the animation was stored as a special bitstream compressed in all 4 dimensions.

Design risk: Can we deliver a gameplay experience that is more than just "additional levels of Crash?"

We believe that game sequels are more than an opportunity to just go "back to the bank." For both of the Crash sequels we tried to give the player a new game, that while very much in the same style, was empirically a bigger, better game. So with the increased capacity of the new Crash 2 engine we attempted to build larger more interesting levels with a greater variety of gameplay, and a more even and carefully constructed level of difficulty progression. Crash 2 has about twice as many creatures as Crash 1, and their behaviors are significantly more sophisticated. For example, instead of just putting the original "turtle" back into the game, we added two new and improved turtles, which had all the attributes of the Crash 1 turtle, but also had some additional differences and features. In this manner we tried to build on the work from the first game.

Crash himself remains the best example. In the second game Crash retains all of the moves from the first, but gains a number of interesting additional moves: crawling, ducking, sliding, belly flopping, plus dozens of custom coded animated death sequences. Additionally, Crash has a number of new control specs: ice, surfboard, jet-pack, baby bear riding, underground digging, and hanging. These mechanics provide entirely new game machines to help increase variety and fun factor of the game. It would be very difficult to include all of these in a first generation game because so much time is spent refining the basic mechanic.

Technically, these additions and enhancements were aided by the new, more flexible information specification of the new tools pipeline and by additions to the GOOL programming language based on lessons learned from the first game.

Crash Bandicoot: Warped! - Every trick in the book!

Development: January 1998 - November 1998
Staff: 15 people: 3 programmers, 7 artists, 3 designers, 2 support

Premise: With only 9 months in which to finish by Christmas, we gave ourselves the challenge of making a third Crash game which would be even cooler and more fun than the previous one. We chose a new time travel theme and wanted to differentiate the graphic look and really increase the amount and variety of gameplay. This included power-ups, better bosses, lots of new control mechanics, an open look, and multiple playable characters.

Technical/Process risk: The tight deadline and a smaller programming staff required us to explore options for even greater efficiency.

The Crash Warped production schedule required that we complete a level every week. This was nearly twice the rate typical of Crash levels. In addition, many of the new levels for Warped required new engines or sub-engines designed to give them a more free-roaming 3D style. In order to facilitate this process we wrote an interactive listener which allowed GOOL based game objects to be dynamically examined, debugged, and tuned. We were then able to set the parameters and features of objects in real-time, greatly improving our ability to tune and debug levels. Various other visual debugging and diagnostic techniques were also introduced as well.

Knowledge from the previous game allowed us to further pipeline various processes. The Crash series is heavily localized for different territories. The European version supports five languages, text and speech, including lip sync. In addition, it was entirely re-timed, and the animation was resampled for 25hz. The Japanese version has Pocketstation support, a complete language translation, and a number of additional country specific features. We were able to build in the features needed to make this happen as we wrote the US version of the game. The GOOL language was expanded to allow near automatic conversion of character control timing for PAL.

Technical/Art risk: Could the trademark look of the Crash series be opened up to offer long distance views and to deliver levels with free-roaming style gameplay?

In order to further differentiate the third Crash game, we modified the engine to support long distance views and Level of Detail (LOD) features. Crash Warped has a much more open look than the previous games, with views up to ten times as far. The background polygon resource manager needed some serious reworking in order to handle this kind of increased polygon load, as did the Al memory manager. We developed the new LOD system to help manage these distance views. These kinds of system complexities would not have been feasible in a first generation game, since when we started Crash 1, the concept of LOD in games was almost completely undeveloped and just getting a general engine working was enough of a technical hurdle.

Similarly, the stability of the main engine allowed us to concentrate more programmer time on creating and polishing the new sub-engines: jet-ski, motorcycle, and biplane.

Gameplay risk: Could we make the gameplay in the new Crash significantly different from the previous ones and yet maintain the good elements of the first two games?

The new, free-roaming style levels presented a great gameplay challenge. We felt it necessary to maintain the fast-paced, forward driven Crash style of gameplay even in this new context. The jet-ski in particular represented a new a kind of level that was not present in the first two games. It is part race game, part vehicle game, and part regular Crash level. By combining familiar elements like the boxes and creatures with the new mechanics, we could add to the gameplay variety without sacrificing the consistency of the game.

In addition to jet-ski, biplane, and motorcycle levels, we also added a number of other new mechanics (swimming, bazooka, baby T-rex, etc.) and brought back most of Crash 2's extensive control set. We tried to give each level one or more special hooks by adding gameplay and effect features. Warped has nearly twice as many different creatures and gameplay modes as Crash 2. The third game clocked in at 122,000 lines of GOOL object control code as compared to 68,000 for the second game and 49,000 for the first! The stability of the basic system and the proven technical structure allowed the programmers to concentrate on gameplay features, packing more fun into the game. This was only possible because on a fixed hardware like the Playstation, we were fairly confident that the Warped engine was reasonably optimal for the Crash style of game. Had we been making the game for a moving target such as the PC, we would have been forced to spend significant time updating to match the new target and would have not been able to focus on gameplay.

Furthermore, we had time, even with such a tight schedule, to add more game longevity features. The Japanese version of Warped has Pocketstation support. We improved the quality of the boss characters significantly, improved the tuning of the game, added power-ups that can be taken back to previously played levels, and added a cool new time trial mode. Crash games have always had two modes of play for each level: completion (represented by crystals) and box completion (represented by gems). In Warped we added the time trial mode (represented by relics). This innovative new gameplay mode allows players to compete against themselves, each other, and preset goals in the area of timed level completion. Because of this each level has much more replay value and it takes more than twice as long to complete Warped with 100% as it does Crash 2.

Technical risk: More, more, more!

As usual, we felt the need to add lots more to the new game. Since most of Crash 2's animations were still appropriate, we concentrated on adding new ones. Warped has a unique animated death for nearly every way in which Crash can lose a life. It has several times again the animation of the second game. In addition, we added new effects like the arbitrary water surface and large scale water effects. Every character, including Crash, got a fancy new shadow that mirrors the animated shape of the character.

All these additions forced us to squeeze even harder to get the levels into memory. Additional code overlays, redundant code mergers, and the sacrifice of thirteen polka dotted goats to the level compression Al were necessary.

Conclusions

In conclusion, the consistency of the console hardware platform over its lifetime allows the developer an opportunity to successively improve his or her code, taking advantage of techniques and knowledge learned by themselves and others. With each additional game, the amount of basic infrastructure programming that must be done is reduced, and so more energy can be put into other pursuits, such as graphical and gameplay refinements.

Andy Gavin is Naughty Dog's co-owner and lead programmer. A native of Virginia, he has a B.A. in Neuroscience from Haverford College and an S.M. in Computer Science from MIT. As a member of the Artificial Intelligence Laboratory at MIT, he designed and implemented a vision system for one of NASA's Mars rovers. In addition to writing much of the C and LISP code in all three Crash Bandicoot Games, Andy created a new programming language called GOOL (Game Oriented Object LISP), designed to allow very rapid prototyping of the widely varying behaviors Crash's creatures and objects exhibit. He can be reached at [email protected].

Stephen White is a long time veteran of both the console and the computer video gaming industry, with over 13 years of published products over a wide variety of genres and gaming platforms. Stephen White began programming video games for the ATARI 800 and the TRS-80 home computers. In 1987, Stephen White co-founded the company ArtisTech Development, which went on to develop several video games for the Commodore Amiga and the ATARI ST, including the multi-award winning paint and animation package Deluxe Paint ST. Later, ArtisTech Development joined Digital Creations, Inc. to create Brilliance for the Amiga. Stephen White has worked for both Electronic Arts, Inc., where he worked on a title for the SEGA Genesis, and helped develop PGA Tour Golf 486 CD for the IBM PC, as well as Crystal Dynamics, Inc., where he developed Solar Eclipse, one of the earliest titles released for the SEGA Saturn. Now a senior programmer at Naughty Dog, Inc., Stephen White did extensive programming of both Crash Bandicoot 2 and Crash Bandicoot 3 for the SONY Playstation. Stephen White has a degree in Computer Electronics, and is certified in Communication Electronics. He can be reached at [email protected].