Extreme Game Development: Right on Time, Every Time

Extreme Game Development (XGD) is an agile game production method based on the popular software development method called Extreme Programming (XP). XP is not simply another fancy development method; it focuses instead on the one truly crucial issue: creating a working product.

Kent Beck and Ward Cunningham created XP while working on a project for Daimler Chrysler in 1996, although it was the result of several years of reflection on previous projects. Later on, Ron Jeffries and Martin Fowler joined them in their work. There are several examples of games developed using XP, or on which XP has been tested [FRISTOM02]. But since XP was created for programmers, it needed to be tailored to game designers, artists, and all the other non-programmers who play a role in game development. The result of this quest was Extreme Game Development.

At Titus Interactive Studio near Paris, France, we are currently starting two game projects using XGD. The first one will be a six-month project, involving programmers for the most part. This will be our "pilot project". The second project has already gone through pre-production, and is about to begin a twelve-month production phase with a full 15-person team. This will be our "application project". Of course, we can't put this project in jeopardy for the sake of testing a method, so our goal will be achieved if we can apply XGD throughout the project and still meet our deadlines.

Are Delays Really Inevitable?

For a long time, it seemed like game developers were genetically programmed to miss milestones. But publishers today impose stiff penalties when milestones are not delivered on time [CHARNE01]. Consequently, everyone is under an enormous amount of pressure, teams work around the clock, and sometimes deliver unstable milestones. Why is crunch time a necessity?

Recently, during a meeting with our programming team, we discussed a level design tool we needed to create. The level designers hadn't yet completely laid out their requirements, except to say that they needed the tool fast. The word "fast" triggered some strong reactions from the programmers. The programmers said that they would need at least three months to create this tool, that they needed detailed specifications, and that most of the time would be used to "shield" the code against manipulation errors. Finally, the programmers said that since they couldn't really say when this tool would be ready, the whole production schedule would be delayed.

If you're a producer or a project manager, I bet you've heard this kind of argument before. When discussing schedules, somebody always says something like: "Needless to say, we'll be late." And indeed, we are frequently late.

Why Games Are Late

Most of the reasons that games are late can be found in Gamasutra's postmortems collection [GAMASUTRA], and some of these reasons are recurrent:

Let's take a look at each of these reasons in a little more detail.

Technology evolves too rapidly. It's a fact that technology evolves at an ever-quickening pace. Currently there are three major console manufacturers on the market, and each console manufacturer delivers a new console roughly every five years. As a result, there is strong marketing pressure to take advantage of the latest technology. This is also true with the PC platform, which is constantly evolving. Development studios need to adapt: often they can develop no more than one or two games on a given console with a given technology. Our industry constantly rides the cutting edge, and developers are always discovering new technologies. New technology is good, but it takes time to learn and adapt new technology in games. Well, that's part of the fun, too!

Developers always want to do their best. Teams always want to do their best: create the best models, the most optimized code, the finest textures, and so on. This drive toward perfection is even a crucial motivating factor. Demanding that a team not perform up to its potential is one of the top ways to doom a project.

Ultimately, developers dream of delivering the game only "when it's done", when they are proud of each and every detail. But the drive towards perfectionism can result in blown milestones. The only teams that can afford relentless perfectionism to the detriment of their schedule are the ones that have plenty of internal financing.

Publishers frequently change their minds. Developers are not the only ones who are baffled by erupting new technologies; publishers suffer from this too. Producers get enchanted by the possibilities of the latest graphics card and ask the development team to take advantage of it. Publishers negotiate with console manufacturers for release exclusivity. Console manufacturers buy development studios so games will be developed solely for their hardware. Any of these factors can have a drastic impact on game engine development and lead to delays.

There is also another reason publishers change their minds. As games are mass-market products, the marketing department (rightfully) has strongly influences the publisher's policies. If the latest title from a competing publisher is a hit, the producer can told to integrate the competition's key features into his game.

This illustrates one of the major drawbacks in the game development process. In contrast to an industry like building construction whose projects are clearly seen and easily sized up, the product of game developers is less tangible. Nobody would ask an architect to add two stories to a building when it was almost completed. But in the game industry, it's not unheard of for a publisher to require a game developer to add two more levels to a game just a few weeks from release.

What XGD Solves

It would be great to have a development process that lets publishers change their mind during the development process, lets technology evolve, lets the developers be proud of their achievements, and still lets you complete your game on time. The question is: how do you go about it?

Above all, the ideal development process would let you react to emerging technology or marketing changes. It's nearly impossible to react to changes in marketing strategy or use a hot new technology if you are bound to a 500-page game design bible in the first few weeks of pre-production, or your entire milestone content is set in stone in advance. You need to keep options open to permit strategy shifts and let you adopt emerging technology. XGD emphasizes a close collaboration between publisher and developer, instead of relying contract-based negotiation. (Obviously your publisher would need to agree to an implementation of XGD, since the publisher is intimately involved.) Contracts are essential in the developer-publisher relationship, but both sides need to work together to create a quality game at the end of the development time.

On the other hand, the developer must agree to deliver a playable version of the game as soon as possible, which is upgraded at each milestone. That way the publisher can decide if there is enough content in the game to release it.

These requirements for a development process have already been expressed in several other methods. The Agile Manifesto (www.agilemanifesto.org), created by the Agile Alliance, is one such document. It states:

XGD is one such agile method. It is a way of adapting XP to the specificities of game development, especially:

Extreme Programming Values

XP is largely based on common sense. When you read the four values and thirteen principles of XP, or the authors' motivation for using XP, you'll catch yourself saying: "That's true", or "Been there, done that". That might be the case, but it doesn't hurt to go back and explore them in a new context.

XP emphasizes four main values: simplicity, communication, feedback, and courage. These values must be kept intact when porting XP to XGD. Let's look at each of them.

Simplicity. This is the most important value of XP. Always keep in mind the XP mantra: "Do the simplest possible thing that will work.", which has also been paraphrased as "You Ain't Gonna Need It" (YAGNI). Coders, for instance, are usually enthusiastic about programming "generic objects" -- classes or functions that can be re-used in almost any case. However, in almost every case, then, "you ain't gonna need it". What's the use in having a generic particle system simulating snow or rain or something when the game takes place underground? Likewise, a building that always stays far in the background doesn't need to be modeled using hundreds of polygons. And that fully modeled entrance door on the building? Well, YAGNI!

Communication. This is especially crucial in game development, where activities vary so widely. The game designer, programmer, artist, musician, voice talent…everybody needs to communicate, exchange ideas, validate concepts and finished pieces. Always give preference to oral communication over long, boring documents or memos, because it enhances reactivity and flexibility. I know of one ill-fated development project where all the ideas were cataloged as individual text files that were buried four levels deep in complicated directory architecture. It was impossible to extract any information from this source, and the game designer always contributed ideas this way, since he was told to. But because nobody knew his comments were there, nobody ever saw them, and quite a few of his great ideas never made it into code. Granted, some documents are necessary evils (game proposals, game concepts, game designs for instance), but they should be kept short enough so that everyone reads them. When you've got a design question or a really hot idea, don't write a note, talk about it!

Feedback. Feedback is strongly linked to communication. Most artificial systems need some kind of feedback. If you don't have a thermostat (an excellent example of a rudimentary feedback device) in your palatial, glass-walled penthouse office, you're going to fry when summer comes. In fact, all power devices use feedback mechanisms in some respect. Why shouldn't your development process contain its own feedback system?

Furthermore, if your thermostat only works when you're leaving the room, it won't be of any use. Likewise, if you analyze your project only after the game has been released, the postmortem won't help on that particular project. And if yours is a young development studio, you might not get a second chance to correct your errors on your next project - because there might not be a next project.

Instead, imagine conducting a Gamasutra-style postmortem every six weeks. With the whole team, you identify:

Your goal is to find a solution for those five things that are broken, or at least identify ways to begin to improve. From my experience at Titus, developers are only reluctant to admit weakness the first time. Once the first feedback session is over, the following ones are much easier. Ongoing sessions of introspection like this are great for keeping track of your project's progress. Take a look at Gamasutra's Postmortem writer's guidelines for some useful guideline tips. It might even be a good exercise in order to publish a real postmortem at the end of your project!

Courage. At first glance this might seem a strange value for a development method, but courage is involved in every step of the process. The publisher needs to make the developer feel confident by acknowledging that milestones are not carved in stone. Starting production without having planned every step might be uncomfortable for many, so it calls for boldness. Finally, it requires some courage on the part of the developer to accept regular feedback and self-criticism.

Translating XP principles to XGD

So, how are we at Titus applying XGD techniques? First, we took advantage of the fairly unique structure we have at Titus: the development studio (Titus Interactive Studio) is in the same building as the publisher (Titus) and the distributor (Avalon Interactive). This let us clear a hurdle that others could encounter when trying to initiate XGD: the client (the publisher, in this case) should be embedded in the development team.

(This notion of the "client" raises an interesting question. Some say that the "client" is really the game designer, since that person knows which features he/she wants in the game. But by definition, the client is the one signing the checks. And by that definition, the producer is really the client. At Titus, the producer will confer with the game designer once he/she has been integrated into the development team, then that person will prioritize tasks. The producer also is also on permanent call to quickly answer any questions that will arise, especially during the repetitive planning stages.)

Planning under XGD is highly iterative: milestones are scheduled regularly and frequently. For our projects, we decided that a milestone every six weeks is appropriate. Even if a global schedule for the complete game development has been defined during pre-production for the publisher and game concept approval, each six-week cycle starts with a "planning game" in which the team and the producer decide which features will be implemented for the current milestone (defined through "user stories", see sidebar). During this phase the producer needs to be flexible and occasionally accept that the initially specified milestone content may be slightly different than what's delivered.

Once the milestone has been defined, the six-week period is divided into three two-week development cycles where precise tasks are defined, scheduled, and achieved. Progress is measured by counting-unit test and functional tests. When the six-week milestone has been delivered, plan a small celebration (it's always good for morale) and then begin the feedback phase to find any glitches that might have arisen over those six weeks. You'll need to correct them in the next six-week cycle.

This six-week milestone iteration is not some kind of blind tunnel. A second practice in the Agile Manifesto ("working software") recommends creating a playable version of the game as quickly as possible, and updated it as frequently as possible. This means that integration, which is usually postponed until the end of the project, is now performed continuously. Integration is a classic bottleneck in game development, and difficult choices must frequently be made when all the different bits and pieces are put together. Some created assets -- representing hours of dedicated work for an artist, programmer or musician -- are wasted. Conventional wisdom says that up to a third of all assets created will never make it into the final game!

Continuous integration includes an automated nightly build [IRVING00], but also automated assets exportation. Get in the habit of modifying assets in some kind of original pool and then exporting them to the format used in the game engine. This way, when you are in your final rush to release the gold master and something goes wrong (when does something ever not go wrong?), it won't be that long, manual, 3D-model exporting procedure. Continuous integration is definitely the answer to one of the project manager's worst nightmares.

A Whole Team

Another problem that keeps project managers awake at night is team turnover. Imagine that your "special effects particles" programmer quits two-thirds of the way through development. This risk is ever present, and when it happens, you're stuck with a gaping hole in your team. How can you avoid this over-specialization? With XGD, all the programmers are responsible for all the code, so there's no longer someone with the title of "special effects particle programmer". This is also true for artists, so the project manager should be careful about assigning titles like "urban environment 3D modeler" or "character clothes texture artist". This does not mean that individual competence should be ignored, but that this particular knowledge should be spread throughout the team -- a team now considered as a whole.

One way to spread this knowledge around is through pair work: two developers work on the same workstation, performing each task. For fairly obvious reasons, this is the most controversial practice in both XP and XGD. In pair work, one person works the mouse and keyboard, and the second person acts in a kind of supervisory role. The one with the keyboard must describe their current actions out loud, and the second one tries to find simpler solutions, catch typing or modeling errors, and so on. Regularly, they swap roles. Artists seem to be the most reluctant to try this work arrangement, as artistic creation is based a kind of intensely personal instinct (whereas coding is more structured and logical). Pair work is a challenging practice in game development: should 100% of the work be done with two people, or should some of the work be performed alone? Here at Titus, we will only be able to answer this thorny question after our milestone postmortems, and we'll share our findings with you.

Testing

Using XGD, testing is not simply a good habit, but second nature. No feature, code function, or asset can be said to exist until it's been tested. Furthermore, all tests should be automated (this is compulsory with XP), and all these tests can be chained and run in batch-mode at any time.

There are two kinds of tests: functional tests and unit tests. The producer, along with the game designer and the lead tester, defines functional tests when defining user stories. For each of these scenarios, one or several scripts need to be written, and the requirement is met when these scripts run successfully. These tests are then added to the global test chain to ensure that this requirement is never broken.

On the other side of the production chain, the developers ensure unit testing, by performing one or more tests on each function or created asset. Programmers can use CppUnit, a dedicated unit-test toolbox. Artists can define automated scripts to test quantitative metrics. ("Are there no more that 4000 polygons in the highest LOD?", "Has the alpha channel been downgraded to 4 bits in my texture?", and so on.)

Automating tests doesn't imply the end of the QA department. Human testers can concentrate on tuning gameplay, focus testing or ergonomic testing, all of these being subjective testing and unsuitable for automation. The result of these tests will be given to the development team during the feedback stage and then translated into new user stories if the producer finds it necessary.

Practical Tools To Apply XGD

It's finally time to focus on how to apply XGD techniques. All of these tools I'm about to mention will be used on our own projects at Titus. Some are already in use, while others are being progressively integrated into our development process.

Project management. Project managers can track their XGD integration using the "XGD dashboard". This dashboard is a simple list of all practices and tools you, as a project manager, are currently using or would like to implement into your development process. Every six weeks, when performing your milestone postmortem during the feedback stage, you give a 0 to 5 score for each applied practice or tool you use. You also add a short comment about each one, and compute your general XGD score. The score's evolution throughout development shows you if your team is losing motivation due to XGD. If the score drops over several iterations, you have to ask yourself if XGD is really right for your team, or if you might not be better off trying another one.

You can also check on team motivation during the daily "stand-up meeting", so called because the entire team stands around the project manager. The physical discomfort caused by standing ensures that the meeting stays short (around 15 minutes, tops). You can use this opportunity to communicate company news, but the main goal is for each member of the team to explain what they are going to do that particular day, whether or not they need help, and who they will be paired with. This kind of meeting is really easy to implement, and will likely be adopted quickly by the whole team. As the project manager, keep in mind that your role is to coach the team, not assign tasks or dictate how pairs are created.

As stated earlier, one of the practices states that the team should be considered as a whole. Creating a game is a team effort, and not just the sum of individual work. All programmers are responsible for the code; all modelers are responsible for models and backgrounds, and so forth.

Asset management. Since one developer can take on the work of another at any time, everyone needs to agree upon some ground rules. This is the reason why the coding rules and the art pipeline need to be defined beforehand - including object- and file-name conventions, coding conventions and indention rules, and so on - in order to reduce the hassle of adapting to someone else's style.

Despite those rules, you might be confronted with situations where a developer unnecessarily modifies someone else's work, or you could even find that the latest code modification crashes all the unit tests. You can't possibly leave broken code in place, since milestones are so frequent and integration continuous. Therefore, an asset management tool is absolutely essential. Thankfully, several tools are available: Microsoft SourceSafe, Perforce, and NxN alienbrain are common choices. Some Open Source solutions also exist, like CVS or SvN, which can be installed either on a Linux or a Windows server.

Game design. Some XGD practices and tools involve the game designer. For game-related user stories, the producer should seek the help and advice of the game designer. What is really necessary for a test to be valid is that it must include at least one scripted functional test.

The game designer can also use a very specific tool for his work: UML - the Universal Modeling Language. XP actually recommends not using UML for extensive software design, but UML can be use to describe game design elements and interaction. [DEMACHY02]

Using UML for game design improves communication between game designers and programmers, and can be useful in dealing with artists and management, too. A UML diagram is a good starting point that everyone can agree upon. For your designs, you can use simple UML diagram editors, like SmartDraw, or rely on more powerful solutions like Open Source ArgoUML and Rational Rose. A UML design can also be refactored using almost mechanical rules: new game design classes can be introduced, others can be removed, and interactions between objects precisely defined.

For improving communication, feedback and simplicity, UML for game design is definitely an XGD tool. It can even be used to define functional testing of object interactions.

Quality assurance. All functional testing should be performed using automated scripts. The best way to create automated testing scripts is to use your game's own scripting language, adding specific test bypasses. Any scripting language can be used, but you will probably rely on Lua or Python, if you don't already.

What should be kept in mind is that all tests must stay in chain, and must be re-run frequently (at least once per day, during the nightly build). The total number of successful tests is the best progress indicator you can have for your project.

You may also have to construct specific scenes for these tests. These test scenes will eventually be integrated into the test process and used over and over, not erased as soon as the test runs successfully (which is commonly the case in game development).

The QA department can also add their own tests. Indeed, despite all the care taken during development, some bugs can get through and be detected duringa QA test. It might be a good idea to record player activity onto the memory card, and then use the scripting language to replay this sequence. Imagine if your game crashed at a particular time, and your programmer couldn't repeat this bug after trying dozens of times. Most certainly he would lose both his temper, possibly even his mind, and finally declare the bug as not reproducible. Recording a player's sequence can help sidestep this problem, and then that test can be added in the test chain to make sure that the bug does not recur.

Wish Us Luck!

XGD is a new method for managing game projects to ensure that games are delivered on time. XGD was not invented from scratch. Rather, it is an adaptation of Extreme Programming (XP) to the game development process. As I said earlier, XGD is currently being rolled out at Titus Interactive Studio on two projects. I'll let you know in a few months how we fared using XGD -- which practices worked and which did not.

Note that several "Extreme" practices were not covered in this article, so I recommend reading the XP books by Kent Beck to give you a better picture of these practices and help you understand the differences between XP and XGD.

Finally, I wish to thank Randall Roberts, who's great with a keyboard and words. (He's the guy responsible for "the glass-walled penthouse office" analogy -- probably thinks that everybody works in one of those.)

For further information

Web :
Extreme programming web site :
http://www.extremeprogramming.org

Extreme Game Development Web Site (mostly in French):
http://www.extremegamedev.org

Books:
Kent Beck Extreme Programming Explained, Addison-Wesley Pub Co, October 1999

Ron Jeffries, Ann Anderson, Chet Hendrickson, Extreme Programming Installed, Addison-Wesley Pub Co, October 2000

References

[CHARNE01] : Jim Charne, "Time is of the Essence"
"Famous Last Words" IGDA column, December 2001 - http://www.igda.org/columns/lastwords/lastwords_Dec01.php

[DEMACHY02] Thomas Demachy, "Programming a Game Design-Compliant Engine Using UML", in Game Programming Gems 3, Edited by Dante Treglia, Charles River Media, 2002 (End of the article)

[FRISTOM02] : James Fristrom, "Treyarch's Spider-Man", Game Developer Magazine, August 2002

[GAMASUTRA]
Gamasutra PostMortem Collection :
http://www.gamasutra.com/php-bin/article_display.php?category=5

[IRVING00] : Francis Irving, Automated build process, Gamasutra feature article, December 2000 - http://www.gamasutra.com/features/20001208/irving_01.htm