Author: by Tom Niwinski

[How to handle the population and testing of an open world game with thousands of items and hundreds of NPCs? Capcom Game Studio Vancouver technical director Tom Niwinski describes the toolset the team constructed for Dead Rising 2.]

Building a large open world game is hard. Add thousands of items, hundreds of NPCs, a few hundred quests, and it starts to become very difficult to keep a handle on where things are. Dead Rising 2 was no exception to this.

We needed a good toolset to allow our designers to effectively add new items to our world, find where they placed their items, and iterate on their work to provide a high quality experience for our fans. During the rest of this article we will discuss our toolsets and what worked well when populating our world^[i].

The Size of the Problem

The Dead Rising 2 world is roughly a square kilometer in size^[ii]. Over the course of the project about 20 people worked concurrently to create roughly 300 missions, 200 NPCs and about 18,000 items that fill the world.

All of these entities needed to be designed, created, iterated and tested, and these are just the things that shipped in the final game (we didn't count the thousands of things that were left on the cutting room floor).

We quickly realized that we needed a toolset that would scale smoothly as the game got bigger, find a good workflow for concurrent placement, and figure out how our content creators could iterate quickly.

The data management came in two flavors. The content creator data (the item placements, NPCs, etc) and the game generated data (pipeline logs, game telemetry, scheduling software, etc). Each had its own set of challenges.

Perhaps we didn't think this all the way through.

Tool Philosophy

The main philosophy is to enable our content creators to do their work without programmer intervention. That's to say, the tools should be friendly enough that any designer can add/modify/delete content on their own (one could call this a data-driven approach to game making). The more content creators can iterate on their own, the more interesting the creations they develop. More importantly their ownership and interest in their work grows drastically, and the features become significantly more polished. Here are the big successes we've had during Dead Rising 2:

Live Editing. Typically a tool will be some sort of viewer that allows a designer to tweak their content data. That data is then compiled, built, or baked, and somehow makes its way into the game. Often this process is long -- minutes if not hours -- involves restarting the game, and usually involves having a programmer enter some secret code.

It seemed much better to have a designer move objects around at run time from the comfort of their PC, so we developed a communications protocol that could talk between a game console and the PC. This was a bidirectional protocol that allowed commands from tools to activate RPCs in the game, and allowed the game to send information back to the tools. Tools could now make the game do things (like spawn objects, move their locations, change an items attributes, etc.) at runtime.

Tools could now be written in the language that was right for the task (C#, Python, Qt, etc.) Designers had nice GUIs with standard buttons, drop downs, menus, etc. While they were creating the content they could easily see their work in the game, and when they were done they only needed to check their work into our source control (no extra compiling, bundling, baking, etc). This meant there was no separate viewer for the programmers to support (so when some new rendering feature was added to the engine it didn't need to be ported into all of the tools, it was already in the viewer.)

The downside was that if the build broke, content creation was negatively affected. This made it that much more important to have solid submissions into our source control. It was also more difficult to do bulk operations, or operations that would require editing across levels. These negatives were easily offset by being able to see exactly what the user would see while editing, knowing that everything fit in memory and would work on the target platform.

The Pile of Data. There needed to be a common place to get/put/query generated data. Hopefully you are now thinking "a database." If your first thought was a directory on the network, after you finish this article, go and install your favorite database and stop writing custom scripts to do selects and joins on text files.

Dealing with text files is easy; any junior programmer can open a file, parse it then write it out again. However, this creates a random assortment of data files, probably in different formats, and to get any useful information out means some custom coding. Having everything in a database ensures that info is in one place, and getting answers to questions about your data is only a SQL query away. Not to mention one can now stop writing/debugging parsing code and worrying about concurrent user access.

Having said all that, for content created data, the tools worked on text files on local machines. Tools that connected to live databases would result in non-deterministic behavior of the game. In other words, while other people worked, they could alter your game. It would also be very difficult to have proper revision control for the data. A copy of the content creator data would be added to the database during the nightly build process so that we could treat this as generated data and run validation tests and other various queries without having to join text files.

^[i] I say populated, as the world geometry and textures were built with an off the shelf 3D packages, it would silly to create/support/educate a propriety 3D package when finished solutions are readily available.

^[ii] While it may seem small, walking a kilometer will take about 12 minutes; that’s why Chuck drives the motorbike.

The Toolkit. Hubris is a trait of Real Programmers who are very susceptible to the "Roll Your Own" syndrome. It is often easier to write your own code than to read, understand the pile of spaghetti that someone else (perhaps even you two years ago) wrote. This often leads to code duplication, and incomplete functionality. Real Programmers are also lazy^[iii].

To alleviate this syndrome, a template was created similar to the way the Visual Studio wizard creates a blank WinForms project. Now the tool had game connectivity built in and the toolkit library linked in; all that was left was to hook up the widgets and the tool was done.

t also made all of our tools look very similar (at least with the game connection side of things). The toolkit also contained an array of utility functions to deal with our database, console functionality (i.e. screenshots, deployment, etc), loading/saving user options, and a random assortment of helper functions.

Plug and Play. Armed with a database, a communications protocol, and a big toolbox, it was now time to make sure we could get data in and out of all of our systems. We instrumented our game engine so play session data could end up in our database.

A few dozen tables would store all the significant events in the game (game version, console playing, items picked up, zombies killed, doors opened, quests completed, Chuck's health changes, etc). This made it very easy for designers to see how our game was being played during development.

The logical extension was to add information that programmers were interested in. We tracked asserts that fired during play sessions, poor frame rate locations, and various memory metrics for different levels. In other words, to know how big some buffer needed to be, add a few lines of code, release it into the wild, and a few days later check what the high watermark was. The hardest part was figuring out the questions; the answers were a few lines of code and an SQL query away.

Next, our bug database was integrated into the fold. Our biggest breakthrough was to add a world location (x,y,z) to every bug. This changed our bug descriptions from "A little to the left of the big green thing, next to the stuff" to three numbers (it also didn't need to be translated).

The other great benefit now that the game talked to the tools -- which talked to the bug db -- was it was one click to teleport in game to the location of the bug. During development, duplicate bugs could be avoided by querying the bug db to see what was active at your location. Bug entry was further simplified by grabbing data (screenshots, inventory, location, etc) directly from the game without the person having to type that information in.

The Midden

We can now look more in-depth at how our world was populated. Our level editing tool was aptly named Midden^[iv]. It controlled all of the game object definitions, where they were placed in the world, our quests, the NPCs, and a few other odds and ends. We chose to control placement of items, triggers, and gameplay information in a tool rather than a 3D package so we could iterate live in-game.

Concurrent placement of items in the same zone would also be problematic (merging the data from 3D packages is not a trivial matter). At the end of Dead Rising 2, Midden controlled about 1000 data files that were edited by many people. There were around 18,000 items placed in the world picked from a palette of almost 2000 different objects.

The core of Midden was just a pretty way of viewing the massive collection of what we called "AFOs" (Asset Factory Objects). The AFOs are arranged in a rough tree form, so a hierarchy can be built up for a given level or quest. Each AFO has a name, a type (class) and one or more key/value pairs which make up the properties for the object. In object oriented fashion, all of the items can derive from their parent items as to not duplicate functionality. In this fashion we can define every entity in our world and build all of the levels, items, etc.

Because of this abstraction, Midden only needs to know how to manage the hierarchy, and how to process AFOs to nicely display its key/value pairs. This means adding more types of objects and extending the properties of an object is now a fairly painless process. The hierarchy management was a simple tree view control; we won't bother to talk about it there.

The interesting part was how we dealt with the AFO properties. All of the important information in an AFO was contained in the key/value pairs, the key was always a string, and the value would be one of a dozen types each with their own GUI control. If we look at our Food Item, the content creator would see this form:

The buttons at the top allow the content creator to select which attribute groups (sub class) they would like to edit^[v]. Through the custom controls (dropdowns, checkboxes, etc) it is very easy for the content creators to do their work without needing a programmer (when Midden connects to the game the editing is live). In addition having widgets rather than text files has greatly reduced syntax errors (it is hard to make a typo when picking from a list, or ticking a box).

From the programming end, adding a new object is a bit of work; it requires a new class (mostly boilerplate code), and some registration code. However, adding new properties is usually one line of code. Because Midden parses our source code and builds up a list of classes, their properties and which display widgets should be used, adding new key/value pairs is all on the game side (Midden just needs to be restarted, no tool side coding needs to be done).

^[iii] As, possibly, first said by Larry Wall in Programming Perl (1st edition), O’Reilly And Associates

^[iv] If you don’t know what a midden is it won’t be as funny.

^[v] Windows seems to be very slow at creating custom panels with hundreds of widgets, by grouping the classes in this way we could significantly improve panel creation/deletion time.

Here is an example of the code to expose the properties of our food item (note the comments serve as a kind of metadata to Midden to add extra flare to the GUI):

cPortableItem::ExposeVariables();
AddFloat( "HealthBoost", &mHealthBoost ); // min:0 max:1000
AddInt( "FoodType", &mFoodType ); // enum:eFoodType
AddInt( "AnimationChooserToUse", &mAnimationChooserToUse );
AddInt( "FoodEffect", &mFoodEffect ); // enum:eFoodEffectType AddFloat( "OuncesOfPureAlcohol", &mOuncesOfPureAlcohol );
AddBool( "IsSpoiled", &mIsSpoiled );

From the beginning we chose to store all of our data in plain text format, even thought it would have been more efficient to store things in a binary format. First, plain text is human readable, and in a pinch editable. Doing a "diff" is far easier between revisions (very useful to find out when things broke).

It is also far easier to grep (find in files), merge text files when conflicts arise and to write scripts to do bulk editing of objects (i.e. add 10% damage to all weapons). Lastly, if it showed up on the profiler or we needed the memory, adding a step in the pipeline to convert things to binary is far easier than reading binary in the first place.

Using Midden to create a palette of objects, it was now time to place them and populate the world. Having the generic AFO system allows placement information to be just another AFO with a location, rotation information.

The designer spawns items in Midden then fine tunes the objects location/rotation using the GUI while watching it take place in the game in real time. When they are happy with their placement save their work and submit it to the source control. No baking required.

While we had the standard property types (int, float, string, bool, enum, etc), the ability to define custom widgets in Midden really paid off when it came time to define item rotations. All angles in Dead Rising 2 were specified in quaternions; in other words, not for human consumption. A widget was built to allow the user to enter the rotation in a number of different ways:

Midden soon turned into its namesake, with so much data it was hard to find the thing you were looking for. Pretty quickly, content creators created a standard as to where in the hierarchy things should be placed. A number of tab views were created to separate item definition from item placement information.

A fully functional Ctrl+F was paramount; it needed to search not just the name of the AFO but also through all of the key/value pairs. Bulk placement was also a problem, adding banks of slot machines or attaching doors exactly to the frame was a very time consuming operation. Though, in the end, quantity was our biggest challenge. It is easy to gloss over the numbers, but imagine if it took only one minute to tweak an item placement. For Dead Rising 2, that is 18,000 minutes -- or almost 8 person-weeks. Yikes!

The Stash

With all the items in the world we needed a better way to visualize them. We often had questions like: How far apart are combo weapon regents from combo rooms? Where are all swords? Where are all the Zombrex posters? These questions gave birth to another tool called Chuck's Stash. The tool would read all of the data files then display the results in 3D. This is what the 18,000 objects look like in Dead Rising 2 look like from space (the rose colored polygons are individual stores in the game).

(Click for full size)

Next came a vast array of filtering options (type, name, time, location, regex, etc.) With this, a content creator could now easily find where and how many items existed, more importantly export the co-ordinates and then teleport to that location in game. This greatly helped in testing, Verifying the Improper Behaviour^[vi] achievement now took 30 minutes (alternatively, one could run around the world and try to find all of the Zombrex posters.)

We've talked about the plug and playability of our toolsets. It should be no surprise that the next step for Chuck's Stash was to read data from various sources. Data could now be visualized from play logs, our database, generated through other tools, or even live from the game.

This now opened a whole new set of questions that could be asked both in tuning the game and in fixing bugs during final. There were the standard questions: Where does Chuck die? Where do the zombies die? How do the zombies die? What stores did the focus group enter? These are a little dry to talk about, so here are a few of the more interesting questions we've asked:

Falling Through the World. While our automated Chuck traveled the world while we slept, the tool kept track of where it fell through the world (logging it all to the database). Visualizing this data in Chuck's Stash gave this picture to the art team (the concentrated red blotches are bad). Collision geometry was fixed up and we could run the test again.

The Frame Rate. With a large open world, populated with thousands of zombies and weapons, it is impossible to test every combination to ensure a quality user experience. Each time the frame rate dipped during a valid play session, it would be logged to the db. Then we could visualize it, go to the location in game, and figure out what the combination was that caused the frame rate drop. Rinse, Lather, Repeat!

Wrapping It All Up

Content creators are often at the mercy of programmers to get their content into the game. Enabling them to iterate and create content on their own is a huge step forward in increasing quality. In summary here are a few of the things I think we did very well when making Dead Rising 2:

^[vi] This achievement was to spray paint all of the Zombrex posters in the game.

^[vii] "If you always do what you've always done, you'll always get what you've always got" - NLP adage