Author: by Jeff Russell

[In this in-depth postmortem, developer 8Monkey Labs explains the creation of PC and Xbox 360 time-traveling shooter Darkest Of Days, outlining exactly what went right and wrong in the creation of the ambitious title.]

Introduction

"Time travel is the future."

- Mark Doeden

Darkest of Days is a time traveling first person shooter developed by 8monkey Labs and published by Phantom EFX. It was released on PC and Xbox 360 on September 8, 2009 and will be available in Europe for the holidays. A free demo is available for both platforms.

Darkest of Days was the first title released by 8monkey Labs, based in Cedar Falls, Iowa. Its production took place along with the growing process of a brand new studio. 8monkey is and always has been small: at the peak of production, we had only 10 full time employees. You could probably call us an "independent developer" but we don't usually think of ourselves that way.

Things took a lot of unpredictable turns, for better and worse, during the course of development. One example: in June of 2008, 8monkey (along with publishing partner Phantom EFX) lost its office space to flooding.

In a few days' time, 8monkey and Phantom had moved all their equipment and personnel twice (the first backup locale was also cut off by flooding two days later as waters continued to rise). You might think losing two office buildings inside a week would slow us down. You'd be wrong.

What follows doesn't even begin to tell the tale of the game's production, its ups and downs of morale, and of course the contributions of all the individuals involved. But this overview may give you a sense of just where this game came from, and how it came about. Maybe we will both learn something. So let's begin.

What Went Right

"You have to make it sweet first. Then you can spend time to make it awesome."

- Andres Reinot, Engineer

1. An Inimitable Crew

On day one of 8monkey's existence, development of the Marmoset engine commenced, and the first year of business was spent developing that technology and creating prototypes for a (then unnamed) time travel adventure shooter. During this time a core team of four people were the only members of the development staff. No members of this team had ever worked on a game before. We were all new.

It's a testament to the quality of the staff and the determination of leadership that Darkest of Days was created at all. Given a green team and a small budget, 8monkey excelled at making the most of available resources. Most members of the team wore many hats (particularly in level design), and folks generally went above and beyond the call of what a mere job would command.

With a small team, every member felt ownership in a large portion of the game, and this motivated our staff to do some of their best work. Even as the team continued to grow, this sense of strong ownership persisted.

Darkest of Days has the unique feel it does today because of the team that created it. Sure, we made some newbie mistakes, but there was a lot of good here. Things like the sniper mission, the stolen Zeppelin level, and the grimly satisfying microwave gun used in the endgame all grew from the ground up during our production process. Most players and reviewers have described the game as unique, and we love to hear it.

2. The Arty Types

Outside contractors were used for a large portion of the game's art assets, most notably its characters. This process was ideally suited to a small company like 8monkey. There are substantial time and cost overheads involved in acquiring full-time on-site employees, and in our case it can be tricky to get people to move to Iowa. So for a lot of our art, we turned to contractors.

Our lead artist (who was on-site) is a polycount moderator, and arranged a lot of solid connections in that community and elsewhere. Freelancers are often regarded as higher risk proposals, widely variable in their work ethic and quality.

Having a good cornerstone of connections in the art community, 8monkey was able to go right to not only some of the best people available, but the ones who would best fit our project. We have all been surprised by the quality and consistency of the work that resulted from these "random guys on the internet". Combined with strong on-site art talent, the art pipeline for Darkest of Days ran well.

We also had good luck with our in-engine art tool, dubbed Toolbag, which was used in-house and by many of our contractors. Initially it was used as an engine preview, then later developed into a full material editor, integrated with xNormal, and finally given animation preview and particle editing tools.

Our tools philosophy at 8monkey has always been that it is best to work in an environment that as closely resembles the end result as possible -- this means using the engine as an art tool, not just for the game. Toolbag and our level editor Habitat both embody this ideal. There's really no way we could have pulled of a project of this size without these excellent tools at our disposal. Toward the end of development we released Toolbag for free to the art community, to very positive reception.

3. Custom Engine (just the good parts)

"...like someone trying to build a robot out of chickens and radios."

- Jake Splichal, Level Designer

The Marmoset Engine was written for use with Darkest of Days. Its development began prior to full production of Darkest of Days -- sort of. Early prototypes of the game were pretty rocky, technically speaking. Nevertheless, over the course of the project, this engine has morphed into our studio's greatest asset.

It's become a useful technology base and tool set that everybody here now knows inside and out. Marmoset was mostly conceived as an "exactly and only" solution to the requirements of Darkest of Days, which means that while it is powerful, it is also simple to use and maintain (the entire engine is less than 200,000 lines of code). It is well-suited to a growing studio.

The custom code has given a very custom feel to the game as well. Marmoset has unique animation systems, lighting algorithms and shaders, and special effects. The result is a game that looks subtly different from everything else in a lot of small ways.

The engine also allowed for free experimentation and fast prototyping. Turnarounds with the codebase are fast (a full rebuild takes less than five minutes), and integration of new features has proven easy. Combined with an in-engine editor for level design with an instant preview mode, iteration times were kept tight across the board.

Toward the end of the project cycle, 8monkey Labs teamed up with Nvidia to add support for GPU accelerated physics through the PhysX API. The PC version of the game made use of this technology with its particle system, adding new debris, smoke, and impact effects.

In the end, Marmoset let us leverage a lot of the power and features found in many big name engines, at a fraction of the cost and while still allowing us to retain control over our technology. On top of all of this, we now have a solid technology base (owned by 8monkey) upon which to build future projects.

4. Orangutan's Afterbirth?

Darkest of Days has a winning concept: you are a 19th century American solider who is pulled out of time just before your untimely death at the battle of Little Bighorn, brought to the future, and trained to go back to various time periods to correct history by shooting a lot of guys, often with advanced weaponry brought from the future. If this sounds like fun, it's because it is.

Darkest of Days had such an interesting premise that we knew we'd get an audience from that alone. Mixed with the variety of time periods, we had a potentially great game on our hands. Every member of the team saw the potential here. Combined with the support of our marketing team, Darkest of Days was able to get a lot of publication placement, preview and review coverage, and a nice spot at PAX, despite our limited budget and "no-name" status.

One wild-card that nobody saw coming was Agent Dexter. Dexter was originally conceived as the player's sidekick and guide through the game, delivering mission objectives and narrative. His existence was outlined in early design documents, and at the time no one thought much of it.

Then his dialogue lines started to come in. Dexter turned out to be not only generally belligerent and derisive of the player, but also full of all kinds of obscene, home-spun witticisms. Combined with the perfect voice talent, Dexter was an instant hit with the team and with players. The character of his speech can really be best illustrated with an example:

"This is going to be an old fashioned cluster-fuck right here. You're headed straight into some of the deadliest hours of American history and it's gonna be uglier than an orangutan's afterbirth."

- Agent Dexter, Darkest of Days

5. Busy Bustling Battles

Central to the game design from the very beginning was the need for portraying a large, living battlefield in a first person shooter. We wanted players to feel immersed in the true scale of warfare, to see the full extent of the destruction and loss of life, and to get the feeling of being overwhelmed. A lot of shooters have mastered the art of a five man encounter, but we wanted to go much bigger than that.

This required a lot from a team making its first game -- we not only had to be able to simulate and display hundreds of NPCs onscreen at once, but we had to negotiate the gameplay ramifications as well. Not only did all the AI have to perform well, but the behaviors had to be written such that they worked well in large groups, small groups, or individually.

The rendering and animation systems had to be ready to take the load of hundreds of characters. Level designers had to walk a fine line between scripted Disneyland-style layouts, and the total chaos of a huge battle. And there were big design questions to answer too -- for example, how do we empower the player when he's one of a hundred characters on the field?

There were some nervous periods in the middle of development, before the performance was brought under control and before some of the game-play questions were answered, where we weren't sure it could be pulled off. But the sense of scale and carnage was just too cool not to have, so we stuck it out.

In the end, we were all pretty surprised by the results. Engaging 20 to 100 enemies at a time is really quite a different play experience from most action games, and we're proud of the resulting chaotic feel of our battlefield scenes. The grand battlefields are now definitely the game's strong point.

What Went Wrong

"I think we opened Pandora's box! And inside's a dozen hornets' nests tucked under the nutsack of a rabid rhinoceros! This ain't good, my friend."

- Agent Dexter, Darkest of Days

1. Xbox 360 Port

Eventually the decision was made to bring Darkest of Days to the Xbox 360, instead of just the PC. Resources at 8monkey were tight already, so a third party contractor was brought in to work with 8monkey's engineers to port the Marmoset engine and the game code to the Xbox 360.

The contractor's employees were capable and hardworking, communication between the two companies was smooth, and the port was ultimately successful -- we shipped on the 360 and even passed Microsoft certification on the first try. So what went wrong?

First, the decision to port the game to the Xbox was made about a year into full production. By this point the engine had seen nearly two years of PC-centric development: the technical design had not called for console support at the outset. So while the decision to make a console version of the game made sound business sense, it necessitated a major overhaul of the technology base.

Fortunately, we were able to keep most of our middleware across platforms -- SpeedTree, PhysX, and OpenAL all had Xbox 360 implementations that did the job. The major efforts in the port focused around the graphics code, resource loading and management, and memory use.

The rendering code had to be almost completely rewritten, creating a second Direct3D renderer in addition to the OpenGL-based PC version. Load times had to be brought under control with a new threaded archive-based loading system (with the nice side effect that PC load times are now amazingly fast). And the entire codebase had to be gone over in detail to look for memory savings and deal with fragmentation.

Obviously, timelines suffered as a result. The amount of work involved in the port was initially underestimated, causing schedule slips. Porting efforts were compounded further by schedule troubles on the PC side, which were amplified into even bigger delays in the console version. 8monkey's engineering team ended up spending more time working on console issues than anticipated, tying up programming resources that could have been put to other use.

In the end it squeaked through. The console version retained most of the visual effects and content from the PC, but had to be scaled back in a few areas to salvage performance and memory. The final console deadlines just before shipping were some real nail-biters, and there were more than a few all-nighters and working weekends before it passed.

The root mistake here of course, in case you hadn't picked up on it already, was planning. Had we known from the outset that Darkest of Days would see console release (or even planned for the contingency), smarter tradeoffs could have been made earlier in the process regarding technology and art choices. Unfortunately, the late port put us in a bit of a corner in some ways, and some hard choices had to be made that could have been avoided had we been more prescient.

2. Custom Engine (just the bad parts)

While the development of the Marmoset engine has been a good choice for 8monkey, it may not have been the most expedient choice for Darkest of Days.

Developing an engine from scratch takes time. A lot of time. Even though in our case it was done at very low monetary cost (it was created by only two engineers), it did take substantial time. Nearly a full year had passed by before the engine was full-featured and usable. During this first year, the rest of the team was attempting to prototype game-play for Darkest of Days. But most of our efforts during this phase were constrained by or centered around engine development in some fashion.

Even during full production, when the team began to grow and content for the game began to be produced in earnest, the engine was not always ready to behave. It never quite seemed to be finished (and indeed, what game engine is ever finished?) During the course of the project, engineering efforts were constantly diverted away from game code and into engine maintenance. In the last year or so, Marmoset did finally began to stabilize somewhat (with the notable exception of the Xbox 360 porting efforts). But the time had already been lost.

So should 8monkey have used a third party game engine? At this point it seems fairly clear that doing so would have shaved significant time from the development schedule, and gotten us to market much more quickly.

But would the game have been better? That's difficult to say. The look and feel of it would certainly have been drastically different, and it would have no doubt lost some of its unique charm. Some technical problems would likely have been traded for others, and certain technical design requirements may have been more difficult to satisfy (large outdoor environments and big NPC counts being two good examples).

3. The Travel Agent From Hell

"Darkest of Days was a game everyone wanted to spend fifty million dollars on... but we had ONE!"

- Aaron Schurman, Writer / Producer

Darkest of Days takes place in five distinct time periods: the American Civil War, Custer's Last Stand, World War I, World War II, and the eruption of Mt Vesuvius in 79 AD. This variety is one of the strong points of the game, allowing players the choice of time lines when selecting missions. But the feature did not come easily.

Asset reuse suffered massively: each time period required unique sounds, levels, characters, buildings and environment art. The number of time periods meant that at most, a period-specific art asset was used for maybe 15 to 20 percent of the "game time" in the final product. Our art team had to churn through props, characters and animations at a hectic pace. Toward the end we all became good at planning clever production of assets that could be shared between two or more periods. We tried to resist the temptation to just go nuts with wooden crates everywhere.

The temporal variety placed a lot of pressure on the engineering and design teams as well. AI features ended up being created for specific periods, and not used in the rest of the game (Civil War formation combat is a prime example; Roman soldiers are another).

The differing nature of the battles made game design a challenge. The game has 24 weapons, most of them era-specific, and they all had to be balanced and tweaked. Some of the future guns are used in several different periods, and these had to be adjusted for use in a wide variety of scenarios.

Making Darkest of Days was in some ways like making five games with the resources for one. It put a large strain on an already small team; saying it was a stretch is putting it mildly. In many cases it prevented us from polishing certain areas of the game to the level we would have liked. The variety still works well for the title and is a major part of its appeal, but it was traded for a measure of quality.

4. Artificial whuh?

"Why can't computers be as smart as robots?"

- Mark Doeden, Director

The battlefield AI slowly morphed into one of the biggest design challenges in the game. The required complexity was initially underestimated, and it became more and more apparent as the project went on that a major change to our AI approach was needed.

8monkey underwent three full rewrites of the AI for Darkest of Days, each lead by a different engineer. The third and final revision built on lessons learned by the previous two and was definitely the strongest, but still had some shortcomings.

One of the main design requirements was performance -- previous versions had not performed well with more than about ten or 20 characters, and the game design called for hundreds. So the first priority was to get the thing up to speed. This included the removal of some AI middleware which was not performing well, and as a result many of the low level algorithms, such as vision and pathfinding, had to be rewritten.

The new implementations had the advantage that they were more efficient and tunable, but even so, the balance between intelligent characters and numerous characters proved difficult to strike. The results on large battlefields were good -- every character thought for themselves and the battle really took shape well.

But once the player stopped and payed close attention to a smaller number of NPCs, the illusion tended to break down. Tuning the performance to be ideal for large groups sometimes resulted in individual NPCs who seemed unobservant or reckless. This constant balance war between AI quantity and quality continued through the remainder of the project.

Complicating the matter was the variety of behavior required from characters in the game. There were characters from five different time periods, civilian and military, human and animal, and on top of all that three types of future agents. Again, a lot of effort was spent on variety here, trading that working time for some quality.

Ultimately the AI system accomplished some of its major goals -- namely performance and the behavior in large scale battles. But it never quite reached the level of intelligence we all wanted from it in small encounters. Had we been able to find a solution to this problem as well, the game would have had a more consistent quality.

5. Divergent Design

"We can't all just go around stickin' our fingers in each other's... pies."

- Jack Monahan, Level Designer

While the concept and plot of the game were well laid out, the specifics of the game design were not always as clear. As a result, team members were sometimes unsure how to proceed with balancing and game tweaks. Often individuals would improvise a solution on their own, without conferring with the group, only to be met with disapproval from design leads or other team members once their work was added to the game. This bred frustration, duplicated work, and wasted effort.

This is also the flipside of the ownership coin. Many team members felt so strongly about their individual contributions that it began to create friction every time a change needed to be made. We all had to be working toward one goal, and keeping everyone's efforts on the same track proved difficult from time to time.

The road to completion was long, and there were some periods where the game did not look or play well. It can be frustrating to keep working on something that doesn't look good yet, and to resist the urge to do a 180 and totally change everything. Ultimately we were able to keep the working focus on nailing the basics of the game, but there was a lot of time and mind-share spent on ideas and proposals to "fix" it, when really what was needed was polish on the core mechanics.

Another source of design uncertainty was a general lack of outside gameplay testing. Most external testing was an effort to find bugs rather than test a design, and was therefore not centered around the central "is this fun" question that every game developer needs answered. Level design, AI, and weapon balance all could have used a lot more design testing than they received. There was a good deal of internal testing by team members, but this often did little to settle design disputes or inform the team.

Toward the end of the project, design problems smoothed out somewhat. Design iterations happened more regularly and with more discussion, and more outside play testing was sought. Everyone "fell into the groove", and as is often the case with many game projects, our team did some of its best work toward the end.

Conclusion

"My watch reads beer-thirty... how 'bout we hop on the next portal ride out of here and get a cold one?"

- Agent Dexter, Darkest of Days

Did the project have its problems? You bet. We had a small team, a small budget (this ain't no 20 million dollar game), and we made some beginner mistakes. We were starting up a new company, and bringing a new IP to life along with new technology. We were told on several occasions that the task sounded impossible. Expecting anything other than a rocky road would have been ridiculous.

But for all this, the game is unique and certainly breaks the mold in lot of ways. 8monkey set out to make something that stood apart from cookie-cutter shooters, something new and different. Here we have succeeded.

Public reactions have been mixed -- some love the unique feel to Darkest of Days, and others don't see past its faults. It is different from a lot of shooters, in both good and bad ways. The game now has a small cult following of players, loyal folks who love this difference from the mainstream.

It might have been a fun experiment to leave Darkest of Days in the oven for another year. This was not possible of course due to budget constraints, but the results would have been interesting. As the project progressed, everyone's work just seemed to get better and better, and toward the end we were really hitting our stride. As 8monkey plans for the future, it's going to be fun to see where we go from here with our new technology and experience.

Darkest of Days was ultimately a blast to make. It's really too bad you weren't there. There were free t-shirts and everything.

Data Box

Full Time Developers: 4-10

Part Time Contractors: 14

Budget: $1.7M

Development Time: 1 year prototype and engine development, 2.5 years production

Release Date: September 8, 2009

Platforms: Windows PC, Xbox 360

Typical Machine: 2.6Ghz Intel Core 2 Duo, 2GB RAM, GeForce 8800

Software Used: Visual Studio, Modo, Mudbox, Maya, xNormal

Notable Technologies: Marmoset Engine, OpenGL, PhysX, SpeedTree, OpenAL

Size: ~200,000 lines of code, 80GB of source content