Postmortem: Mythic Entertainment's Dark Age of Camelot

Dark Age of Camelot was the best-selling computer game in the United States for the week of October 7, 2001, and was still comfortably in the top five when I wrote this. This Postmortem is an overview of how this successful title was conceived and developed. My role on the project was as the game's producer.

Mythic Entertainment has been developing online games as a company since 1995 - forever in this field - but the company's founders had made online games even before then. In fact, as a company, we probably have more experience than any other company in developing online games of all types - over the years we have developed role-playing games, first-person shooters, top-down spaceship shooters, and strategy games. When I last wrote a Postmortem it was back in May 1998 for Aliens Online, our online first-person shooter based on the well-known Alien movies.

After Aliens Online, a nonaccelerated game, we created our first 3D-accelerated game, Spellbinder: The Nexus Conflict. During that project, we developed a relationship with NDL, makers of the NetImmerse 3D engine API toolkit. We learned a lot about 3D engine development over the course of that project and became very comfortable with software and art development in this environment. We finished Spellbinder, which went on to be a mildly successful Internet shooter, and it still has a small but loyal following.

After completing the Spellbinder project, we decided to create a graphical online role-playing game to compete with the then-new wave of online RPGs such as Ultima Online and Everquest, which were taking traditional text-based games and adding a graphical front end, with very successful results. Over the years, we had developed several nongraphical online role-playing games, including Dragon's Gate and Darkness Falls: The Crusade. Because of our experience developing RPGs, we knew that we had to have a slightly different slant on our new title in order to distinguish it from the RPGs that were already on the market. Darkness Falls: The Crusade (DFC) featured a built-in player-versus-player (PvP) conflict in which three different teams, called Realms, fought each other for control of magical artifacts, known as Idols. We really liked this concept, which served to keep DFC players hooked on the game - especially because no other online game featured such team-based conflict as a core part of the game design. So, in late 1999, we decided to make a graphical version of DFC. The project was dubbed "Darkness Falls 3D," and we began preliminary work researching client engine and server technology.

Right off the bat it was obvious that we had two major factors going in our favor. First, we determined we could use a much-enhanced version of the Spellbinder graphics engine as DFC3D's client, just as we were able to use DFC's server code as a platform for the new game's back end. Having such a solid client and server right at the start - with associated client/server messaging - alone saved us at least a year of development. Second, and even more advantageous, DFC's server came with that game's database of objects, monsters, and weapons. Indeed, we went into the Camelot project with a huge head start.

We were proceeding along under the DFC3D concept until our president, Mark Jacobs, came up with the idea of basing the game, at least partially, on the Arthurian legends. It was a great idea, since the stories of King Arthur are in the public domain, which meant we could use them with no fear of licensing issues. Of course, because the game was based on the idea that three Realms were in conflict, we quickly came up with the idea of basing the other two Realms on Norse Viking myths and Celtic Irish legends, respectively. Having the myths and legends of three cultures gives Camelot the feel of being three games in one, since each Realm has different races, classes, guilds, terrain, and monsters.

Because everyone knows what happened in Arthurian England, we based the game after Arthur's death and developed a back story of conflict among the three Realms. The game was rechristened Dark Age of Camelot, and around January 2000 we began the project in earnest. A year and a half and untold numbers of Monty Python jokes later, we finished the game.
The initial versions of Dark Age of Camelot used the rights for a tabletop role-playing game called Rolemaster as a basis for the class and spell systems. Not long into the project, the company that created Rolemaster, Iron Crown Enterprises, filed for bankruptcy, and we lost the rights. This turned out to be good for us, however, because we were no longer required to adhere to a set of rules based on the license - although we did have to scramble for about a week to rename and retune spells and classes and otherwise clear Rolemaster content out of the game.

As a company, Mythic had never before been able to devote all of its resources to any one game - we'd never had a project big enough to pay for it. Because of the sheer size and scope of Camelot, we wanted to ensure that everyone at Mythic devoted themselves fully to the project. Doing so required an influx of money, and that's where New York's Abandon Entertainment stepped in. Abandon owns a couple of small companies, each of which specializes in different types of entertainment: a film studio, a web company, and a couple of game content development companies. Abandon wanted to become more involved in game development, so it purchased a minority stake in Mythic. This money allowed us to devote everyone on staff to the Camelot project, while also expanding and hiring much-needed programmers and artists. Our spreadsheets showed that we had enough money to support exactly 18 months of development starting from January 2000, giving the project a hard end date of September 2001.

By the summer of 2000, we had nearly our entire team in place. We had about 25 developers working full-time on the project - quite a small number compared to other online RPGs, but our existing technology allowed us to reduce substantially the amount of technical programming staff required. We had five programmers, ten world developers, seven artists, and several other people working on the game.

Rob Denton, Mythic's vice president and chief technical brain, was responsible for all client and server programming, as well as the client/server messaging that tied the two together. His input was critical during design discussions, as he could tell us whether an idea would work or not. He immediately categorized features into "doable," "not doable," and the dreaded "on the list," which meant that it could be done, but he wouldn't commit to it. Brian Axelson was in charge of server programming as well as design of the game's combat system - a critical component in a PvP-centric game. Jim Montgomery provided Camelot's client interface coding and also designed and coded the game's magical spell system.

CJ Grebb and Lance Robertson led the art team. CJ was responsible for the game's look and feel, while Lance handled figure modeling and animations and managed the team's deadlines. Their team used 3DS Max and Character Studio to create Camelot's character and monster models and animations. The character models were technically advanced, as each in-game character has several different parts buried in it that can be turned off and on by the game. So, each model can have a helmet head and a regular head (with hair) without having to load in a new model. Mike Crossmire created the game's spells in 3D Studio, tweaking the NetImmerse system to display animated spells with spectacular results.

The other major group in Camelot's development was the world team, led by Colin Hicks. This group was responsible for quests, monster placement, object placement, and just about everything else having to do with creating the world of Dark Age of Camelot. Camelot's economy was designed by Dave Rickey. This economic system ensures that players must continue to spend money as they rise in level, which limits the amount of money that stays in the game. Dave and Mark Jacobs designed Camelot's trade skill system, which enables players to make armor, weapons, and other objects in the game - all tied to the economic system.
Among the myriad tasks that I did as a producer (writing, designing, persuading, arguing, and such), my job was to make sure all the teams worked together. I hosted an almost-daily morning meeting (at the wretched hour of 8:30 a.m.) where Colin, Rob, CJ, Lance, and I got together to make sure that we were all on the same page. I was also responsible for maintaining the master game client - all files added to the game had to be given to me, so I could verify they worked and then integrate them with the rest of the game.

For the game's sound and music, we contracted with Womb Music, based in Los Angeles, which had provided music for some of our previous titles. Rik Schaffer, the main guy at Womb, composed a wonderful soundtrack that consisted of several long main scores, as well as many shorter pieces in the style of Celtic, Norse, and old English folk songs, adding a sense of depth and quality to the world.

What Went Right

1. Community management/ beta program. From the beginning of the project, we knew we had precious few dollars available for marketing, and that our best chance to capture public attention would be to have a big presence on the various role-playing fan sites around the Internet. One, the Vault Network, provided us with some message board space, a news page, and a couple of moderators, and we were off and running.

We devoted a lot of time over the year and a half that Dark Age of Camelot was in development to interacting with the future fans of the game. We hired a community relations manager whose sole job was to read different message boards and report back to us what was happening in the community. From the beginning, we took our fans seriously and made many tweaks and additions to the game based on their commentary and ideas.

2. No bureaucracy. Since the founding of Mythic, we have striven to have little bureaucracy. We have no levels, no directors, and few managers. We have a president, a vice president, and a producer. That's it for management, although for Camelot we did have to assign a lead world developer and art co-leads, just to streamline the day-to-day processes of the project.
Because of this simple command chain, we experienced no power struggles. We feel this is the best way to make a solid, cohesive game - a small group controls what the game is and how it is presented to the user. Because of this approach, decisions are made quickly, and features can be implemented without an endless line of approvals and politics.

3. Smart business decisions. Our close relationship with Abandon Entertainment was a critical factor in the success of the game. Abandon's purchase of a minority interest in Mythic ensured that we had enough money to fund the game from start to completion. Abandon's management was smart enough to realize that we knew more about game development than they, so they largely left us to make game-related decisions ourselves. They were involved in the project, of course - some Abandon employees even became avid beta players of the game, even though most had never played an RPG before. Abandon's investment meant that we did not have to rely on any outside influence in designing or creating the game, which means that Camelot is wholly ours.

With Abandon teaming with us, Mark Jacobs, our president, decided to take a big chance and wait until the game was almost complete before looking for a distributor. In most cases, game companies seek out publishers, which typically have a hand in the design and production of the game and then distribute the game to the retail chain. With Mark's gamble, we produced the game ourselves (with critical financial help from Abandon and business advice from our business development person, Eugene Evans) and then looked only for a retail distributor. This gamble could have placed us at the end of the project with a great game but no way to get it into the hands of our customers. It all worked out in the end, of course, with Vivendi Universal stepping in and distributing - but on our terms.

4. Sweet serendipity. The Camelot project was helped immensely by factors completely out of our control - in other words, blind luck. Several high-profile online RPGs that were slated to launch at about the same time as Camelot were either pushed off (Shadowbane) or canceled outright (Dark Zion, Fallen Age). Also, the week we launched was originally scheduled to be the same week as the launch of Warcraft III, which will almost certainly be a huge seller. That project was also delayed, which ensured that Camelot launched as the only large-scale game, and the only online RPG, when it debuted on October 9, 2001. This little bit of good fortune gave the game a big initial boost, as there was little direct competition from other new products.

5. The joys of open source software and stability. Long ago, during the development of our early titles, we decided to use Linux wherever possible as our server back-end OS, and we kept to this same practice when creating Dark Age of Camelot. We have extensive Linux experience in-house, and it made sense for us to stay with a platform that we knew could handle the task and also was, well, free.

Because running Camelot would require a considerable amount of data management, we initially planned on using Oracle to store account and character information. However, Oracle's quoted license fee of more than $900,000 quickly removed them from contention. Once we got over our shock and amusement at Oracle's pricing, we turned to a Linux-based freeware solution, MySQL, to manage Camelot's data storage, which so far has worked admirably.

Everyone developing games should at least investigate open source solutions for their servers. It's saved us a pile of money and has been stable and reliable. In fact, prior to Camelot's launch, it was axiomatic that MMORPGs were unstable and prone to crashing during their first month or so. From the outset, we were determined to buck this trend. We co-located our servers directly at UUNET, on the network backbone, which ensured a wide network pipe to the Internet. With this Internet connection, we can increase our band-
width with just a few hours' notice to UUNET.

With the combination of reliable server code and a stable Internet connection- all running on open source software - Camelot went live on October 9, 2001, with virtually no problems. That first night, the game went down for about an hour and a half due to a database configuration problem, but since then, the game has been remarkably solid and stable. As of this writing, it hasn't been down due to server error for more than a few minutes ever since the first night.

What Went Wrong

1. Development of customer service tools. We really tried to avoid the customer service problems that are characteristic of some recently launched online games. One of the most important factors in keeping customer service reasonably effective was a smooth launch. Obviously, giving players fewer problems results in fewer calls to customer support. We did an excellent job with the launch - it went very smoothly. However, we could have better foreseen other parts of our customer service plans.

First, we had a lot more players in the first week after Camelot went live than we ever could have forecast - 51,000 boxes were sold in the first four days alone. Our forecast numbers called for a much smaller number, and we hired our customer service staff based on this smaller number. Also, we put off creating customer service tools until much too late in the development cycle - some had yet to be developed when the game went live. These missing tools really hurt the customer service staff and added to the time it took to help each player with in-game problems. Eventually, wait times became much too long, and customer support as a whole suffered because of it. As I write, we still are trying to work ourselves out of this hole.

2. Lack of a cohesive marketing plan. We went into the Camelot project with a lot of experience in developing software, but no real experience in creating a marketing plan. We got a lot of help with advertising from Abandon Entertainment, but there was no overall project plan. Basically, we took out ads in magazines that we thought were important and tried to keep on top of the Internet community. We didn't regularly issue press releases nor attempt to do a press tour or invite reporters to the Mythic offices to show off the game.

It's difficult to gauge just how much this hurt us. Our focus on Internet marketing gave us strong support among fans of the genre, but our lack of commercial marketing kept our company profile low, and we never received much mainstream media coverage because of it. Fortunately, we made up for our slow start, and then some, by our successful presence at E3. Abandon funded, designed, and staffed a large booth for us at the show, complete with medieval motif and lots of giveaways.

3. O Dungeons and Cities, where art thou? The first major update we made to Camelot's graphics engine to differentiate it from Spellbinder was to put in the rolling terrain system that makes the world so lifelike. We spent a long time making the outdoor areas of the game beautiful and well stocked with monster encounters. The ease with which we did this gave us a false sense of security when it came to developing our dungeon/city technology.
These areas in the game required a large number of models and characters in a much smaller space than the outdoor terrain, so creating dungeons and cities proved to be a much more difficult job than we thought. Because we put off doing the technical designs for the interior spaces for so long, in the end we simply didn't get enough of them done. The game launched with only three capital cities (one per Realm) and about 15 dungeons.

4. We have a great game but no servers! In a great "Why didn't they tell us about this in college?" situation, we went into the final months of the project with no credit rating. Mythic Entertainment has been around for a long time, but we simply hadn't ever borrowed any money, and so we didn't have a credit history. This turned out to be a problem when we went out to lease our servers from Dell and were flatly denied. We pointed out that we had plenty of money in the bank, but to no avail. Dell simply wouldn't lease us the computers until we had a credit history. In the end, we were forced to purchase the servers outright from Dell, which obviously had a much greater impact on our bottom line.

5. Postrelease fan communication. As good as our communication with Camelot's fan base was during the game's design and beta periods, it began to suffer soon after the game's release. The community simply grew too large to communicate with in the manner we had during beta, when we simply went out to Internet message boards and posted our thoughts and plans. With the game live, it was obvious we needed a much more coherent way to communicate with our fans, one that would not send them to numerous different fan sites to sift through literally thousands of messages.

This situation grew into a big problem when players became extremely frustrated by what they perceived as a lack of communication from us. About six weeks after release, we realized that we needed to create our own web site to publish information about the game: release notes, plan files, server status, Realm War status, and many other little things that we knew but our players didn't. This web site, dubbed "Camelot Herald," launched the following week and so far has been a great success. Fans of the game can now go to one web site to get all the information about the game in one place and with no interference.

For the Ages

It was a great pleasure to create Dark Age of Camelot, as it is the first big title that Mythic Entertainment has ever worked on. It was a wonderful thrill to see our names on top of the best-seller lists for those couple of weeks in October 2001, and we hope to be working on the game for a long time to come. As long as players are interested in playing the game, we'll be there adding content and updating it.