Book Excerpt: A behind-the-scenes look at building Ultima Online

What follows is several excerpts from Braving Britannia: Tales of Life, Love, and Adventure in Ultima Online (http://www.bravingbritannia.com). In addition to speaking with dozens of players about their experiences in the grandfather of MMOs, the book features interviews with several of the game’s early programmers, designers, and producers, including Chris Mayer (Fallout 76), Raph Koster (Star Wars Galaxies), Starr Long (Shroud of the Avatar), Gordon Walton (Crowfall), Rich Vogel (DOOM), and others.

Before Chris “Faceless” Mayer went to work at Origin Systems as lead programmer, he experimented with Ultima Online as an early adopter. Outside of the game, Mayer worked as a software engineer in Austin, Texas, utilizing his employer’s faster-then-average internet connection to play Ultima Online during lunches or after hours.

Mayer had always had an interest in creating games and hoped one day to do so in a professional capacity. In fact, he’d been preparing for the opportunity throughout his youth.

“I’ve always wanted to make games since I was a kid,” Mayer said. “It’s what got me into programming. One of my favorite games was Adventure Construction Set, where I even spent a week while on vacation with my family designing the game on graph paper and notebook pages before coming home to my computer and coding it all in.

“I taught myself Applesoft Basic in fifth grade and took a class in 6502 assembly in seventh grade. In college, I learned various flavors of assembly, C, C++, and a few other languages I have never used since.”

Until Mayer could figure out how to get his foot in the door with a game studio, he was content to simply play the games being released. Throwing himself into UO whenever he had a free moment, Mayer said he and played “religiously,” and became an active member of the community, both inside the game and through the Vault Network message boards, a popular website of the era dedicated to roleplaying games.

In the summer of 1998, Mayer’s close friend Scot Salmon received an email from Origin Systems. Having seen many of Salmon’s helpful posts on the Vault Network’s forums—and knowing of his background with computers—a representative of the company invited him to apply for an open programmer role.

While Salmon wasn’t interested in becoming a programmer, he knew someone who was, and pointed the recruiter in Mayer’s direction. Following an interview with three programmers at the Origin Systems offices, Mayer was offered the job by producer Rich Vogel.

Mayer said that shortly after Ultima Online’s launch, the game’s original programmers—who had been contracted to build the UO codebase—were assigned to other games.

“When I walked in the door at Origin, Rich told me that only one of the original programmers remained on the project and that he was primarily a client guy,” Mayer said. “Kirk ‘Runesabre’ Black was already working, and it turned out Kirk and I were the only ones who could work on the backend.”

And just like that, Mayer was in over his head. Not only did he not understand the game’s codebase, but with thousands of players inhabiting UO’s servers, everything seemed poised to explode on a regular basis.

“For the first few months of my employment, I came into work every day thinking the entire service would combust into flames,” Mayer laughed. “We somehow kept it together while slowly making improvements.

“That was, to this day, one of the most exciting jobs I’ve ever had. MMOs were so new, and no one really knew what they were doing at EA when it came to online games, so Kirk and I pretty much got to do whatever we wanted.”

While players across UO’s virtual world of Britannia were swinging swords and casting spells without a care in the world, Mayer and Black were putting out fires and solving problems as fast as they’d appear—the equivalent to riding unicycles across tightropes over a forest of razor blades.

“It was hard,” Mayer said. “I was also unfamiliar with game architecture in general. Couple that with the fact that the UO team was under stupid amounts of pressure to ship the game when they did, and they had to take many, many shortcuts.

“That made the codebase very fragile and a frightening tangle of dependencies between systems. We had to deal with service outages practically every night for a long time until we could stabilize the codebase.”

The first priority for Mayer and Black was stabilization as a whole, fixed through a branching solution that allowed the programmers to make hotfixes to certain sections of the game without risking unintended changes across the entire system.

“A branch is a line of source code that is independent of any other branches of development,” Mayer explained. “We might put a branch out on Live while a different branch is being tested by QA, while perhaps a third branch is being worked on by engineers. All these changes are independent, so if we have to fix something [on the live version of the game], we only need to touch the Live branch and test the fixes there.”

Once the servers were stable, Mayer was promoted to lead programmer. Now thoroughly “in the fire,” he set to work on Origin’s next project, the game’s first official expansion called The Second Age (“T2A”). With the project came new challenges, including a new landmass, two new cities, and a ton of new monsters for players to battle. And with all those changes, of course, came more stability issues.

“I remember getting my first client bug trying to ship T2A,” Mayer said. “I was very comfortable working in the backend because it resembled the kinds of systems I had to work on in school and at National Instruments.”

But while Mayer knew programming, UO was his first gig involving 2D and 3D graphics. New problems reared their heads including rugs falling under floors and even player houses disappearing altogether.

“I had no clue what I was doing, but managed to get it working by walking through the code methodically,” he said. “Kirk and I worked tirelessly to keep the servers running, including having to miss the launch party for that expansion.”

Soon, Mayer was tasked with tracking down exploiters—players who abused faults in the game’s code for their own gain. Exploits were often used to make duplicates of powerful objects, or raise a player’s skill in a way that went against the game’s design principles.

Mayer recalled a time that one of the game’s servers crashed repeatedly with little explanation. Putting on his detective hat and looking at the code, Mayer saw no logical reason for the recurring problem.

Night after night, Mayer watched as one of the servers suddenly blinked offline. In turn, it caused a chain reaction, causing many of UO’s other game servers—even those stationed in separate data centers across the world—to crash shortly thereafter.

“It made no sense whatsoever,” Mayer said. “Unless… someone was doing it on purpose. Then, such a pattern could be explained.”

Trying to find some method to the madness, Mayer asked the employees in charge of the game’s authentication services to correlate the crashes with players who had logged on before it occurred. It didn’t take long to find a match.

“One player was always on all of the servers before they crashed,” Mayer said. “But what was he doing? We had to know.”

Mayer sought assistance from the customer service department and consulted with a Game Master about tracking the player. With one conversation, Mayer had successfully set up a sting operation in a virtual world.

“I wanted to track this player’s movements in-game,” he said. “When that player logged in, they called me downstairs and took me to a GM’s computer where he was following the player invisibly.”

Whereas any other day, Mayer did everything possible to keep the game’s servers from crashing, on this day he needed it to happen.

A plan was put into motion for the Game Master to dress up as a new player and follow the person of interest. While in disguise, the GM would annoy the player to the point where the suspect would kill him. This death would subsequently cause the player to turn into a murderer, or “go red.”

If all went according to plan, Mayer believed the suspect would crash the server in order to revert it back to its last save point—roughly 30 minutes before he became a murderer—and reveal how his trick was performed.

“Now, I want you to imagine for a minute what it’s like to be a GM, having to listen to every conceivable kind of abuse from players who were pissed enough to call customer service, and take it like a professional,” Mayer laughed. “Then, being given permission by your boss to just unleash on a player.

“This GM went off on this guy and became the most annoying player ever to play UO. The guy couldn’t take much of it and finally killed our poor GM. Now, it was time to kill him. The GM went off screen, turned himself into a poison elemental, and one-shot the guy.”

Just as they’d planned, the suspect had become a murderer. While the suspect was resurrected by his friends, Mayer and the GM sat invisible, hoping he’d divulge his secret to crashing the server.

They didn’t have to wait long.

“After watching this guy curse to his friends about how crappy that was, he said something like, ‘It doesn’t matter. Watch this,’” Mayer recalled. “I saw a stream of unfamiliar characters come into the chat space above his head, and it clicked. T2A shipped with localization for the first time (which attempted to convert foreign languages into English), and clearly there was a problem with the chat message that used Unicode—16-bit instead of 8-bit ASCII—characters.

“That network message was being sent to a buffer without a size check, and that caused a buffer overrun and corrupted the stack.”

The server crashed just as it had over the nights prior. Mayer and the GM had caught their guy red-handed. Mayer coded a fix for the issue and the player was banned from the game.

Despite the constant need for troubleshooting, Mayer said his experience at Origin helped him understand the MMO technologies that ran countless games over the next two decades.

“In 1998, this stuff was largely the Wild West, especially at Electronic Arts,” he said. “EA was the first big game company to make an MMO, and there really wasn’t a lot of standards or best practices developed back then.

“Nowadays, this stuff has matured. Not only have the game companies built this knowledge internally, but they adopt practices used in other industries that look more like online games than they do the single-player ‘ship in a box-type games’ of the past. Each company I’ve gone to since EA reflects growing knowledge and investment in the space as online gaming becomes the norm.”

Since Mayer left Origin in 1999, he has applied the knowledge gained to myriad other projects. Mayer went on to lead programming teams at Sony Online Entertainment, NCSoft, his own company (Night Owl Games), and currently serves as development director at ZeniMax Media’s Bethesda Game Studios Austin where he works on Fallout 76.

Ultima Online’s alpha test started in April of 1996. The month-long trial was the first time anyone outside of the development team had logged in to Britannia to experience Origin’s highly anticipated online world for the first time.

Having spent the better part of a year helping to build said world, Raph “Designer Dragon” Koster held his breath as the server opened and the team’s fantasy world populated.

That’s when everything went wrong.

“On the very first day of the alpha, players appeared in the tavern in Britain,” Koster remembered. “They waited for other newbies to walk out the doors, then they stood at the windows and shot arrows at them until they died.

“The newbies couldn’t shoot back, because the archers were hidden behind the wall inside the building. This is, of course, exactly what archery slits were invented for, but it made for a truly lousy first impression of the game.”

Thankfully, things got better from there.

Koster had been hired at Origin Systems in 1995 to help design something called a “massively multiplayer online game.” While there was some confusion on what that actually meant, Koster felt his game design compass pointing him in the right direction.

Designing games from a young age, Koster spent part of his youth in Lima, Peru, where he adapted popular American arcade titles—unavailable in Peru—into analog games for his schoolmates to enjoy at recess. Upon exiting primary school, he took on the duties of Dungeon Master for a regular Dungeons & Dragons campaign. After getting his hands on an 8-bit computer, he taught himself the coding languages of BASIC and Assembly and made new games for his friends to play, this time in digital form.

Reconnecting with games again in college, Koster and his wife Kristen “Kaige” Koster moved to Tuscaloosa, Alabama, in 1992 to attend graduate school. It was there the Kosters experimented with MUDs and saw the true potential of online fantasy gaming.

“We fairly quickly moved from playing MUDs to making them, and by mid-to-late 1993, we were already working as hobbyist developers on MUDs, particularly LegendMUD, which launched on Valentine’s Day in 1994,” Koster said.

By October of 1994, Kristen and Koster took over the operations of LegendMUD from its original creators, and were constantly designing and innovating new elements.

“Chafing with the limits of LegendMUD, we were working on ideas for a new MUD that had much more robust simulation, including crafting, NPCs with agendas and A.I., creatures with resources, a real player-driven economy, and much more,” Koster said.

It was LegendMUD’s co-founder, Rick Delashmit, who referred the Kosters to Starr Long and Richard Garriott at Origin Systems, knowing the studio was looking to crack the world of online gaming in a big way. Garriott invited the Kosters to apply for open design roles with the company, and the couple gathered their thoughts on how they would approach an online fantasy world bolstered by next-generation graphics.

“We ended up sending in our brainstorm ideas on dynamic ecologies and quest systems as part of our job application,” Koster said.

The pair was hired and reported for duty in Austin on September 1, 1995.

“My wife, Kristen, and I were hired as a pair, and set to work under Andrew Morris, the original lead designer, who was also previously co-lead on Ultima VIII,” Koster said. “Our friend Rick Delashmit had already been working at Origin for many months at that point, developing the original prototype that was built on the Ultima VI engine.

“When we showed up on the first day, we asked what the game was and were shocked to discover it was based on our applications.”

The design elements that the Kosters had pitched were a hit, and the development team went to work implementing them into the next iteration of Garriott’s famed Ultima world.

Though hired as designers, the Kosters realized that in order to make UO what they saw in their mind’s eyes, they’d have to employ a Jack-and-Jill-of-all-trades mentality to make it happen. After all, the prototype they were shown was just that, a prototype. UO would run on a different engine entirely. But that engine didn’t yet exist.

“The engine wasn’t ready yet, so I did things like write design docs, and a huge amount of script in a scripting language that didn’t actually exist yet, all of which I had to throw away and rewrite when the real scripting came online,” Koster said. “I did a ridiculous amount of technical art design, as we figured out how the rendering engine worked.

“Kristen and I did quite a huge amount of the art for the alpha test, because we didn’t have enough artists!”

In Koster’s mind, everything from writing the game’s core script to creating a movable item fell into his “designer” responsibilities.

“I much prefer wearing many hats,” Koster said. “I ended up credited for design, art, and code on UO. And code meant actual client and server code—all the scripting that designers did was counted as design work, not programming.

“That sort of cross-disciplinary thing is still very important to me. In directing large projects, it means I can talk to any of the members of the team in their own lingo and communicate more effectively across departments. In making my own smaller projects, it means I can have a completely unified artistic vision because I can do every job.”

Every day at Origin was marked by opportunities to create, and by cutting their teeth in the world of MUDs, the Kosters were in a position to take their innovative ideas from LegendMUD and show them off on a much bigger stage.

“A huge amount of the design in UO came directly from things that we were doing on LegendMUD, or wanted to do on Legend or a new MUD that we were planning,” Koster said.

“We had been designing virtual ecology and economy stuff for the successor MUD and it ended up in UO. The bardic system in UO was straight from Legend.

“Pets in the MUDs were summoned servants, and we had that in UO as well—elementals, for example—but extending it to actually taming real animals was a conscious step further. Heck, the Wall of Fire that killed Lord British was a spell in LegendMUD.”

After six months of skunkworks development, Ultima Online’s alpha prototype was ready in May of 1996. Inviting around 250 players to experience what the team had created, Koster and his coworkers watched as players tried to unceremoniously burn Britannia to the ground. When the alpha closed on May 20, 1996, the team went to work on the beta iteration, and in the process, the game’s servers were rewritten to handle 2,500-3,000 concurrent players.

Despite all the carnage and destruction that came early in Ultima Online, Koster pointed out other styles of emergent gameplay that did follow what he and Kristen had originally envisioned for the game.

“[Players] used the ability to dye clothing in order to make uniforms for their guilds, and they used it to hold weddings, with coordinated bridesmaid dresses.” Koster said. “They set tables for meals scheduled for the evenings, so that military deployed overseas could have dinner with their family at home. They started holding sporting events. They founded theater troupes and taverns and police forces.”

The release of the game was a milestone for Origin Systems and many of the game’s early developers used the event to get off the train, as it were. Among those who exited the company was Kristen Koster and Andrew Morris, who had served as UO’s lead designer from the beginning.

With Morris gone, all eyes suddenly turned to Koster.

“I do remember when [Morris] left, he came up to me and basically said it was in my hands now and that it would be great, which was a nice thing for him to do,” Koster recalled. “I wasn’t given the title of lead designer at first, because I didn’t have any management experience. I mean, I was 25 years old on my first game job. I was named creative lead.”

Now tasked with leading the Live design team, Koster had to live up to his “Designer Dragon” moniker. Because of UO’s breakout success as the fastest-selling game in Electronic Arts history, additional resources were funneled into the MMO darling. These resources came in the form of the Ultima IX: Ascension development team. Origin pushed pause on the next single-player Ultima game and refocused the team as UO support. While the decision made sense financially, it wasn’t one the Ascension team was particularly happy with, especially since it meant they had to work with a 25-year-old design auteur.

“I checked in with everyone on the team on how individual features were going and whether they fit together with everything else,” Koster said of his day-to-day responsibilities. “If stuff was off track, I had to get one of the producers to give orders to change things if the person didn’t want to listen to me—since they didn’t actually work for me.

“I maintained the patch notes, tracking all changes everywhere in the project. There wasn’t any established practice for that, I had to invent one. I was basically the community manager for the game for quite some time. I wrote game script and built areas and designed new systems. We did 12-hour days for months on end, often without weekends. It was an insane work schedule. You were so tired that you felt almost delirious. I would drive down the road and see tiling patterns in the grass by the side of the road.”

The insane schedule Koster maintained won him the opportunity to lead UO’s first expansion, The Second Age, and earned him the official title of “lead designer.”

Following the expansion’s release—and burned out from more than three years of non-stop work on UO—Koster gave up the design reins and shifted his focus to Origin’s next MMO, Privateer Online, based in the studio’s beloved Wing Commander universe. When the Privateer Online project was canceled in 2000, Origin saw its next mass exodus from the company, but this time the employees who left were joined by Lord British himself. Koster used the opportunity to bow out as well.

Many former employees caught wind that Sony Online might be a good place to land, especially as it prepared to make its own MMO splash with Star Wars Galaxies. Sony welcomed Koster with open arms.

Despite all the design elements Koster and Kristen put into Ultima Online, he said that most of his influences can still be traced to old-school MUDs where technological limitations often breed the best innovation. That conscious innovation was what kept Koster’s wheels turning when he sat down each day to breathe life into UO.

“I think the biggest thing was just pushing the field forward a lot,” Koster said. “Not just going graphical, others were doing that. And not just hitting the large scale—someone would have done that if not us. But the main current was going to just be pure hack and slash without UO.”

Koster and Kristen’s innovations not only inspired Origin’s team of developers, but the countless MMOs that have come along since UO.

“You have housing, crafting, and pets because of UO,” Koster said. “Today’s guild systems, today’s character customization, today’s community management, today’s ability to play economic games, and most importantly, the awareness of virtual worlds as societies and communities.

“Those are specifically things that Kristen and I brought to it all. Probably a lot of them would have come along eventually, but they came when they did because we put them there. And we did it because we had a vision of what virtual worlds could be.”

This excerpt from Braving Britannia: Tales of Life, Love, and Adventure in Ultima Online by Wes Locher was reprinted with the author's permission. The book is now available in both paperback and Kindle editions. Read more excerpts or order your copy via the book’s website. (http://www.bravingbritannia.com)