Neverwinter Nights Client/Server Postmortem

Author: by Scott Greig et al

Replay back to the end of 1998. Y2K was going to eat all of our machines unless every programmer who knew what the acronym COBOL stood for was working on converting all databases to four-digit year formats. Most gamers who could not program were too busy to care because we were playing Starcraft and Half-Life to an obsessive degree. BioWare had just finished a game called Baldur's Gate, and we came back from our Christmas vacations in January 1999, slightly frazzled but assigned to another project: Neverwinter Nights.

Neverwinter Nights (NWN) was supposed to be the best multiplayer Dungeons and Dragons (D&D) role-playing game (RPG) ever made. Not only were we going to tell a story of our own creation, but we were going to enable others to tell their stories, on their own servers. Dungeon Masters (DMs) could log in to a server and help (or hinder) players to their heart's content. Designers could translate their D&D modules from pen and paper via a toolset that we provided for them. It sounded pretty cool to us -- then we realized that we had the task of making it! It really started to sink in when other developers where telling us that they couldn't wait to play it and that they were glad that they weren't developing it.

Although Baldur's Gate was a good single-player RPG, we did a number of things very poorly for the purpose of accomplishing the vast and ambitious goals of Neverwinter Nights. The major issues that we are going to address (both the good and the bad) are the design for user-created content, design limitations imposed by the multiplayer focus of the game, the staffing requirements required to fulfill this vision, developing the game for multiple platforms simultaneously, and reputation systems. It is our hope that you will be able to learn something from what we did right and the problems that we ran into.

User-Created Content

At the core of every game there are a few core requirements that dictate the form of the entire design. Sometimes these are dictated by technology, sometimes by user interface design and game play. In the case of Neverwinter Nights, it was the vision. The vision was to build a game where the user-created content was just as important as the content provided by the game creators. Somehow we needed to get a significant percentage of our users creating custom content. We were sure that the desire was there, it seemed that every fan we talked to had some old pen and paper Dungeons & Dragons modules scribbled onto graph paper collecting dust in their bookshelves. We just needed a system that allowed the fans to dust off those old modules and translate them into a format readable by Neverwinter Nights.

Neverwinter Nights certainly wasn't the first game to try this. Stuart Smith's Adventure Construction Set released by Electronic Arts in 1987 was the first commercial attempt at this. This was a complete RPG construction set. You had the ability to edit maps, created custom graphics, edit creatures and items and it even came with a pre-made mini adventure. The biggest problem with this product was that once you built an adventure you only had your limited circle of friends to share with. If you were lucky, you had a modem and had access to a bulletin board system where you could download a handful of files.

Neverwinter Nights wasn't even the first attempt at RPG construction set for D&D. In 1993, SSI released Unlimited Adventures, using the classic engine and content from their Gold Box series of AD&D games. Like Adventure Construction Set, this was a classic case of an idea that was ahead of its time. It wasn't until the wide spread use of the Internet that a single user could have access to the entire pool of user-created adventures.

We knew that almost the entire user-created content for other games was created by a fraction of a percent of the user base. This presented a real problem. What we really wanted was a role-playing game creation toolset that "even your grandma could use."

When we were first considering the creation of Neverwinter Nights, there was a thriving mod (user modification) community for games such as Quake and later Halflife. While many of these mods were of professional quality, the numbers that were being created didn't justify the effort that creating a full RPG construction tool would entail. It seemed that many of the users that would have liked to built mods for Quake and Halflife just didn't have the technical skills required. On the other hand, games like Starcraft had a thriving custom map community. Some how we needed to take the ability to create 3D levels like in Quake, add to it all the tools needed to create a RPG adventure and still keep the system within the reach of users capable of creating maps in Starcraft.

After some experimentation with arbitrary mesh systems and height field manipulation, we decided to provide a system that used 3D tiles. The major advantage of the system was the simplicity of area creation. Users could simply select a terrain brush and paint rivers, bridges, corridors and roads. If you are expecting a large number of end users to use the tool, then they have to be able to see concrete results the first time that they try it or most of them will give up at that point.

Another advantage of using tiles was that the areas could be represented with only a minimum of data. This was also extremely important. One of the key game play features was hosting an adventure for other players to connect to. What we didn't want was a long delay while the newly connected client downloaded a massive file containing all the art for the area. With a tile system we could send a full area description with just a few kilobytes of data.

The major limitation of any tile system is that there are never enough tiles. There is always just one more special tile or combination of tiles that the end user wants. This is also true for creature and item models too. Since the creation of 3D art is beyond the skills of the majority of the toolset users, we did not include any direct ability for the users to modify the core art resources. We did realize that there would still be a few brave souls who would want to create custom 3D art and other types of game modifications.

Sometimes, the end users can be too clever for their own good (and ours too). When we released Throne of Bhaal, the expansion pack for Baldur's Gate II, there was a popular rule tweak modification in circulation. Unfortunately the creator of this modification released a version during the weeks between our gold candidate and the time it arrived on the store shelves. The problem was that our install package for the expansion pack did not overwrite files with newer time stamps. This caused the expansion pack install program to fail on every user that had that particular modification installed.

When we were designing Neverwinter Nights, we planned for a method of allowing end users to make significant modifications with out breaking the core game or causing the game to crash when non-modified clients tried to join a modified server. The result was the 'hak pak' file (remarkably named because the file extension is .hak). This file would contain all of the non-canonical files (whether introduced by the end-user or modified versions of our files) and would be associated with only specific adventure modules. If a client tried to connect to a server running a module that required a hak pak, the client would be informed and gracefully disconnected.

On the plus side, the hak pak system did accomplish all of these goals. The major problem with the hak pak system was in the way in which art resources are added into Neverwinter. To reduce network bandwidth, art assets were generally referenced by index numbers. For example, the appearance of a creature was dictated by its appearance id. This referred to a row in a table that contained all appearance-related data. This included the name of the 3D model, sound information, movement speed, personal space etc. To create a new custom creature model, a new entry had to be added this table. Storing a custom creature in a hak pak was not a problem since the hak pak also stored a modified version of the appearance table. Unfortunately, as we released more creature appearances or as users tried to merge hak paks together, conflicts in the table row numbers were created. In the end we had to release a hak pak merging utility to rectify this situation. If we could do it all over again, we would consider building a system where all information was encapsulated in a single info file and creatures store the name of the file instead of an index. The server could broadcast the correspondence of new creatures to indices as the initial area is being loaded so that each client would know how to link the hak pak resources to the server's indices.

Another consequence of officially supporting user created content was the amount of testing required. In a normal game you only have to test the content of the actual released game content. If you are officially supporting user created content, you now have the problem of an unlimited testing set. For example, since we needed the pathfinding information to be integrated into the tiles, we had to deal with the fact that not only are all of combinations of tiles too numerous to test, you also have to worry about the users placing objects over key pathfinding locations. In the case of Neverwinter Nights, there was also additional content and features that were not used in the official campaign that also needed significant testing.

Multiplayer Issues

We were acutely aware of the issues surrounding the multiplayer version of Baldur's Gate after its initial release. Integration of key features of Baldur's Gate was still going into the engine two weeks before the ship date. There was little time to go back and fix things to make the same campaign work as well in single and multiplayer.

The first major problem that we wished to address was the issue of bandwidth. Multiplayer was a primary focus of NWN and we wanted to ensure that we did everything in our power to limit the bandwidth consumed by each client. In Baldur's Gate, the bandwidth used up by each client during regular game play usually ran at over 2 kilobytes per second, and we would routinely send around 100 KB save game files to each client while others were joining in. For NWN, we expected to be able to handle six or seven clients through a single 56K server. A rough calculation led us to approximately 500 bytes / second. The approaches taken in Baldur's Gate had to be reworked.

If we wanted to send a single game resource of 100 kilobytes across to the other side (as we did in Baldur's Gate), it would have taken 3 minutes to do so through a 56K server, and most people would have considered this unacceptable. This required us to have all of the data already on the machine during regular gameplay. However, this seemed to conflict directly with user-created content. Thus, we initially shelved the idea of automatically downloading the hak pak file during regular gameplay from a server, since it would be significantly slower than downloading off of another web site. After launch, we discovered that the lack of automatic downloading of hak pak files was a serious barrier to entry for people who wanted to play multiplayer, since almost every module has a different hak pak. This has not been fixed as we write this, but should be fixed by the time you read this!

Since the world was subdivided into a number of different areas, we needed a way of sending down the area's description in the minimum amount of bandwidth. We discussed the tile system earlier, but it isn't just as simple as saying "tile 432 at location x = 20, y = 30". The lighting and shadows in each area had to work in a bandwidth-friendly manner. To make an area look attractive, you sometimes needed hundreds of lights. Since the video cards are unable to compute dynamic lights in these quantities at a reasonable speed, some form of static lighting system was needed. The standard methods, such as shadow maps, were fundamentally too slow to generate on the fly, and they generated data too large to download on the fly. The solution that we eventually come up with was to embed the light information into the tiles themselves. We then only needed to specify which lights for each tile were active and what color they were. When the client received the area description from the server, it then calculated the vertex lighting values for each vertex within the range of that light. Shadows volumes for the sun or moon were also pre-calculated at this time. We did discover that the higher end machines were able to generate shadow volumes on the fly for a limited number of lights. This resulted in the distinctive look of shadows in Neverwinter Nights.

As the last step of bandwidth reduction, we needed to do a better job on reducing the bandwidth used during regular game play, since a lot of redundant data was sent to each client in Baldur's Gate to ensure that each connected machine could run its own AI. However, the server runs all of the AI in NWN. This allowed us to use a networking model similar to the Tribes networking model [Frohnmayer00].

In a multiplayer game with only a few attributes required to display creatures properly, this shouldn't be too taxing on the server, and it wasn't when it was first added. However, we not only had a location, orientation and animation to play, but we also needed to display any one of a number of visual effects on a creature. These visual effects were allowed to stack on a creature to indicate all of the various effects the creature had on it; some creatures such as liches may have up to seven visual effects on them. For the player's creature, they had a number of long lists including prepared, memorized spell lists, as well as prepared and useable feats. The server regularly examined these lists to see if there were any changes to the spell list that needed to be broadcasted. Thus, we had numerous expensive linked list to linked list comparisons for the creature that the client controlled. We ran out of time to optimize this before the game shipped, and we uncovered the beautiful fact that in our zeal to reduce bandwidth usage, the CPU actually became the major bottleneck on the server, preventing some high-end machines from hosting more than 40 player characters.

In spite of the restrictions of downloading files during regular game play, we were still required to send some data (in the form of a character file) from the client to the server. For the pen and paper fans amongst us, it was the equivalent of handing a character sheet to the Dungeon Master and asking "Can I play this character?" Of course, there were all sorts of security issues associated with this. Most servers would not have active DMs that could visually inspect and approve characters, so we were required to write code to validate that the rules were followed in the creation and playing of a character when it was brought into the server. The process of writing the validation was scheduled to take a couple of weeks. However, we severely underestimated the amount of time that this would take, due to the complexities of the D&D rule set. In the end, we actually had to remove some of the level-up validation checks to meet our deadline. Unfortunately, the missing checks were quickly discovered and one of our revisions was made to repair this issue.

What were some of these complexities? In role-playing games, characters also include all of the items that the creature holds, or has in his or her backpack. The majority of the size of each creature wasn't the creature itself, but the items that the creature held! However, the standard items were not that much of a problem. Each item could be examined to see whether a creature had the appropriate abilities to use the item; furthermore, each item could be validated to see if it was appropriate for the creature! One does not expect a first-level character to be carrying around a +5 vorpal sword. Thus, we had to develop a system that could be left on a server to prevent that from happening. This system not only had to work through the import of a character on to the server, but also had to work through the trade of items.

We implemented an item level restriction (ILR) system, which forced you to only have equipped items of a certain cost. If the item was too expensive and the server chose to turn on the ILR system, you couldn't equip it, you couldn't even bring it in your backpack to give to someone else! However, some of the items required for the plot had properties, so plot items were initially given a zero cost. This made all plot items usable by level 1 characters. However, we didn't actually have a different check to equip and to possess the item. Our enterprising QA staff discovered that they could make a "plot" +5 vorpal sword, hand that to a first-level character, and it would be fine to both equip and possess! So, we had to change the system again to impose the item level restriction when the item is equipped, but still maintain its cost at 0.

We also had secret inventory slots for creature hides, claws and bites. This allowed us to implement some of the more bizarre rules and "on hit" properties for Dungeons & Dragons monsters. The end users discovered that, even though they had no in-game method of putting items in these slots, that they could hack the character file, and put the +5 vorpal sword in the creature's "claw" slot, allowing them to swing their fist and have a very good chance of killing characters instantly. The exploit may have taken significantly more time to discover if we did not provide a toolset that showed end users that all creatures had these "hidden" slots!

We could not guarantee that the end-user could not write a script within the scripting language that could take down a server, insert it on their character, and destroy the game experience for others. Thus, clients were forbidden from sending scripts that could be run by the server; hence, there were no scripts on anything in the character file. However, this imposed all sorts of odd restrictions on the game design. In The Sims, each object contains all of the code for how the sim creatures deal with the object. In NWN, we were required to store a global script on each module that dictated how every object that could be brought into the world could be activated. Since end-users could create their own objects, they might accidentally create an object that had exactly the same visual properties (appearance and colors) as a special object in the module. Hence, we had to provide a password system (tags) that allowed module makers to reduce the likelihood of a collision between their objects; only if the passwords matched would the object behave in the way defined by the module. This makes it very difficult for end-users to integrate their own custom content, since they had to cut and paste the appropriate part of their Item Activation script into another module, as well as moving over the item file into the next module. However, security of the server is paramount in a multiplayer game.

Now, you're probably wondering why we went to all of this detail to tell you all of this. The parable is this: Leaving a character in an insecure environment (the client's machine), and giving them the tools to modify their own characters requires a gigantic engineering push to make it work properly. However, it can be made to work.

Our original plan for the NWN story called for approximately 30 2-hour missions that people could play together, enjoy a wonderful experience, and exit from the game. However, market research led us to believe that a vast single-player story was important to end-users; they did not want to see Mechwarrior-style missions where the party would be briefed, charge off like raging lunatics, slay and loot the villains, and then return to the tavern for a nice cup of mead. Thus, the story was reorganized into four chapters, which meant six times as many creatures and objects to deal with in the same amount of RAM!

The mission to chapter conversion required refactoring in some systems to guarantee that the amount of RAM being used was kept under tight control and a number of CPU intensive algorithms had to replace the RAM intensive algorithms that we had used earlier in development! It also increased our save games to a size unbeknownst to mankind. It also forced us to dramatically optimize our file format a couple of months from the ship date to get the save times down to the small levels they are at now. (At the point where the end-users started seeing the beta toolset with the old generic file format in it, some save games were taking 7 minutes to save. Our producer mentioned that he saved his game, made a tuna fish salad, made a sandwich using the salad, ate it, sat back down at the machine and it was still not done saving!)

Of greater concern was that the server wasn't required to have a 2-hour stability between complete reset of campaign data; it had the potential to live and survive for 10 hours in the same campaign world, and anything that could support 10 hours without crashing should be able to handle 24 hours times 7 days without crashing! Unfortunately, we never did get the time to refactor some of the other systems to ensure that the server could stay up for very long periods of time. We are still working on those issues today.

How Many Programmers Does It Take To Screw In Neverwinter Nights?

Neverwinter Nights was by far the largest project that BioWare has ever attempted. By the time in was completed almost everyone in the company had made contributions to it. At its peak, there were 25 programmers working on the project. This was a radical departure from Baldur's Gate, our largest project previously attempted, which only had 7 programmers. The fact that we had so many programmers on Neverwinter Nights was a direct result of the dual need for an unbelievably large feature set, and a relatively short development time of three and a half years. This is comparable to Baldur's Gate, which took us almost three years to the day to program. Our best estimate is that Neverwinter Nights ended up with about double of both the number lines of code and identifiable features. This is a good indication of the loss of efficiency that is generated by increasing the complexity of project while trying to maintain the same time lines. Another interesting "truth" that we learned was the fact that if you have a senior programmer managing and providing technical design for more junior programmers, once the number of junior programmers exceeds 4, then the senior programmer is unable to do any actual programming of their own. Even with a team of experienced people, there still needs to be someone that is ultimately responsible for each major system to keep the system from self-destructing.

With a four-fold increase in programmers, we had to implement new ways of dividing up tasks and maintaining areas of responsibility. Separating out the tools and graphics teams worked very well because of the well-defined points of contact between their systems and the core game. Dividing up the game code itself was a completely different situation. Our experience with Baldur's Gate had taught us some lessons on the importance of integrating the multiplayer code in right from the beginning. We didn't start actually multiplayer programming in Baldur's Gate until over halfway through the project. We had a single programmer go through all of the systems and add the multiplayer support with the predictable significant number of multiplayer problems. In Neverwinter, we started with multiplayer support from day one. We created two teams, one for the client and one for the server with the mandate that only communications between the client and server modules would be through the network layer. In fact for the first year, we only had stand-alone client and sever versions of the program. This separation was continued even after the integrated client/server executable was built. By having separate programmers on the client and server, there was very little temptation to "peek over the wall" and share data directly. This created a game where there essentially wasn't a single player mode and as a result, we had many times less multiplayer specific issues. The downside of this decision was the increased complexity in both implementation and organization. Since every system had to be implemented on both the client and server before it could be tested, it caused several programming bottlenecks as the programmers were waiting for their counterpart to finish. In addition, there was also a significant lessening of ownership for each system. Since no one programmer was directly responsible for the system, it was harder to keep the programmers interested and motivated.

The separation of the programmers that caused problems with the game code did work well between the tool and game programmers. The main difference between these two situations was in the manner that they interacted. First of all, the tool to game data path was essentially a one-way process. The tools wrote out data files and the game reads them. Since the tools were also required to read the data back in for editing purposes, they were able to validate the integrity of the data without the game code being written. Secondly, the data was mostly written into Generic File Format (GFF) files. This was a tagged field file format that allows for adding and deletion of fields without destroying the original data files. This allowed the tools to generate newer versions of the files and still have them readable by older versions of the game. Contrast this against the two-way client/server communication. In this case bandwidth was at a premium causing the message formats to be bit packed. Any changes to either the client or the server broke the system until the other side was also updated.

Another issue was the passing of ownership of game systems. Due to demos and the need to keep ahead of both the design and art teams, game systems were not implemented in the most optimal order. Occasionally, the expertise of a particular programmer was needed on a more critical system and the ownership of the system that they were previously working on had to be reassigned to another programmer. Ideally when this happened, the other system was just delayed until the more critical system was finished. Unfortunately, there were times when both systems were needed for a particular demo. In the case of the inventory system, ownership was passed four times before it was completed. In our experience, the more times that the system was passed along, the more brittle that system became.

We concluded from this experience was that the areas of responsibility should be divided along well defined sections preferably into sections that can be implemented and tested independently by a single programmer. In addition, passing of ownership of any given system should be avoided if at all possible.

Ship of Tools

This is not to say that the Neverwinter Nights toolset did not also have its own set of problems. It didn't take us very long to determine that producing a full role-playing toolset that would be accessible by a large segment of our audience would be a massive undertaking. While we only had a single tools programmer for Baldur's Gate, we had seven at the peak of Neverwinter and we still had to delay some of the optional features like the plot wizard until post release.

The key to making any product user friendly lies in the design and implementation of the user interface. It's acceptable for internal tools to be of the "point click and crash" variety because you can spend time training the development staff not to press particular buttons in particular sequences. Please don't get the impression that we are saying that buggy tools are ok, it's just that doesn't take a lot of revisions of a user interface before you are spending more time developing a tool then you would spend time training people to use a slightly more confusing version. If you are planning on releasing a tool to the general public and expect them to actually use it, then you better assume that most of them are going to give up if they have to crack the manual before they can see any results.

Because the large number of interface revisions that we expected, we decided that the toolset had to be written in a rapid development environment. Since we also wanted to share source code between the toolset and the game, this limited it to a development environment that supported C/C++. Our final choice was Borland's C++ Builder. Overall this was an excellent tool to use although we did run into several incompatibilities between it and the game's development environment, Microsoft's Visual C++.

Another problem involving the toolset was caused by the tight design timeline. Ideally the entire game and toolset should have been completely finished before any designer started to make final content. In reality, the tight timeline forced the designers to create content before the various games systems were complete. This also meant that the toolset was also still in development while the designers were creating final content. Theoretically, the parts of the toolset that the designers were using should not have been affected by new features. Unfortunately, general instabilities were the cause of significant work loss.

One of the biggest problems was the addition of last minute fixes. Since the design team had the tightest schedule, the development of the toolset was scheduled to allow the greatest number of designers to work without being blocked. If a bug or a missing feature blocked any designer, a fixed version of the toolset was immediately posted. Nothing could ever go wrong with this plan! Let's make up some time by steaming through this iceberg patch!

Inconceivably, this had the unfortunate effect of posting untested code and other newly introduced bugs. After a couple of errors, we only did scheduled releases of the toolset (just as we did scheduled builds of the game). Before the designers would have access to the new version of the toolset it had to pass through a complete cycle of testing by our QA team. While we would still have a few designers blocked for a day or so waiting for a new feature or bug fix, the amount of work lost due to crashes or data corruption was significantly reduced.

My Player's Handbook says …

We have a fairly open communication with our fan community at BioWare Corp., and using a licensed rule set (such as Dungeons & Dragons) does get you a lot of fans who are excited about the game. However, this induces a much heavier load on implementation. For example, an obscure exception, which may or may not be mentioned within the help text for a given feat, was unimplemented. Having a number of people, both internal and external to the project, who are intimately familiar with the rules facilitates spotting little errors such as these before the game reaches store shelves.

However, there is another issue when converting a pen-and-paper rule set to the computer. Not only do you have to adequately define, document and test what is being implemented, but you must describe the variances between that and the written rules. Our procedures were somewhat lacking in that regard: internal and external QA were reporting many bugs that were not bugs in NWN, and end users had different expectations based on reading the core rulebooks. Our technical design document, which the designers spent a lot of time writing, was not required reading for everyone who worked on NWN. And the programmers were no better than anyone else! One day, the designers got so frustrated they actually took away the core rulebooks from the programmers' desks to get us to read the document.

Graphic Goodness

Like the designers, the artists were also under significant time pressure. Neverwinter Nights had the dubious privilege of not only being a full role-playing game but a role-playing construction kit too. Since creation of 3D artwork would be beyond the skills of practically the entire user base, Neverwinter Nights needed to include an art set that was as complete as possible. Of course while there was no way of even scratching the surface of the full set of monsters, characters and environments the twenty-five years of Dungeons & Dragons fans have created, we still wanted to provide a representative sample. As a result the art team needed a full three years to produce the required art elements. The only problem was that the graphics engine itself took nearly two years to create. To get the artists working early, we defined the art constraints as best as we could and accepted the eventual passes over the artwork required when the graphic engine was finally completed. Once again, we were sacrificing efficiency for gains in the actual completion date.

You've Got A Bad Reputation Around Here

We asked a few designers about what they thought about working on our next project versus Neverwinter, and we got a unanimous opinion - "it's way easier! We don't have to deal with the NWN reputation system!" This struck us as odd: how could a reputation system become a universally despised system?

The NWN reputation system is described in greater detail elsewhere [Brockington03], but a short description will be required to explain why it was a mess. Every person in the world belongs to a faction, and this faction has a standard reaction to every other faction in the world. These factions are allowed to change through standard mechanics (such as assisting or attacking a member of another faction). Each player character (PC) belongs to its own faction, so that every non-player character (NPC) faction could have its own set of reactions to the player. We intended on having four different factions to start with: Commoners, Merchants, Guards and Monsters, and the designers could add any number of additional factions that they wanted.

Well, let's start with each two-hour mission. The designers did not want all monsters to behave the same way. They wanted some monsters to look friendly to the player, and only attack once the dialog was over. Thus, they had to be in their own faction. Each set of animals had to react independently of one another, so the deer would be put in a different faction from the cows (for example). Plus, if there were important plot creatures, they invariably got factions of their own, too. So, each two-hour mission had approximately 20 factions to keep track of and assign values to. That's 380 "reactions", or 190 if you could assume symmetry between them. That didn't seem too bad.

Then all of the two-hour missions were stapled together into chapters. Most of these factions could not be integrated with one another so four two-hour missions became 80 factions. So, instead of entering 190 reactions, the designer responsible for the module had on the order of 3160 reactions to enter and keep track of. In fact, Chapter 2 (our largest chapter) had over 180 factions near the end of development. (For those of you that are keeping track, it worked out to 42000 reaction values, but only 21000 if you could include symmetry.) This, by itself, required 3 seconds to load and parse in the initial generic file format code, and took 2 megabytes to store. So, we had to revise the file format to make it faster, but the conversion didn't go as smoothly as planned; some data was lost. Designer task: Confirm 21000 reaction values.

So, we have 180 different factions that could be hostile to the player at some time. However, any one of those 180 factions could be hostile to another one of the 180 factions. This led to one of our favorite bugs concerning the reputation system: the disappearing orcs. The orcs were placed in the Aurora Toolset, the module designer then starts the chapter, and plays through the chapter, testing for balance. By the time the designer reaches the area containing the orcs, there is only treasure lying on the ground; the orcs are long gone. The AI was accused of forcing the creatures to drop their treasure and run away. Upon further investigation, the orcs had been told to "wander" near the area that they were standing in. Nearby, there was a large encounter area that spawned deer. However, orcs are hostile to deer. So, the orc would "wander" into the area, and a deer would appear. The orc then proceeds to make short work of the deer. All is fine in the world, but the deer don't like orcs any more. The encounter area resets, and says "there's a hostile creature nearby", and the deer runs headlong at the orc. The orc says "Fine! Deer Stew #2 coming up!". Repeat ad nauseum. Unfortunately, the orc doesn't have an unlimited capacity to heal himself. After about 25 deer are spawned in, they finally get enough lucky attacks on the orc to kill the orc outright. To tie things off, the encounter area would reacquire and destroy the deer, since keeping extra encounter creatures that weren't actively fighting or watched by a PC was just a waste of CPU time. Hence, the orc's treasure would be left on the ground, and no sign of the victorious deer was to be found!

Another fun bug occurs when faction members span over multiple areas. Let us say there are orcs and deer living in peaceful co-existence in many areas of the game. However, orcs hate player characters, and one of them casts an area of effect spell that attacks both a PC, but kills a deer. All bets are off at this point, the deer and orcs are at war. Off on the other side of the world, in a completely different area, the deer rears its head and goes headlong after the orc. The poor orc doesn't understand what is happening … and suddenly all of the ambience of an idyllic setting is destroyed with deer and orc carcasses everywhere. Thus, most factions that spanned multiple areas were subdivided into each area to prevent bizarre behavior. Which means even more values to test and verify!

Each and every time the reputation system changed, it had global ramifications on game play. One wouldn't think that a change to how the game engine handled henchman AI could cause deer and orcs to hate one another … but it did.

Every time the designer thought about the system, they thought about verifying 21000 numbers, retesting all of their areas every time there was a reputation system change, and trying to keep all of the various factions far enough apart so they wouldn't cause a suicidal rampage (at least the Montagues and Capulets had a wall) … and it's not hard to wonder why the reputation system became despised.

Let's Add Another OS!

At the start of the project, we stated that we were going to implement the game on four different platforms: the PC, Macintosh, Linux and BeOS. This was theoretically possible because we managed to isolate the machine-independent libraries from the machine-dependent ones. Everything seemed fine. One of the programmers was responsible for keeping up with the latest build of the game, telling us how we were breaking the build on the other operating systems. Similarly, getting your code base up and running on three different compilers (CodeWarrior, Visual C 6 and gcc) stopped us from making some horrible mistakes that were not caught by default with the settings we had set up.

However, as the project progressed, we encountered numerous issues. The programmer who worked on the porting to other operating systems was the only programmer on the project who was comfortable looking at code anywhere in the code base, and rapidly became the go-to guy when you needed a "little something" done in the code, and we needed a lot of unscheduled "little somethings" in a game as feature-rich as Neverwinter Nights. We also had a horrible time getting driver support for some of the other operating systems. This tended to stall development until the graphics team could help by working around the driver problem, or we waited until the driver support was finally ready.

Last, but not least, we had problems with the toolset component of the project. The Aurora Toolset was built in Borland C++ Builder, based on the fact that we heard all sorts of rumors of a Linux and Macintosh version of Builder. However, this turned into a monstrous headache for development. C++ Builder would throw floating-point exceptions during the middle of the graphics engine, which would not show in the middle of the same library run through Visual C. We couldn't link the graphics library with Builder without going to a DLL, so we had to make the graphics library source tree work on C++ Builder as well. The Macintosh version of Builder vanished into the land of vaporware, and we were left holding the bag.

The server code initially tried to isolate some of the rules into classes that Builder could use, but the tendrils of game code reached in. A decision was made early on that the Aurora Toolset wouldn't attempt to enforce the D&D rule set, that was the domain of the game. After that decision was made, the game code invaded the rules components like shoppers descend on Wal-Mart on the day after Thanksgiving. When it became clear that the decision on rules enforcement was ill-advised, there was little we could do except include the entire server into the Aurora Toolset. So, we are trying to duplicate the rules code in both C++ Builder and Visual C for the most important components. We are investigating alternatives to C++ Builder for future projects.

When the pressure finally hit to get the game shipped, the priorities of getting the Windows client and server done won out over getting the ports finished. Although we managed to salvage enough resources to have a Linux standalone server ready for download when the game hit store shelves, it still took us nearly six months to ship the Mac and Linux clients, and these are now in the final stages (at time of writing).

Conclusions

In June 2002, we received word from our publisher that the Windows version of Neverwinter Nights (translated into English, German and Korean) was done; it was a hairy 3 and a half year ride. Not only did we manage to get the game done in a time frame that was not going to bankrupt the company, but we feel that we achieved all of the marketing hype. We had a single-player game that we were proud to release, but we also had a relatively stable multiplayer framework for playing through the same story, with or without a dungeon master. We also had a toolset that has been used to create over 2000 modules within the first six months.

Our fan community has been giving us wonderful feedback. Their suggestions are improving the game and making it better suited to their needs in the months and years ahead. BioWare has announced two expansions to Neverwinter Nights, which should benefit both the custom content creators and those people who are looking for another dose of the game.

We hope that you've learned quite a lot about some of the trials and tribulations that Neverwinter Nights went through. If you have any other questions, please feel free to ask.

References

[Brockington03] Brockington, Mark, "Building A Reputation System: Hatred, Forgiveness and Surrender in Neverwinter Nights", Massively Multiplayer Game Development, March 2003.
[Frohnmayer00] Frohnmayer, Mark and Gift, Tim, "The TRIBES Engine Networking Model", Game Developers Conference 2000, pp. 191 - 207.

______________________________________________________