Bringing Engineering Discipline to Game Development

As the manager of Kesmai Studios, I oversee the internal development division of Kesmai Corporation, the oldest and largest maker of massively multiplayer games. The games we develop (such as Air Warrior, Multiplayer Battletech, Aliens Online and Legends of Kesmai) support hundreds to thousands of simultaneous players via the major online service providers (America Online, Compuserve and others) and over the Internet (at http://www.gamestorm.com). Our games are played much longer than most stand-alone boxed games. For example, two of our current games are over ten years old, and several were released more than five years ago. All of these games have been updated many times as the underlying computer technology has evolved.

A year ago, we decided to change radically our methods of developing games. We were dissatisfied with our maintenance costs, the effort required to achieve our desired level of quality, and our inability to predict production schedules. This article is a description of what we did to achieve this change, although you should view what you're about to read as an "after-action report" - our development processes are continuously evolving to meet our objectives.

What Is This "Engineering Discipline" Thing?

Are you sick of not being able to predict when the development of your game will be complete? Are you tired of creating bug patches for a game that shipped months ago? Are you having trouble adding a feature that the new vice president of marketing demanded halfway through development? If any of these scenarios sound familiar, let me offer you a way of dealing with them.

Engineering discipline, as I define it, is determining and applying processes to the development of game software with the goal of improving the quality, maintainability, extensibility, and schedule visibility of the software. Being able to deliver these particular elements consistently in game software is unprecedented, in my experience.

What Is the Reality of Current Game Software Development?

While both the team and management share a desire for the elements of quality, maintainability, extensibility, and schedule visibility in their software products, the primary mission of most game developers is to ship a fun game by Christmas. The desirable ship dates for most products are dictated by the realities of the retail market. With the majority of all software sales occurring between the months of September and January, the pressure to ship before Christmas is incredible. The final ship date for Christmas is about two weeks before Thanksgiving, given the normal retail distribution delays and processes. Some products succeed when shipping outside of this narrow window, but typically this success is limited to about ten highly promoted and anticipated titles. Odds are that you are not working on one of these titles.

Almost everyone with experience in game development has had to work on someone's legacy code. Most experienced programmers have had to fix an incompatibility or bug for a product developed by someone else, or have had to start a sequel from a previous product code base. In general, game software isn't written to be maintainable. Rather, the primary goal is getting the code "finished enough" to ship it as a commercial product. As such, many game developers believe that any documentation and coding standards come at the expense of getting the game done sooner. This lack of documentation and standards worked for most game developers for a time, as teams used to be relatively small and time to market was the most important issue. But times have changed.

Why Consider An Engineering Discipline?

The main reason that we at Kesmai considered a new way to develop software was that the methods commonly used in the game development business weren't working well. Our products' life cycles were longer than those of most game companies, and our maintenance costs were prohibitive. We also noticed that most products in the games business aren't financially successful, most miss their intended ship dates, and most have significant bugs and require a patch (or three). Within a viciously competitive environment, as an industry we must find a way to lower some of the risks involved with building game software.

Not all risks are controllable while developing game software. You can never be sure if consumers will like what you offer (though there are methodologies for reducing this risk, too). You never know when Microsoft might change the operating system components. Sometimes, a competitive product is unexpectedly introduced, and you must react to it. Management often will dictate large numbers of unexpected changes.

Given this chaotic environment, it's important to acknowledge and manage the risks that are under your control. For risks that you don't control, but are within your sphere of influence, you must make agreements with all parties so that no surprises occur without renegotiation. This gives you a manageable baseline with which to start, so that you can react with more clarity and confidence to the truly uncontrollable risks.

How We Approached an Engineering Discipline

Because our company is in a business where products must be maintained for several years, we felt we had to make a radical change to compete successfully. We first examined the various training opportunities available. We looked at several software development methodologies used by other industries, including the SEI Capability Maturity Model (CMM), ISO 9000, SPICE (Software Process Improvement Capability and dEtermination), and others. We found that all these methodologies contained elements useful to our goal of instituting engineering discipline. They also contained elements that could stifle creativity and ignored the importance of having fun software in favor of schedule predictability.

Because no current methodology that we investigated suited our needs completely, we decided to invest in training our personnel in the SEI's CMM methodology, with the proviso that we didn't want to follow their developmental model exactly. We sent 80 percent of our development and QA personnel to this training. We also investigated several project-management training programs. After one course given by the Learning Tree Company, two of our producers came back very excited and charged up. They really liked the particular instructor, so we contracted that instructor to give a one week class on project management to our entire development staff.

At this point, we had most of our personnel "speaking the same language" and interested in improving our software development methodologies. This was a key step. This kind of change in mindset cannot be mandated from management; it must be a joint effort between developers and management to work. We started our process of instituting an engineering discipline with three initiatives: programming standards, the peer review process, and the software process working group.

Programming standards. We quickly devised a good set of programming standards that the entire development staff could buy into, although there was some debate as to how these standards would be instituted. These standards were created in reaction to the difficulty we experienced in maintaining our legacy code. While these new standards didn't help us maintain our old code, they made sure we wouldn't perpetuate poor coding practices in our new games. Thus, our payoff will come when we hire new people to maintain games in the future. The products that our developers will maintain in the future will have a defined coding style and methods. The table of contents for our programming standards, which illustrates the elements that we believed were important to standardize, are shown in Figure 1.

Peer review process. The definition of the term "peer review" is encapsulated in this quote from the SEI Capability Maturity Model: "The purpose of peer reviews is to remove defects from the software work products early and efficiently. An important corollary effect is to develop a better understanding of the software work products and of defects that might be prevented." Peer reviews, in this context, have no impact on personnel evaluation at all; they're simply a method of process improvement - never a methodology for placing blame. This peer review process only took a few weeks to hammer out, and is now embraced by our developers. Reviews go on almost every week in our studio. The primary value of peer reviews is in finding problems and potential problems long before they would normally be found in the testing stage. A less obvious but equally powerful benefit is that all participants in the peer review process learn to avoid both common and subtle errors in their own work. We encountered some resistance to peer reviews, because there was concern that management would use these reviews to rate our developers. Once it was clear that this was not the case though, our developers embraced the process.

Our first peer reviews found significant problems that had defied other debugging methods, and every peer review we've conducted so far has found and corrected problems. As our people gain more experience with peer reviews, they're uncovering problems earlier in the development process - bugs might otherwise have taken weeks of debugging had they still existed when the game was in beta testing.

When used effectively, peer reviews are a powerful tool to improve both the quality and the effectiveness of your development staff. The relative cost of finding and correcting errors early in the development of a title is incredibly low compared to trying to correct bugs when you're about to ship. If you don't already conduct peer reviews, I recommend starting today.

Software Process Working Group. The Software Process Working Group was responsible for creating and delivering a number of items, which are consolidated into a document called the Kesmai Studios Software Process Guide. The Kesmai Studios life cycle has five phases:

1. Concept
2. Requirements Definition
3. Design
4. Implementation
5. Maintenance

This list is just one of many ways to define a product life cycle. Your company may have different phases and terminology. There is no one true way, as each company must define its life cycle to match its business and market requirements. We picked some new projects to test these processes as we created them. I will use a fictional product called Bazooka Bunny to illustrate some of these phases.

Concept. We created a document to describe what made up a valid game concept for our product life cycle. It's used to select what products are built in our studio. Because our development focus is massively multiplayer games, this quality is heavily emphasized within the concept document guideline, shown in Figure 2.

For the purposes of our development life cycle and internal selection processes, we wanted our concepts to be very compact; other companies might want much more volume and detail. A committee made up of members of the studio approves concepts for further development.

Requirements definition. The requirements definition phase is embodied by two documents, the Requirements Baseline and The Plan. Our Requirements Baseline initiation and revision process is shown in Figure 3. Notice that there is a defined process for creating a Requirements Baseline, as well as for modifying the process of creating the Requirements Baseline (in the event that we decide that the process isn't optimal). All processes used must have feedback mechanisms so that they can continuously improve based on the input from the teams that use the process. A simplified example of a Requirements Baseline for our fictional Bazooka Bunny product is shown in Figure 4.

You may have noticed the large number of departments that are referenced in the Requirements Baseline. Involving so many people at this step is initially time consuming, but it gets all the key departments involved and gives them the ability to give their input about the product. It also secures their agreement that these are their requirements for the product. Any future changes result in the creation of a new agreement, and all the parties must re-approve the changes. While this applies friction to the process of making changes to the project, it does discourage people from suggesting frivolous changes. Additionally, important changes are allowed to move forward with a consensus from the key parties involved. Once these requirements are formalized for the first product, most of these requirements are reused for follow-on products.

The Plan document, alluded to in Figure 3, outlines the general path that development will follow and describes additional elements beyond the key requirements listed in the Requirements Baseline. It also outlines all project deliverables, such as a terrain editor, mission editor, manuals, and so on. The Plan is a living document that changes as the product moves from the requirements phase through the design and implementation phases. The Plan demonstrates to management that the team knows what they are doing and how they will go about getting it done. Specifically, The Plan includes the following items :

Every project develops a large number of work products during the course of the project. Some of these work products are deliverables that are given to people outside of the project team, such as the software itself, user manual(s), and test plans. Other work products are mostly for internal use, such as analysis and design documents, source code, and build procedures. The Plan document outlines all of the work products that will be created and gives a rough timetable and order of delivery. The intended target audience for each deliverable is also defined.

In order to prevent changes from completely derailing a project during development, an effective change control plan must be established early in the project cycle. This process is written into The Plan so that change control procedures are established from the very beginning and are used on every baselined work product.

We define high-level project milestones in The Plan and leave the detailed scheduling information in a separate document. Milestone target dates are initially estimated based on the project scope and budget. As the development progresses and the schedule becomes more accurate, these milestone dates are changed within The Plan. As such, we always keep The Plan up to date.

Design, Implementation & Maintenance

Design. The design of the product flows from the requirements definition. The design is made up of a series of stages that are contained in four main documents under our definition of this phase (Figure 5). These documents are the Software Design Specification, the Software Project Plan, the External Resource Specification, and an initial Software Testing Plan. The Software Design Specification, in turn, is composed of three subphases: creative design, technical design, and resource analysis.

The creative design subphase is made up of a series of four documents, titled Overview, Thematic Content, Interface Storyboards and Prototypes, and GameMechanics. The combination of these documents describes how the game will look, feel, and play. In addition, the initial External Resource Specification is created during the creative design phase, which lists all the art, sounds, and other externally created elements necessary for the product.

The technical design subphase is made up of a series of three documents: Requirements Analysis, System Architecture, and Module Specification.

Finally, the resource analysis subphase is where The Schedule, the final External Resource Specification, and the initial Software Test Plan are all created. The Software Project Plan is simply our schedule for the game's design and implementation, and it's a fairly common document to most projects. The Schedule is based upon the detail provided by the combination of the Software Design Specification and the External Resource Specification. The Software Testing Plan, which is initially created in this phase, is a roadmap for our product testing group to use to validate the functionality of the product.

Implementation. Once the design phase is complete and approved, the project moves into the implementation phase. The implementation phase is divided into the construction subphase and the release-and-refine subphase. Most of our larger products use a staged delivery process, so we construct multiple stages and then move the final stage into a release-and-refine subphase. The Software Testing Plan, which began in the Design phase, is completed during the construction subphase. This is a complete testing plan for the product describing the methodology and elements of a proper test of the product.

The majority of the construction subphase is spent doing the traditional programming, integrating, and testing of the product's modules as it moves towards completion. In each stage of the construction subphase, the Software Testing Plan is being incrementally completed.

Once a project has reached the alpha release milestone, it moves into the release-and-refine subphase (Figure 6). In this subphase, the focus is not on developing new features, but on testing, fixing bugs, and completing the product for the production release to players. During the alpha release stage, it's inevitable that features will be changed or added, but the focus in the release-and-refine subphase is on stability and fine tuning. This is the last development phase of a project, and it consists of four key stages:

1. Alpha Release
2. Closed Beta Release
3. Open Beta Release
4. Production Release

Maintenance. Once a game achieves good stability and it's at the gold-master stage, it moves out of development and into maintenance and enhancement. The project has been completed, and although work will likely continue throughout the life of our massively multiplayer games, it will be at a maintenance level. Most likely, maintenance will be handled by different personnel than the original development team.

Problems, Benefits & Future Expectations

Problems Faced at Kesmai
Lack of management buy-in was not a problem we had, but it might be one that you will encounter. Without the commitment of upper management to instituting engineering discipline through process improvement, it's almost impossible to accomplish any real changes. The process of making significant changes to production methods takes the commitment of everyone involved, and if management changes directions in mid-stream, all benefits can be lost. Our management and our developers devoted the time and effort that it required.

Our first significant problem was coming up with a vocabulary for everyone to use. Common terms meant different things to different people. This problem was primarily a training problem, and sending the majority of our people to SEI CMM training, combined with in-house project management training, largely overcame this problem. However, due to the complexity of the processes involved with developing games, and the number of departments involved outside of development who don't share our special vocabulary, we still find the vocabulary of disciplined engineering problematic.

The next large problem that we encountered was the team's impatience to start working (programming) on a new project. Planning a new game down to the final details before writing a line of code (other than necessary "proof of concept" prototyping) is very counterintuitive to most game programmers. Discipline came into play here, as both management and the development staff kept focused on the new processes even when tempted to follow old patterns of development.

Getting all other departments - such as our publishing, product testing, marketing, and operations departments - to cooperate with our new processes was also a challenge. We overcame many of these problems by including these departments in the creation of our processes. A grassroots, organization-wide commitment to process improvement really made a difference for us in this area, but there are always holdouts who prefer the traditional methods despite potential benefits. You need to have the key people in the departments you work with educated and on-board with your planned improvements.

The sheer amount of paperwork and process involved is daunting, particularly for groups used to a seat-of-the-pants development process. I'm sure many people reading this article wonder how any "real" work gets done with all the paperwork involved. We discovered that all this paperwork inevitably gets done, either by a predefined plan or as an emergency later in the project. In our experience, most slippage in schedules is driven by elements that weren't explicitly defined at the beginning of a project. Our process improvement effort simply acknowledged all the known requirements at the beginning of a project rather than in the midst of development when it adds to both schedule and cost.

Benefits Accrued to Date
T he biggest benefit that we've realized so far has been improved morale. Our development staff is excited to be involved in a process that is progressive and has the opportunity to take our group to a high level of effectiveness. There has been unanimous participation in the creation of and compliance with the new processes. The process changes are showing results, and developers feel more in control of the project. Mid-stream changes in projects still occur, but because projects are now less chaotic, dealing with these changes is much more manageable.

Getting the project baseline requirements written down and agreed to by all departments was a major benefit. When requirements change (as they often do), it puts you in a position to document the change and inform all parties of the effect it will have on the schedule. It allows cost and schedule trade-offs to be done rationally and reduces needless feature creep. Thinking about and documenting these requirements has headed off innumerable future problems.

Having a full design, a technical design, prototyping technology, and solid estimates of programming tasks at our disposal allows us to plan the creation of the art, sound, and editorial resources better. This has helped reduce the false starts and rework of these elements, in comparison to previous projects.

Management has a much higher confidence in the schedules of the products, and a much better idea of where the product is relative to the projected completion date. The actual schedule exists only after the technical design is completed and approved. We only use broad ranges of time prior to the completion of this phase, which requires a high degree of trust between management and development teams.

Future Expectations
We expect that as our personnel gain more familiarity with the processes, we will see further improvements in productivity. We also believe that we'll reap benefits in our maintenance efforts once our products move to that stage. Our standards will dramatically reduce the time that it takes to train new programmers to maintain legacy products. The time that it takes to find bugs and make additions to these legacy games will be significantly reduced. As we move our current set of products into production, we expect the quantity and severity of found bugs to be much lower than was previously the case. The combination of completely planning the product prior to implementation, peer reviews of key software modules, and test plans created by the team should continue to raise the quality of our software. Finally, our time-to-market should decrease as our software development process improves and our teams become accustomed to the new procedures.

It has been exciting to move towards building game software under our new model. Our new procedures allow our developers to be more effective in a normal workday, and although the occasional crunch still occurs, we no longer face grueling, continuous 16-hour days during a project death march. We have already seen a difference in our shop, with people rising to new levels of contribution and productivity simply because the environment allows it. The standard sweatshop atmosphere of game development only allowed the "coding cowboys" to be heroes. Now, we have teams entirely staffed with heroes, doing great work without having to resort to heroics. Being involved with fostering this move to engineering discipline is the most significant effort that I've been involved with in my 20 years of creating games. I believe it has the potential to take much of the chaos and uncertainty out of development and let us focus on creating great game play in our future titles.