The Game Developer Archives: 'Postmortem: Ensemble's Age of Empires'

Author: by Staff

[In this classic March 1998 postmortem reprinted from sister publication Game Developer magazine, Ensemble Studios reveals the story behind seminal PC real-time strategy game Age Of Empires.] I had an experience in a local computer store recently that caused me to burst out laughing. I had stopped to self-indulgently admire the top-ten PC games display when I overheard the following exchange between two young men: "What do you think about this one, Age of Empires?" wondered the first. His companion shot back, “Aww, the Borg at Microsoft just combined Warcraft and Civilization to cash in on these kind of games.” Always eager to boost our sales, I took this opportunity to tell the young men how AoE wasn’t the creative product of a giant corporation, but of a small group of talented people living right in their own backyard. For us, Age of Empires was not only a game of epic proportions, it was an epic journey for a small band of people determined to turn an idea into a real game company. Along the way, we laughed, we cried, we consumed pizza and caffeine, and we learned a great deal about making games. Designing the Past Perfect Obviously, Age of Empires didn’t start out looking like the final product. Despite some accusations, Dawn of Man (AoE’s original title) wasn’t created to be a Warcraft II clone. (In fact, Warcraft II wasn’t released until after AoE’s development was well underway). Instead, the final design was evolved and refined over time, with a few significant design events along the way. One of the best things I think you can have in a game company is a staff that plays a lot of different games. This was true of our staff at Ensemble, and was helped in no small part by programmer Tim Deen’s habit of buying and actually playing almost every new PC game as it came out. It was Tim who brought Warcraft II to the attention of the rest of the Ensemble staff. At that time, many of AoE’s game elements, such as resource management, empire building, and technology research, were taking clear shape. However, we didn’t really know what to do about combat. Warcraft II was a splash of cold water in the face, waking us up to how much fun real-time combat could be. Several times a week, the staff would stay late to play multiplayer Warcraft. These impromptu events continued until AoE reached the point in development when it became more fun to play than Warcraft. Another major shift occurred a little over halfway through development, when the designers were looking at AoE’s localization plans. They realized that AoE would be sold in Asia, but didn’t include a single culture from that region. We held a company-wide meeting and decided to add early Asian civilizations to the early European, African, and Middle-Eastern tribes that we’d already developed. Though the addition would create more work for the artists and designers, the enhanced appeal that the game would have in Asia would make this a profitable decision. All of these changes occurred because the game’s designers weren’t afraid of taking design input from the rest of the staff. Making a game that everyone would be proud of and would want to play was something that got more than just lip service at Ensemble. Perhaps the best example of this core value is the Wonder, the penultimate building that a player can build and use to win the game. In early 1997, AoE was great for slugfests, but everyone felt that the game play needed to offer something more. Numerous ideas were tried and rejected. Then Mark Terrano, our communications programmer, hit upon the idea of building an “Armageddon Clock” that would force players to drop what they’re doing and respond to the new challenge. AoE is chock full of little ideas and touches that were thought up by the artists and programmers. This participation tangibly increased the sense of ownership and pride that we all took in the game. One of things that is truly underappreciated about the designer’s job is play balancing. The designers spent months and months adjusting costs, strength, speed, and many other statistics in an effort to create a game that was fun to play and yet didn’t offer any loopholes or cheats. At this point, I realized that Tim Deen was truly a gamer’s gamer. During development, if any of the various iterations of AoE’s design opened up a way for one player to take advantage of another player and thus make the game one-dimensional, Tim would find it. And when we didn’t believe his assessments, he would promptly show us by using the loophole to kick our butts at the game. For the better part of a year, play balancing was a prominent task, and it paid off in giving AoE more staying power and better game play than many others in the recent crop of real-time strategy games. Blazing the Multiplayer Path Multiplayer support was an integral part of the early design, and AoE was structured in such a way that most of the game could not differentiate between human and computer players. When DirectX first came out, it appeared that DirectPlay would be the best choice for providing communications over the widest variety of connection types. To support a high number of moving units, we went with a modified game synchronous model, where the entire game is simultaneously run on all machines. Only moves, changes, and commands are communicated to other machines. This approach has the advantage of minimizing the amount of data that has to be sent. The unanticipated danger of using this model is that it can generate a condition where the game on one machine becomes out of sync with the game on other machines. This caused some very hard-to-reproduce bugs with AoE. Load metering, the process of determining how much bandwidth the game updates required, was done before the computer AI was completed, and was based on the data flow model taken from human players. As a result, we initially missed the fact that computer players would sometimes issue large numbers of commands in quick bursts. We did, however, address this oversight with the first patch. An area where AoE’s communications worked out better than expected was the game’s ability to dynamically adapt to latency. A sliding window delays the actual game time when a command takes effect, so that all players receive the command and do not have to pause before executing it. The problem of handling players who have dropped from a game presented Mark Terrano with difficult challenges. Since drops are unpredictable, usually there is no way to know what happened. The problem could be the game logic, Winsock, the physical connection, or the ISP, and could exist on either the sender’s or receiver’s side. Getting the game to handle drops by anyone at anytime required a great deal of work. One of the lessons learned from the multiplayer experience was to make full use of communications testing tools, such as automated logs and checksums. Also, we discovered that creating a simple communications data flow simulator program could provide great benefits and isolate the communications code from the rest of the game. DirectPlay also turned out to be problematical. Difficult-to-reproduce bugs, quirky behavior, and poor documentation made the going more difficult. Guaranteed packet delivery for IPX was one of the more notable examples. At the CGDC, Microsoft promised to deliver this feature with DirectX 5 and even included in the beta. However, when DirectX was actually released, this feature was nowhere to be found. The cost of that one missing item was the extra time we had to spend writing our own guaranteed delivery system and a bad packet generator program with which to test it. Painting the Scene All of the 2D sprites in AoE began life as 3D models. Age of Empires contains 20MB of in-game sprite graphics. Even though all of the displays are 2D, we decided early on that all of the graphics in the game would be taken from 3D models. We used 3D Studio and 3D Studio MAX for art production. Because 3D rendering is so time-consuming, each artist was issued two machines each, both usually 200MHz Pentium Pros with 128MB of RAM, which at the time was better equipment than the programmers were using. The objects in the game were created as 3D models that had anywhere from a couple thousand to 100,000 polygons. The models were then textured, animated, and rendered out to a .FLC (Autodesk Animator) file with a fixed 256-color palette. So far, the process I’ve described is identical to that of many other games. At this point, however, the artists added another time-consuming step. The .FLC files were handed off to a 2D specialist, who took the animation apart frame by frame and “cleaned up” each image with Photoshop. The clean-up process involved sharpening detail and smoothing the edges of the irregular shapes. Since most of the sprites in AoE had screen dimensions of only 20 to 100 pixels in each direction, the visual quality improvement that we realized was significant. When AoE was shown at the 1997 E3, the artists received numerous compliments on their work from their peers at other companies. The choice to go with 3D models for the in-game objects provided benefits for other art needs, as they were readily available for use in the static background scenes that appear on the menu, status screens, and the various cinematics. The cinematics, including the three-minute opener, were a fulltime project unto themselves. The 256-color palette (actually 236) used in AoE was something of a problem. The palette was chosen and set in stone at the very beginning of the project, before most of the models and textures had been created. As a result, it turned out that some portions of the color spectrum, such as browns for wood textures, had too few available colors to get the best visual quality. The palette wasn’t revised during the development process because that would have required rerendering and touching up every image in the game ‘ far too expensive time-wise. On the other hand, the palette did have a wide and balanced range of colors, which contributed to the overall bright and cheerful look of the game’s graphics. If we do another 8-bit color game, we’ll generate the palette at a point further along in the development process. Going for Speed Performance is an issue for all but the simplest of games, and it certainly was for AoE. When I joined Ensemble, the game was still in an early form and slow. The two biggest problems were the graphics engine (which was just plain slow) and various update procedures, which produced occasional pauses of up to a second in game play. If we were going to sell to anything but the most cutting-edge systems, some serious optimization was in order. The story gets personal here, as I did a great deal of the work on this part of AoE. I started by trying to get a handle on what the over 100,000 lines of C++ code did (the source would rise to over 220,000 lines before it was finished). After spending a few weeks studying what we had, I proposed a major overhaul of the graphics engine structure, including writing a major portion in assembly. AoE’s original design team asked if the frame rate of a benchmark scenario could be raised from its current 7 - 12 fps to 20 fps. I told them yes. Inside I was sweating bullets, hoping that I could deliver that much improvement. I couldn’t just go ahead and rip out the old graphics engine, as it would hold up everyone else, so I spent the next five months working mostly on the new engine. Along the way, I managed some incremental improvements that upped the frame rate to 10 - 14 fps, but the big breakthrough came during an all-nighter, when the last piece of the new design was put into place. Much to my surprise, the benchmark scenario was now running at 55 fps. It was exciting to come back into the offices the next day and see the formerly sluggish animation running silky smooth. But my work was not all on the graphics engine. I also spent a great deal of time identifying and optimizing myriad processes in the game. Since the game was real-time, many improvements involved spreading out a process over several turns rather than of stopping the game until it completed. In the end, the optimizations paid off handsomely and allowed us to raise the default resolution from 640x480 to 800x600. A practical lesson that I learned from this experience was how much additional overhead and slowdown a game can acquire as it approaches completion. Often, early in a game project the engine will show great performance but the game’s not finished. When you replace the simple test levels with the complex final levels, add all the AI, UI, and bells and whistles, you can find a world of difference in actual performance. This was true for AoE as well. As we approached completion and all of the loose ends were tied off, many of the performance gains were traded in for new features. Things That Worked Out (S)well 1. The Game was broken into separate engine and game components. About halfway through development, there was concern that the code base had expanded far enough beyond the initial design in some areas that every new change and addition would look like an ugly hack. Lead programmer Angelo Laudon and Tim Deen took two weeks and separated the code base into two separate sections, the general engine (Genie), and the game itself (Tribe). The conversion was very successful and allowed the AoE programmers to retain a nicely object-oriented design. The benefit here was that it made the code much easier to modify and extend, saving the programmers a great amount of development time. 2. We made the game database driven. Thanks to the object-oriented design, almost nothing about any object in AoE is hard-coded into the program. Instead, huge tables of information describe every characteristic of every object that appears in the game. The designers used a system of over forty Paradox tables to control and shape the game. As a result, they were able to constantly update and tweak the game, and then test their changes without having to involve a programmer. 3. We stayed in close contact and working together with the publisher. I can’t say enough good things about how the close contact with our publisher, Microsoft, helped the development of AoE. By keeping them "in the loop" on our progress, they worked with us instead of against us as things happened. The best example of how this relationship aided development is the way we handled schedule slippage. Each time something took longer than expected or new problems cropped up, we effectively communicated the delay to Microsoft. With a clear understanding of what was happening and why, they reaffirmed their commitment to assist us in producing a quality game, whatever amount of time that would take. So instead of being panic-stricken and whacked out, we remained professional and focused on our goals. 4. We played our own game. While this may sound simple, it’s very important. Throughout the development process, every person in the company not only play tested, but played AoE with the purpose of having fun. As a result, we were very in tune with why the game was fun, and what people were likely to get out of it. We had 20 guys who were determined not to let the game play be compromised or watered down. 5. Performance was good. Performance truly means a lot if you want your game to have broad appeal. Age of Empires can adequately run an eight-player game in 16MB of RAM on a P120 system. Contrast that with Total Annihilation, which requires 32MB and at least a 200MHz CPU for an eight-player game. Achieving this level of performance required a group effort. The programmers expended extra energy on keeping memory consumption in check and identifying bottlenecks, while the artists culled extra animation frames and reorganized the graphics to maximize free memory. 6.The company respected its employees. I have to say something about the way Ensemble Studios treated its employees and their families. It is well known that software development, especially game development, involves great sacrifices of time and can be hell on personal relationships. Ensemble’s thoughtful management respected that by going out of their way to include families at numerous company dinners and other events, and to make them feel welcome to come by the offices at any time. Additionally, after crunching hard to meet a milestone, they would insist that employees take a couple of days off to catch up with their families. People were allowed flexible schedules if they needed them, and flowers or other tokens of appreciation were sent to the families periodically. The result of this deliberate action by company management should be obvious; company morale and loyalty was higher than I have ever seen in fourteen years of software development. My wife loves my job as much as I do. 7. Localization really worked. From the beginning, we knew that Microsoft wanted to release AoE in six different languages, including Japanese. About halfway through development, we updated our code base to provide full localization support. This required stripping out and replacing all text references in the source code and maintaining all game text in an external resource file. It also placed severe restrictions on how we could draw and display the text. We had to use the Windows GDI exclusively, something most games shun like the plague. It also meant that interface items such as buttons had to be sized to hold the largest possible translated form of their captions, limiting the clever things one could do with the design of the user interface. But we buckled down and did it, following the guidelines exactly. And to our pleasant surprise, the conversion was swift and painless. We felt even better when the translators at Microsoft told us that localizing AoE was the smoothest and most pain-free project they had ever done. The benefit to us was enormous in that we had a single executable program file that was the same for all translated versions of the game, thus avoiding the huge headache that comes with tracking bugs and releasing updates for multiple versions. 8. We worked as a team that respected all its members. A project of AoE’s size required that we all work together in close quarters for extended periods of time. One of our expressed criteria in hiring new people is that we must be able to respect each other. This respect, complemented by the company’s actions towards its employees, fostered an excellent sense of family and team spirit among everyone. We avoided having different groups develop a sense of isolation from the project, and as a result, attitudes and spirits remained high even during the worst crunch time. Had tempers flared and cliques developed, I honestly don’t believe that AoE could have made it out the door in 1997. Things That Went Wrong Or We Could Have Done Better 1. We held the beta test too late in the development cycle. A public beta test of AoE was held in August 1997, but we didn’t come near to exploiting the full potential of it. We were too close to the end of the project to make any game play changes, despite the wealth of useful feedback we received. Manuals were already set to be printed, and most of the design had been set in stone. All we could really do was fix any bugs that were found. For any future projects, we vowed to hold the beta testing earlier. 2. There was inadequate communication with the QA people at Microsoft. For most of the project, bug reporting was handled through a database and developers didn’t directly communicate with the testers. As a result many bugs wound up taking much longer to resolve, and new features went untested. An intermediate database was simply not enough to let testers and developers know what the other was really thinking. In future projects, we would like to assign a specific tester to each programmer and have them communicate by phone every couple of days. Near the end of development, this did happen for a couple people, and for them productivity with testing and bug resolution was drastically improved. 3. We sometimes failed to coordinate development through the leads. Yet another area where personnel communication could have improved the development was among our own teams. Each team Programming, Art, Game Design, and Sound has a lead person who is responsible for keeping track of what each member of his or her team is doing. The lead is the go-to person when someone outside has new requests for the team. As the development of AoE progressed and the pressures rose, adherence to this system broke down as people went direct to get their needs filled quickly. We paid a price for it. People didn’t know about programming changes or new art that was added to the game, and the level of confusion rose, creating a time drain and distraction. We all had to stop at times just to figure out what was going on. 4. We failed to adequately test multi-player games with modem connections. One problem with our development environment is that it isn’t comparable to the typical end user system. During the course of development, the multiplayer portions of AoE were tested extensively. When we played a game in the office, our fast machines, stuffed full of RAM, communicated with each other on our high-speed LAN. When we tested Internet play, our communications were handled through the company’s T1 line. One thing that we failed to realize in our testing was the fact that most players would be using dial-up modem connections to commercial Internet service providers. And it wasn’t just us; the same situation applied to the testing labs at Microsoft. As a result, modem connection games were under-tested. Bugs that were harmless when ping times were low resulted in dropped games for users on slower Internet connections. And our high-speed network masked the fact that under certain not-so-uncommon circumstances, AoE could require 15K of network bandwidth per second -- six times what even a 56K modem can provide on the uplink side. As a result, we were taken a bit by surprise when reports of multiplayer game problems rolled in. Though our first patch fixed this problem, the situation was unacceptable. Now, each developer has a modem and several different ISPs are used, as modem testing is a big part of our testing process. 5. Portions of development relied on products that were not delivered on time. There was a second reason that modem games were under-tested; problems with the delivery and quality of DirectPlay from Microsoft. Features that were promised, and even included in beta releases, weren’t present when the delayed final release was delivered. Direct X 5a wasn’t available to us until a month before the game shipped. In the meantime, our communications programmer was burning the midnight oil writing the functionality that was expected but not delivered. Waiting on promised drivers and SDKs is one of the harder things that developers have to deal with; even those with Microsoft as a publisher have no control over them. 6. We did not plan for a patch. The version 1.0a patch, even though it was a success, was problematic in that as a company we had not planned for it. The general argument is that if you know you are going to need to release a patch, then you shouldn’t be shipping the game in the first place. While one can take that stand, it’s also a fact that no game developer’s testing can match that of the first 50,000 people who buy and play the game. Your customers will do and try things that you never dreamed of, while running on hardware and drivers that you never heard of. Looking around, nearly every significant game released this year has had a patch or update released for it. Rather than deny this reality, we would like to allocate resources and expectations in the future so that we can release any necessary updates days or weeks after our games ship, rather than months. 7. We didn’t manage “surprise” events as well as we could have. During the development period, we experienced several sudden events that caused us, as a company, to stop what we were doing. These events included the creation of a demo version of the game and materials for press coverage of AoE. While most of the events were beneficial to the company, we weren’t very good at handling them, and they threw off our schedules. These disruptions mostly came late in development, when their impact was felt the most. One of our goals for future games is to minimize the impact of unplanned events by giving advance notice when possible and restricting them by minimizing the number of people that have to respond to them. 8. We didn’t take enough advantage of automated testing. In the final weeks of development, we set up the game to automatically play up to eight computers against each other. Additionally, a second computer containing the development platform and debugger could monitor each computer that took part. These games, while randomly generated, were logged so that if anything happened, we could reproduce the exact game over and over until we isolated the problem. The games themselves were allowed to run at an accelerated speed and were left running overnight. This was a great success and helped us in isolating very hard to reproduce problems. Our failure was in not doing this earlier in development; it could have saved us a great deal of time and effort. All of our future production plans now include automated testing from Day One. The Age of Empires development team. Matt Pritchard is front row, second from right. Patching It All Up Once AoE was shipped off to production, we threw ourselves a big party to let off some stress. It turns out we were a bit premature in our revelry. Shortly after AoE arrived on store shelves we began receiving feedback on problems with the path finding, unit AI behaviors, population limits, and multiplayer play. Additionally, some bugs were found that a player could exploit to gain an unfair advantage in the game. Management was stirred to action at both Ensemble and Microsoft, and the decision to do a patch for AoE was made. Although time was taken away from other projects, and it took longer than desired, the patch was a success; it vastly improved the quality of multiplayer games, fixed the bugs, and addressed the game play concerns that had been raised. And that brings the development of AoE to where it is today, which I hope is somewhere on your hard drive... [Matt Pritchard designed the graphics engine for Age of Empires. He also worked on Age of Empires II, Age of Mythology, and BlackSite: Area 51. This article was originally published in the March 1998 issue of Game Developer magazine.]