The Tools Development of Turbine's Asheron’s Call 2

Games bound for today's marketplace require the development of software tools. Whether the tool is a simple script to search text files for art asset file names, a complex model and animation exporter, or a massive world builder, these tools help game developers get the job done quickly and efficiently. The subject of tools arises in practically every Game Developer Postmortem, falling evenly into the "What Went Right" and "What Went Wrong" categories. Our tools development process at Turbine Entertainment Software underwent a lot of change during the Asheron's Call 2: Fallen Kings product cycle. This is what we learned.

During development of the original Asheron's Call, we created tools as needed - even the world-building tools were left until the alpha timeframe. While this off-the-cuff approach to tools development allowed the engineers to spend their time creating fundamental engine features, it left the artists and content designers out in the cold, forcing them to edit many text files by hand. The process of text-file manipulation was error-prone and lengthy; Turbine resolved to do things better.

For AC2, Turbine made a conscious effort to be more tools-aware when developing its next-generation engine. The core engineering group developed tools in tandem with the graphics, client/server, animation, and physics systems. Engineers were asked to expand their object interfaces, encompassing functionality not only for the client but for the tools as well. Tools-awareness was not limited solely to the core engineering group as it was with AC1.

We had several goals in mind while planning for AC2 tools functionality: First, we did not want to lag behind the engine development. The tools would reflect the current state of the engine, allowing users to test and use features immediately. Second, because AC2 would be much larger than AC1, we needed to speed up content iteration. The ability to preview and tweak engine assets without having to reload the client after each modification was critical. Finally, we wanted to hide the complexity of the engine from our users. Interfaces to revision control, logically organized dialog boxes, and a rendering window that was identical to the client facilitated use of the tools. We found that being so closely tied to the development of the Turbine Engine greatly benefited our tools.

What Went Right

1. Management and corporate buy-in. From the beginning, Turbine's vision for our internal tools was far-reaching. Many changes were being made to the Turbine Engine for AC2, so the idea of having a core engineer develop tools only part-time or wait until later in the content cycle was foolhardy and dangerous. We decided to create a dedicated tools position in the core engineering staff, recognizing that having a full-time engineer available for tools development would address many of the artist and content designer concerns. Instead of leaving tools development until the majority of the assets had already been created, the tools would be authored at the same time as the engine.

But what good is a full-time position without a plan? Turbine had several clear goals for the tools development process for AC2 and a roadmap to get there. We planned sufficient time, schedules, and resources for the process: we did not want to be developing tools at the alpha stage or spending nights and weekends cobbling together code to get the job done. We drafted milestones and goals for the tools that aligned to concurrently developing features in the engine. Through tool prototyping, we often discovered engine bugs early in the process. We also learned in the process that while schedules are nice, flexibility in tools development is critical; we often were required to rearrange our milestones and due dates based on functionality that the users needed "now."

Additionally, employees outside the core engineering group provided plenty of support. Artists, game systems engineers, and designers had helpful opinions and input into the tools development process. We brainstormed tool features and prioritized our lists. As the tools lead, I was able to schedule artists and designers to produce content used to prototype tools and work out bugs prior to their release. The company's support made my job that much easier.

2. Leveraging the engine code wherever possible. Turbine built the original set of world-building tools for Asheron's Call using the actual Turbine Engine game user interface. This presented a few problems: long compilations and tweaking screen sizes and positions made tools work unpleasant. Moving forward, Turbine decided that the tools should provide a consistent interface around an engine-rendered viewport. We developed the game on and for the Windows platform, so it seemed natural that the tools should be as well.

Several benefits were immediate: With the user interface and rendering engine separated, code could more easily be shared between the tools and the engine. WYSIWYG - what you saw in the tool-rendering viewport was what you got in the client. Many lighting and visual effects could be previewed and adjusted before building client data, which saved a great deal of time. The flexibility of Visual Studio allowed us to lay out dialog boxes quickly and gave us the ability to use window controls, such as listboxes and tree views, that weren't currently available in the Turbine Engine game UI. We also extended existing window controls to provide object information in new and logically organized ways. Using the English text representations of enumerated type data in listboxes aided the designers with easily understandable names, while at the same time prevented errors that would occur if those enumerated types were entered by hand into a text field. As a result, we dramatically reduced broken builds due to bad data.

A common API of virtual functions was provided for every object in the database. This made each instantiatable object "tools-compliant." Our world-building and art asset tools could query, modify, and update any object with the correct interface calls, which spawned one of the favorite expressions at Turbine: "Every engineer is a tools engineer." Engineers could use the common API to create new database objects with tools functionality built right in. Failure to fill in the API hooks resulted in debug assertions when an object of that type was loaded into the tool. Distributing the workload of tools development at the lowest level forced other engineers to provide previously omitted functionality. I would often find that providing functionality for a new object type became trivial; the majority of code had already been written.

3. Frequent tools design discussions. Turbine's internal tools cover a wide range of functionality. Art and content tools allow the user to preview and tweak all art assets before they are converted into the game data. World-building tools provide fast modification of terrain and the ability to place art assets into the game world easily. Engineering tools automatically generate code and data files and provide an interface to the source control system. At the beginning, there were hundreds of tools tasks that needed to be organized and prioritized.

Early in the tools development process we created two committees: one to address art and asset concerns and one to address world-building concerns. These committees met weekly, consisting of representatives from each functional area at Turbine: art, design, engineering, and game systems. The status meetings permitted opportunities to brainstorm new functionality, discuss problem areas and possible solutions, and reprioritize task lists and schedules. In this way, everyone felt they had input to the tools process, making it "our job" instead of "my job."

Keeping people up-to-date was a high priority. Daily status reports to a centralized mailing list let any interested artist or designer see what was happening to the tools. When new versions of the tools were checked in, another mailing list identified the new features, the bugs fixed, and the next required tasks. Each month, we prioritized the list of outstanding tasks and reshuffled the schedule, if necessary. We probably kept people too informed, but the adage "forewarned is forearmed" was definitely true.

4. Feedback, feedback, feedback. Unlike Turbine's monthly episode cycle where the client program is released to thousands of subscribed users, all internal tools were built daily and provided only to those who wanted them. The customer wasn't hidden behind a bulletin board or an anonymous IP address. And, in some cases, my customer was the person sitting right next to me. When they were unhappy, boy did I hear about it. Continual feedback made the tools better, but we worked hard to create productive feedback loops.

One single point of contact existed for all tools-related questions, problems, and enhancements: me. While this provided me with the opportunity to extend my organization skills, it also gave the users comfort to know that someone was always looking at a problem. ICQ and MSN Messenger gave the artists and content designers a way to contact me at any time. However, walking over to a desk and checking out a problem is often the only way to catch a crash that has no good reproduction case. I made a point to provide feedback as soon as possible, even if I was just letting a user know I couldn't get to their problem immediately or that their enhancement would be added to the list of items to discuss at the next committee meeting.

A program crash is always unwelcome, especially when artists and content designers get in the groove of creating content. When a crash occurs, they will sigh, curse the program, and reload to continue what they were working on. I had no idea how often our tools crashed, and the users were more interested in getting work done than in reporting problems. We overcame mystery crashes by using a common "assert" dialog box in the debug versions of all Turbine's software: the tools, the client, and the server. This dialog box contained a stack trace, an optional message, and a Send E-mail button. Sending the e-mail directed the contents of the dialog box to an internal mailing list. Program crashes still put out the artists and designers, but now there was a clear record of where - and sometimes how - the assert occurred. The entire core development team monitored the assertion mailing lists, which proved to be very useful. For critical problems, we set up the development environment on every machine in the office. If a problem needed resolution immediately, or there was no good reproduction case, we could break out of the assertion dialog box directly into the code and analyze the stack at that point. This was useful for debugging not only the tools, but debug versions of the client as well.

In some cases, the assertion mailing lists were insufficient to track the progress of a problem. When a problem severely affected the performance of the tools, we entered bugs into Bugzilla, the open source defect-tracking system. Bugzilla provided us with several important benefits: an HTML interface for reporting, modifying, and querying bug status; an e-mail interface to notify the appropriate engineers; and, because it's open source, a very attractive price tag. Once the baseline tools functionality was implemented, the ability to get a listing of bugs, priorities, and due dates was indispensable for organizing a daily or weekly task list.

5. We met our goals. There was a sense of accomplishment when Asheron's Call 2: Fallen Kings moved from production to Live Team support, not only from the client and server teams, but with respect to the tools as well. "Meeting our goals" may sound trite, but the Live Team would be using the tools we produced going forward as they created content for the monthly episode cycles. A defined list of objectives let us systematically fulfill the needs of the artists and content designers.

Our first major goal was to speed up content iteration, which we did in several ways. To remove the possibility of entering bad data into the tools, menus and lists were pre-populated only with valid options. Allowing the database objects to reload in place from disk, artists could change models and textures behind the scenes, which eliminated the "modify, shutdown, reload" shuffle. Finally, overnight generation of data became possible by enabling batch processing of various assets.

Hiding the engine complexity from the user was our second major goal. By using an engine-rendered viewport in the tools, we guaranteed that content visualized in the tools would be identical to the client. We also created APIs into external tools; enabling us to add revision control functionality and remove some checkout/merge/check-in problems. Next, we presented coherently organized data; we imposed a visual structure on the data hierarchy and clarified our database inheritance scheme. Lastly, by modifying our build system, we were able to automatically get the latest code from source control, build any number of executables, and e-mail the user with a summary status at the completion of the job.

Overall, our internal tools development was worth the effort we put into it.

What Went Wrong

1. Coding before design. All the old lessons drummed into my head during school still apply: design in any complex software system is crucial and cannot be skipped. In the early phases of tools development, I tended to jump right into the code pile and start hacking out a solution to the problem. This caused no small amount of headaches when a seemingly small task blossomed into a days- or weeklong struggle.

Our art asset tools were originally limited in that we could edit and load only a single entity at a time. To speed content placement, it soon became apparent that we would need to create groups of entities. Doing so meant loading multiple objects into the tool concurrently and providing grouping/ungrouping functionality. I jumped into the task and finished in less than two days. When demonstrating the process to the users, it turned out that several important features had been skipped, prompting questions such as: What world-space position were the entities moved and rotated relative to? If one object in the group was selected, was the whole group selected? Did the tool remember if several different groups were created? My understanding of the problem wasn't complete and it showed. It was a frustrating lesson to be learned all around.

Obviously, more up-front design would have saved lots of time in this case. I now create a one-page, high-level summary of new features, which I have affected parties sign off on. A detailed design, usually down to the class and interface level, takes just less than a day. Applying this procedure cut tools development time by about 30 percent and drastically reduced the number of migraines I take home.

2. Feedback, feedback, feedback. As mentioned previously, creating our feedback loops was a lot of work. In the beginning, there wasn't much feedback from the users. Content designers did not realize that tools programmers didn't use the feedback functionality in the same manner; when the program crashed, they figured, "That's the way it is." Users found creative workarounds for various problems: "Don't press the 'V' key when your object is selected or it will crash." Later in the design process, I got a lot of requests for desired features which ended up being used rarely, if at all. During content crunch periods, feedback was nonexistent.

The dungeon creation process in AC2 highlighted a good example of feedback problems. Originally, the tech specified 400 objects in a dungeon, including all lights, decorations, creatures, and structures. However, late in the AC2 process, designers hated working on dungeons, especially when the number of objects ballooned to more than 1,000 objects: teleporting into and loading a dungeon location could take upwards of 20 minutes. Once loaded, rendering the dungeon yielded one to two frames a second; every object was drawn regardless of its distance from the camera. The content team shouldered these issues stoically, so I didn't find out about it until I had the chance to build a dungeon myself.

It turned out that the process of loading a dungeon recursively built an internal graph of data connections every time a new object was loaded. This system worked fine for 400 objects, but got worse the more objects were added. A few lines of code to pause the data graph construction during loading dungeons and functionality to cull objects at a user configurable distance dropped load times to 45 seconds and increased frame rate to a manageable 15 fps.

I have since learned to schedule some time for myself where I can use the tools in several different circumstances. There's nothing quite like getting a taste of your own medicine to spur improvements.

3. No good testing regimen. Insufficient testing plagues any software designed for internal use only. Many different trade-offs must be considered: How much time and effort is going to be spent testing a product that isn't going to generate revenue? Who is responsible for the testing of the internal tools, the developer or the quality assurance team? And how much of the QA budget can be utilized testing tools instead of the client?

The developer making the change in a specific area would usually test our tools and administer a functional overview test. Such limited and random testing didn't find many problems, so usually the users would end up being the alpha, beta, and production testers. Every so often we ran across problems big enough that they required several new versions of the tool to fully isolate and fix them. Having users on different versions of the code base led to several head-scratching dead-ends before the version inconsistency was recognized.

We started solving this problem with a growing internal QA team and the diligent use of bug-tracking systems for our tools.

4. Engine limitations. Wrapping a rendering viewport with a Windows system provided many benefits, but ended up causing other problems. The world-building tools allowed users to switch between three different workspaces: an entity workspace to preview and tweak art assets, a world-building workspace that allowed content placement and terrain modification, and a dungeon-building workspace for assembling dungeons and placing creatures. Since the tool never knew what the user would be working on, massive amounts of memory were required to make sure all assets were preloaded, adding to the amount of time the tool needed to start up - going from double-click activation to "ready to work" took up to five minutes.

As I mentioned earlier, having the tools and engine closely bound was beneficial, allowing the core engineering team to find bugs in the Turbine Engine earlier. On the other hand, every memory leak or problem in the engine was exacerbated in the tools. One of the goals of the tools called for handling data errors cleanly and permitting the loading process to continue; users would be warned and their object would fail to load, but the program wouldn't crash. Such was not the case: text files that failed parsing by the low-level loader often caused a nonrecoverable crash, which required restarting the tools.

It also turned out that not all engineers were tools engineers. We found a few cases where one of the tools API functions returned "true" from a virtual function but provided no actual code. Developing a Windows application also has its own quirks and can be difficult for engineers unfamiliar with the framework.

Finally, we wrote the tools generically to handle core functionality that could be shared across games using the same version of the Turbine Engine. It turned out to be difficult to integrate game-specific functionality on top of the existing tool set.

Going forward, we have resolved to educate our engineers, create smarter objects, handle assets smartly and completely, and use profiling tools whenever possible. We're still working on the load times.

5. Lack of tools documentation. Implementing new features and fixing bugs requires the majority of tool development time. There doesn't seem to be enough time in the day to write documents and get the work done. "How-to" documentation takes a good amountof time to do well, and it's a task not everyone enjoys. The AC2 content process suffered from a lack of tools documentation, and since the tools knowledge had to be passed by word-of-mouth from content designer to new hire, we lost valuable time. There were a few short documents describing complex features, but roughly 30 to 40 percent of the functionality went unused because no one knew it was available.

Moving forward, we created an internal "tools news" mailing list. When new tools were compiled, the list was updated with the new version's features, bugs resolved, and short "how-to" information. However, not everyone read the mailing list, and I would regularly get questions for how processes worked and feature requests for tools that already existed. In the future, we hope to have more documentation and as a team be more committed to both updating and referencing it.

Tools of the Trade

As games become larger and more complex, the potential for build-breaking errors becomes more and more likely. Software tools - both internal and external - can provide many features: error checking, error prevention, content generation, and productivity enhancements. If someone must do a repetitive, complex, or boring task, a software solution is ideal. During Asheron's Call 2, Turbine vowed to be smarter about tools development and planned accordingly.

At Turbine, we devoted roughly 20 percent of our core engineering budget to tools work. With a dedicated engineer and a plan for tools functionality, we knew exactly what we needed to accomplish and how we hoped to do it. The team found that sharing code between the Turbine Engine and the tools saved a lot of time previewing and tweaking art assets. We learned that design is critical for any software project, not just our client software. Feedback and discussions gave everyone a vested interest in the tools. We've got some work to do for testing and documentation, but we're moving in the right direction.

Tools developers know theirs isn't the most glorified position in a game company, but it is one of the most critical. People at fan gatherings don't want to talk to you, and half the people in your company don't know exactly what you do. Your job can be the bottleneck that strangles a game, and you may ask, "Is it worth it?" A content designer once told me that in the original Asheron's Call, he destroyed the town of Arwic using the first-generation terraforming and content placement tools. Time spent? One week. Modifying a similar set of terrain for Asheron's Call 2 with the new tools took an hour and a half. I'd call that improvement.

______________________________________________________