Analysis: The Heartbeat Of A Game Project

[In this analysis, originally printed in Game Developer magazine earlier this year, former High Moon Studios programmer Noel Llopis provides a guide for setting up your own build server to quickly and reliably compile code for various platforms or ensure assets and levels load correctly, leaving you to work on what's really important -- the game.] Have you ever given some thought to why you decided to become a game programmer? I’m pretty sure it wasn’t to do mundane, repetitive tasks. Yet sometimes we find ourselves spending a significant portion of our time making sure that the code compiles for all platforms, or that there are no potential bugs lurking in the depths of the game, or even building the assets for each level and running them to make sure they load correctly. Clearly, those are all things that need to be done, but if they are so repetitive and mindless, couldn’t we put some of the computers around us to good use and have them do the job for us? A build server will do all that and more, much faster and more reliably than we could, and it will free us to work on the thing that made us fall in love with this industry in the first place: the game. Getting Off The Ground Before we can start thinking about setting up a build server, we need to be able to build the game with a single command from the command line. No clicking around, no GUI apps, no multiple steps, no magical incantations that only work during a full moon. Just type a command and the build for the game and all its libraries starts. This is not just a necessary step to set up a build server; it’s a very good engineering practice. So if you’re not there, spend some time on it right away and you’ll be glad you did when candidate submission time comes around. Building the game with a single command should be fairly easy, but the specifics will depend on your environment and build system. If you’re using Visual Studio, you can put the game and all the libraries in a single solution with the correct dependencies. Then you can invoke the devenv.com command line program specifying the solution and configuration you want to build: devenv.com mygame.sln /build Debug. You can wrap that up in a single batch file buildgame.bat for extra convenience and you’re done. If you’re using another build system, such as make or jam, you can probably already build it with a single command. If you’re using a bunch of mode-made scripts, at least wrap them all up in a single script file so they can be run with a single command. Just building the game with a single command isn’t enough. We must have a way to automatically detect whether the build succeeded or failed. Fortunately, most build systems (including devenv. com and make) return an error code when the build fails. If you’re rolling your own build script file, make sure to capture build failure and return an error code as well. Setting Up The Build Server A build server should be a dedicated machine with access to version control. Whenever a new build is needed, the server syncs to the latest version in version control, starts a build, and notifies the team in case of any errors. That’s a fine start, but we could make things much better. For example, we could include the error message in the notification email so programmers can see right there what the problem was instead of being forced to sync and build the game themselves. We could also trigger a build in different circumstances (for example, code checked-in, forced by a person, or some other event) instead of only at fixed intervals. We might want to distribute the build across multiple machines, or keep logs and make them available on a web page, or format emails better, or use more direct notification methods … Put away those Python reference manuals because fortunately, someone has already done all the work for us: CruiseControl (and CruiseControl.Net). It’s a free, open source build server program with all the bells and whistles that you could possibly want. And did I mention it’s free? There are three main parts to it: 1. The build server. It runs as an application or a Windows service. It’s configured through a very simple XML file that tells it when to sync, where to sync, what to build, and how to report it. 2.The web front end. CruiseControl features a pretty, web-based dashboard showing all the builds, their status, past logs, and other pertinent data. 3. The system tray notificator. This is a little app that runs in the system tray and shows the status of all the builds and notifies you of any changes right away with a message and by playing some sounds. This is my favorite way to keep up to date with the build status. You’ll be up and running in about 10 minutes. The most complicated part is probably installing a web server (if you don’t already have one) and getting the web dashboard running. You’ll spend a few more hours tinkering with it to get it “just right,” and then you’re done. The only time I have to mess with it is to upgrade to a new version every so often. Other than that, it’s virtually maintenance free. At this point you’ll have a fully featured build server in place. It verifies that the game can be built from the latest checked-in version of the code. It notifies developers of failed and successful builds right away. It increases version numbers, keeps a build history and statistics, archives executables, and emails logs. CruiseControl and CruiseControl.Net are the two build servers I have most experience with. There are other build servers out there, with slightly different features, integrations with different environments, and so forth. Some of them are commercial and come with full support in case you’re more comfortable with that model. It’s important to stress that a build server is not intended to be the only machine that builds the game. Every programmer (and maybe every member of the team) should be able to build the game in his or her own machine from scratch. The build server is there to verify that all the checked-in changes build correctly on a clean machine, and to make sure that all platforms and configurations are building successfully. Any official builds should be created exclusively from the build server, though. Especially any builds distributed externally to publishers or manufacturers. This ensures that the build is clean, was created in a repeatable manner, and is free of any idiosyncrasies from a particular machine. How Often? Once the build server is in place and is producing successful builds reliably, the question arises of how often to make builds of the game. It used to be considered good practice to do a weekly build. The team would start ramping things up on Thursday to try and get a build out the door by the end of the day on Friday. Anybody who has done that knows how stressful it can be and how it can easily become a bottleneck. Why wait a week if you can do one every night? More teams started switching to the daily build, which is much less stressful because there are fewer changes in each new build. It also gives the team a chance to fix anything that was found broken in the previous day’s build. Soon, teams took it beyond the daily build and started making two builds a day, or even one every hour. The build server has been very appropriately described as the heartbeat of the project. A “green build” is one heartbeat and one small step forward. A “red build” is done when something is wrong and needs to get fixed as soon as possible. If you have a red build several days in a row, the project is in serious trouble. The more often you make a successful build, the better. You’ll find fewer surprises and stay more on course that way. My favorite approach is continuous integration. With continuous integration, the build server starts a new build as soon as there’s a new check-in. If multiple check-ins come in while the build is in progress, another build starts right after it’s done, with all the new changes queued during that time. When following this practice, programmers sync to the latest version often, make small changes, and check-in code frequently, rather than batching many changes. Very conveniently, Cruise Control has a setting to start builds whenever anything changes. The main benefit of continuous integration is that you are notified as soon as a check-in breaks the build—not a day later, or even an hour later, but minutes later. It tells you, “The last build was good. This one is not.” You can look through the last couple of check-ins that happened during that short time period and quickly narrow down the problem and fix it. Imagine trying to narrow down an elusive crash bug from all the check- ins for a full day or two! Another benefit is that all programmers are working on a version very close to the latest one. This means that there are fewer source code conflicts when checking-in code, and fewer surprises lurking in the code. The flip side of that is that working on the latest version is living in the proverbial bleeding edge. It’s not unusual for someone to check-in code that has some accidental bad side effects. As long as those bad check-ins are limited, and that whenever they happen they are fixed right away, I have found the benefits to outweigh some instability in the main branch. Some of the ways to minimize disruptions when working with continuous integration are: - Make sure that any code compiles before checking it in (that should go without saying!) - Execute a fast set of unit tests to verify that basic functionality is working correctly, and - Have the build server notify everybody as soon as there’s a broken build so it can be fixed and so that nobody else syncs or checks-in any code while the build is broken. Need For Speed Ideally, I’d like to check in some code and see whether the build server found any problems right away. In the real world, things can be much slower. After all, the build server needs to sync to the latest code, kick off builds for multiple platforms and multiple configurations, and perform some other time-consuming steps. Even so, there is work we can do to get feedback as soon as possible. Perform incremental builds during the day, so only the affected sections of the code need to be built. It’s still a good idea to do a full build at least every night to make sure that everything can be built from scratch. Set up each platform and configuration as separate builds. That way you get feedback as soon as one of them completes. The only downside is if an error makes it through that causes all the builds to fail, get ready for lots and lots of broken build sounds playing all over the company. Speed up build times through good physical dependencies, modularity, precompiled headers, and good use of forward declarations. Split up different builds and configurations in different machines. The easiest way is to set up one machine per platform and configuration (or maybe do a couple of configurations per machine). Cruise Control lets you easily integrate several build servers into the same web dashboard and system tray application, so this is a very easy solution. Don’t Skimp on Hardware Get the beefiest computers you can afford. Throw fast CPUs, disk access, and gigabit ethernet. Get multiprocessor cores and make sure your build system takes advantage of them. Does it sound like a lot of money? Not when you take into account how few servers you’ll have and how much time you’ll save all the members of the team. I have tried several distributed build systems, and even though they can sometimes be beneficial for some codebases, I’m still not a huge fan. I find that you can often achieve the same (or better) results by using multiple processors and good build architectures, and you avoid the complexity and overhead of a distributed build system. One “gotcha” we ran into when we scaled our build farm beyond about 15 build servers was that each of them was hitting our version control repository every few seconds to see if anything had changed. That wasn’t a trivial operation, and so many servers doing it so frequently definitely slowed things down to a crawl. To remedy that, instead of having the build servers poll the overtaxed source control server, we had the source control server push out a notification. Whenever there was a check-in, the source control server changed a timestamp in a file located on an internal web server. We changed the build servers to constantly monitor the internal web server for changes in that file, and whenever it changed it triggered a build, which completely eliminated the overhead on the version control server. Beyond The Build So far, we’ve only been talking about building the game. But the build server is a great tool that we can put to good use for many other purposes. Why restrict ourselves to just the game? All the in-house tools would also benefit from getting the same treatment. We can even take it a step further and deploy the freshly-built copies of all the tools on a network drive or web page so they’re available to the whole team. The build server can also double up as a symbol server. That makes it much more convenient for programmers to debug an earlier version of the game and libraries and have all the debugging information available without having to rebuild everything locally. There’s no reason to limit the build server to just building source code. One of the most useful things you can do with it is use it to build game assets as well. Building assets is usually a slow process. Having a fast asset build system that can correctly perform incremental builds is crucial to keep asset build times down. Build servers are general enough to perform just about any task. Running both unit tests (small tests on each class or function) and functional tests (tests that exercise a larger module or even the whole game) are perfect uses for the build server. Functional tests can be pretty slow, so make sure that they’re treated as a separate build and not as the last step in building the game. Nobody wants to wait for hours for all the functional tests to complete before they can see the successful build status after a check-in. The sky is the limit with what the build server can do. We use it to run static analysis of our source code, checking for spots in the code that can lead to subtle and dangerous bugs (uninitialized variables, implicit type conversions, and the like). Another great use is to run through the different levels of the game, recording frame rate at different points of each level, logging the results, and failing the build if it ever drops below a certain threshold. Having the performance history for specific levels can be really useful to narrow down why a particular section is chugging at 20fps but was running at a solid 60 a couple of weeks ago. For bonus points, integrate all the collected data into easy-to-visualize graphs available through the web front end. The build server is definitely the heartbeat of a project. Keep those check-ins coming and those builds green, and you know you’re heading in the right direction. [Noel Llopis regularly contributes articles to Game Developer Magazine and the Game Programming Gems series, and he is the author of the book C++ for Game Programmers. Some of his past titles include The Bourne Conspiracy, Darkwatch, and the Mechassault series. He earned an M.S. in computer science from the University of North Carolina at Chapel Hill.]