Opinion: Debugging Builds From 6,000 Miles Away

[In this reprinted #altdevblogaday-opinion piece, Jury Rig Software technical director Darren Vine guides you through setting up a build process for Windows and Xbox 360 games that will let you track bugs down faster.] One of my least favorite things about programming is debugging reported bugs with little or no information. You know the ones I mean, 'I was testing the game and went to make a cup of tea; when I came back it had crashed. Could not repeat'. During the final weeks, these bugs seem to come in thick and fast, and are hard to track down quickly. When the testing department is on the other side of the world, getting more information is hard. Time differences mean a reply can take a whole day. The build is often stale by then, leaving the team unsure if it has been fixed or not. For these situations, dump files are your best friend. Most platforms have them these days, and they are easy to use. That trickiest bit is setting up your build process. I will cover Windows and Xbox 360 builds in this article, mainly because they are the hardest to setup. I will also mention a bit about other platforms. Building Your Project For all platforms, you must build the project with debug information enabled. For Windows/Xbox, this is pretty simple. In the project properties, set C/C++->General->Debug Information Format to either Program Database or Program Database for Edit and Continue. Also set Linker->Debugging->Generate Debug Info to Yes. For the others (Apple, Sony, etc), you typically need to just use the –g flag. While here, you may as well generate a MAP file as well. This is in the Linker->Debugging window for Microsoft (-Wl,-Map,output.map for most other compilers). Generating Dump Files Most platforms these days have a dump feature. Some platforms do full dumps, but they can be fairly large. 512MB can take a while to download, even with good internet connections. Fortunately, mini Dumps are often just as good and only a couple MB at most. For the Windows apps I add this code to the WinMain function:

Then I add this code somewhere in the project as well

This code won't fire when a debugger is attached but works perfectly on testers' machines. The only difference with a mini dump is that it only dumps the basic information: registers, threads, and stacks. Full dumps include the whole of the system memory as well. Handy in some situations but only when you suspect a memory problem. These are generally easier to track down with a custom allocator (see my other series here). Get the testers to attach these dump files to their crash reports. Backup The Information You need to keep a back up of your executable and PDB files. For Microsoft builds, you really only need the PDBs, but the executable may come in handy. Do the same with the MAP files. Make this part of your build process. If you have a build machine, this is the ideal place to do this. The space requirements are not massive, and they don't need to be stored on a fast machine. The Symbol Server A dump file in itself is pretty pointless to anyone that can't read Assembly. Unfortunately, the number of people on the dev team that can do this is normally few and far between these days. Even if you do understand Assembly, I find a wall of code, stack addresses, and raw values a bit pointless, when I can get better information with no extra work involved. What I really want is for any one on the team to be able to do debug these files. Fortunately, Microsoft has a tool called the symbol server which can help. It is free and you can download it here. The symbol server matches the PDB files to the dump file. This gives you pretty much all the information you would have had, had it crashed on your own PC. Setting up a symbol server is pretty easy. I keep ours on the build machine. Create a directory, and share it for everyone on the network. Typically, I keep this directory within the same directory as the other information I have also backed up. When your build process has completed, you need to run the following command:

This registers the information with the server and prepares it for Visual Studio. Visual Studio Setup You need to point Visual Studio to the server by going to Tools->Options->Debugging->Symbols. Add your network symbol server path to the .pdb locations box. If you don't have it already, I recommend adding http://msdl.microsoft.com/download/symbols as well. These are the Microsoft symbols which are handy. For Xbox users, you need to point it to the symbols in SDK directory. It is important to set up a local cache directory in the box below the pdb locations, as it will really speed things up. The first time you try and debug with Visual Studio, you may find it takes a while. This is because it is copying all the latest data from the MS server and storing it in your cached directory. After the first time, you shouldn't notice any stalls. Also note that if you have a really, really long wait, it is probably because you don't have access to your symbol server. I don't know why Visual Studio takes so long to determine this! Debugging Debugging the files are the same as debugging any other project. Double click on the dump file, and then hit F5 after it has loaded. It will load all the symbols and give you all the information known at the time of the crash. It really is that simple, and everyone on your team can debug crashes from the other side of the world with ease. Other Platforms Other platforms have similar features. For platforms like the Sony PlayStation, I build with debugging, backup the files, and then use another tool to strip the debugging information using the tools provided. This is because you need the executable with symbols to debug (often asked when you load the dump file), but due to TRC requirements, you must not release with symbols present in the executable. Summary I have used the above many, many times in the past. Every platform features some decent debugging aids these days, and dumps are some of the best. They are small and can be sent from anywhere. Importantly, the entire team can now trace bugs quickly and easily. Even if they are still hard to find the exact cause, you do know the rough area to start in. [This piece was reprinted from #AltDevBlogADay, a shared blog initiative started by @mike_acton devoted to giving game developers of all disciplines a place to motivate each other to write regularly about their personal game development passions.]