The Linux land has a reputation, especially among developers used to Windows, of being – let's say – somewhat savage, uncivilized. We've all heard the ghost stories: things being downright broken, lack of documentation and general despair; people coming, exclaiming: "what the fuck?!" and going right back. 

Well, I've been exposed to the Unix philosophy, coding practices and general way of things for just under 10 years. This experience makes me comfortable with the platform, but I've still had a lot to learn in the past year in making the transition from Linux development as a hobby to making a living out of it. I'd like to share some of the less obvious tricks I've put up my sleeve – this time regarding the build system.

Unless mentioned otherwise, I will assume a 64-bit Debian system, "testing" branch (this is what SteamOS is based off of).

Multilib

This section may not apply to you if you don't meet these two conditions:

If you're fine with 64-bit only or the SR toolchain, you may skip all the way to Clang.

If you, however, try building a 32-bit program on a 64-bit system (or the other way around) with the stock toolchain, you will most likely fail to build it, despite setting the correct architecture on it with a -m32, -m64 or a -march switch. Apparently, you are somehow missing the right libc and libgcc packages, even though they may seem to be installed.

The point of failure may be different, if any dependencies apart from the standard library are used in the program. That is usually a trivial issue, though, fixed by enabling multiarch and getting the :i386 package (for Debian, more on that here).

I used to maintain a schroot (kind of like a VM, but native – the kernel and some parts of the file system are shared with the host) with a 32-bit variant of the system just for the sake of building 32-bit binaries; however, there is a better way.

Enter gcc-multilib. Install this package and suddenly the 32/64-bit cross-compiles start working as expected. It's magic!

Clang

Okay, this one has been repeated ad nauseam recently, but it really makes a world of difference; also in terms of build times. A full rebuild of my current project takes:

You may say ~40 seconds is not a big deal; it is to me.

Not mentioning the not-so-obvious benefit of having another point of reference. And, given the fact that Clang aspires for full command line compatibility with gcc and usually builds everything that gcc can (it even defines the __GNU__ macro alongside __clang__, and supports the GNU extensions to the language), it really is a drop-in replacement.

It brings a lot of good and can be quickly switched back for gcc. What's not to like?

Linking

This build step is a major source of grief for anyone who's ever worked with a AAA codebase, as linking several hundred of object files and libraries can take ages. The Linux situation is made even worse by its aging linker, the GNU linker a.k.a. ld, which is single-threaded, requires objects and libraries to be specified in the order of dependency etc.

gold

The good news is that we are not doomed to use GNU ld. A few years back a small team at Google recognized the problem and decided to fix it, resulting in the development of GNU gold – a brand new linker built from scratch, whose most prominent feature is that it's multi-threaded.

Switching from ld to gold in most cases is a question of a command line switch for GCC/Clang, or at most making a symlink. Here's the command I use to link my current project's binary on a Debian system, stripped of all irrelevant options:

clang++ -B/usr/lib/gold-ld

For my current project, switching the linker from GNU ld to gold took the link time down from 18 to 5 seconds on my Xeon E3-1240 v3.

Library groups

Another thing the GNU ld is infamous for is the seemingly unreasonable requirement of specifying libraries in the order of reverse dependence (i.e. an object's or library's dependencies follow the object or library; there is a great explanation of that on Stack Overflow, if you're interested). This can get very annoying once you have to deal with circular dependencies, and while there is a way to do this "properly" – specify the library again – I'd much rather not have to think about it.

Turns out there are command-line switches that do just that (see the ld manual) – we can define a group of archives:

--start-group archives --end-group
 The archives should be a list of archive files. They may be either explicit file names, or -l options.

Or the short-hand version:

-( archives -)

Just dump your libs within the brackets and you're set.

There is a caveat, though: this causes the linker to perform an exhaustive search through all the libs in the group until all symbols are resolved. The manual warns of this having a possible performance impact. I haven't noticed any in my project, but just keep that in mind.

Debugigng symbols

Bruce Dawson has given an excellent talk about Linux debugging at Steam Developer Days 2014, so I'm not going to repeat here what he said. Instead I'll just reiterate the two main points about symbols:

There is one thing that Bruce has not mentioned in the talk, though, and which is – in my opinion – crucial to daily work. Update: He did suggest it to me in a tweet post-factum, though, so kudos to him anyway!

Caching the gdb-index

When working with a large codebase, such as a triple-A-grade game engine, you may notice that the debugging symbols may be... well, somewhat heavy:

$ ls -sh ADVGame-Linux32-Debug.dbg
244M ADVGame-Linux32-Debug.dbg
$ ls -sh SuperSecretProject.dbg
254M SuperSecretProject.dbg

Ouch.

What's even worse is that those ~250 megs need to be loaded by the debugger and processed to build an internal data structure for quick symbol lookup; with the files above, this can take 10 to 30 seconds on my machine, and it happens every single time you load the binary in GDB.

There goes fast iteration time.

We can, however, cache this data structure offline instead and fold it into the build process! I think it's a fair tradeoff between build time and runtime, considering that the latter almost always implies the index generation cost anyway. The GDB manual describes the method well and concisely; here's what I'm using for my project (assuming SuperSecretProject is the game binary):

mkdir -p $(OUTPUT_PATH)/gdb-index
gdb -batch -ex "save gdb-index $(OUTPUT_PATH)/gdb-index" SuperSecretProject
objcopy --add-section .gdb_index=$(OUTPUT_PATH)/gdb-index/SuperSecretProject.gdb-index --set-section-flags .gdb_index=readonly SuperSecretProject SuperSecretProject

Some trivial variable substitution makes this look less like a magic incantation. I promise!

As David Konerding has pointed out in the comments, newer releases of the gold linker can build the gdb-index for you at link time, which should be faster and is obviously easier (no GDB batch mode commands hodgepodge!). It's now a question of appending --gdb-index to your linker command line, or -Wl,--gdb-index, if you're using the gcc or clang compiler driver to link.

Thanks, David!

Conclusions

Hopefully, at least some of these tricks have been new and useful to you. If that's the case - glad I could help! If not – why not share your tricks in the comments?

Also, I know what some developers are thinking: we shouldn't need to be doing all this, we shouldn't need to do so much research and documentation crunching just to do such basic things. And you're probably right. I'd like to explore this non-orthogonality between the Unix philosophy and the game developer mindset in a future blog post.