Drop the Beat: All Walls Must Fall’s Procedural Music Mixer

After nearly 2 years of development, this week we’re launching All Walls Must Fall on Steam Early Access! But not only the game - a fully mastered version of our Original Soundtrack will also be available on Steam, with 100% of Soundtrack income going directly to the artists.

To celebrate, in this post I’ll cover the custom procedural music mixer that our talented Audio Designer, Almut Schwacke, and myself have put together to play music in the game.

Berlin, 2089

All Walls Must Fall takes place in Berlin, in an alternate future where the Cold War never ended - and entirely in a sequence of Berlin nightclubs. We’ve put together a soundtrack to match, with a few tracks from Almut alongside a number from guest composers, who we’ve drawn from both the Berlin Techno scene of today and the world of Video Game composers. You can check out the full OST playlist here, but today I’ll be focusing on the latest track that we’ve put in the game, Synaesthetic by Ben Prunty (FTL, Darkside Detective, Into the Breach). This was part of the Drone Warfare stretch goal that we hit during our Kickstarter campaign. The soundtrack version of the track is awesome, but in-game the track will sound different every time you enter a club where it’s playing, as the music adapts and reacts to the gameplay and the progress of each mission. You can check out the track here, but read on to discover how it works in-game.

The Beat

All Walls Must Fall is a tactical game where every action happens on the beat of the music. We use a hybrid system for keeping track of time that’s similar to a simultaneous turn-based system: the game is paused while the player is deciding on their next course of action, and when the give a command, the game unpauses, the action is played, and then the game pauses again. Every action happens synchronised to the music, and different actions have different timings, so for example, the shotgun has a steady BOOM-BOOM of the double-barrel, whereas the SMG has a quick rat-tat-tat.

This all has to be tightly synchronised to the music, and the music itself must be able to play through the entire mission. To achieve this, as well as have the music dynamically adjust itself based on the gameplay, we decided to create a system that emulates the way a techno producer lays out a track in a sequencer, by separating tracks into loops and having the game tightly control which loop is playing at any one time. We do all this with Unreal’s native audio engine - no third-party plugins are used.

Loops

A “loop” (or “stem”) is a single sound file (essentially a wav) that represents one repeatable piece of music, normally played by a single instrument. In the case of Synaesthetic, the track is playing at 110bpm, and each loop is 8 bars long with a 4/4 time signature. Some examples:

The basic Kick Drum loop that underlays the track

One of the 2 variants of the the main melody

One of the 6 melody variants that the bells can play

This is one of the two variants of the high hat

This is the other variant of the high hat

The composer creates different loops, each one with a different instrument and with a number of variations. These are then delivered to us and are tagged with metadata, which the mixer uses to determine which ones should play at any one time. Synaesthetic has a total of 26 unique loops, spread over 11 instruments.

Mixes and Transitions

At any one time, each instrument can either be playing one of its loops, or no loop at all. A selection of loops is called a “mix”. The game begins a new mix by selecting a valid loop for each instrument in the track, and then choosing somewhat randomly for how long the mix should play - 8, 16, 24 or 32 bars being the valid mix durations for Synaesthetic.

We’ll cover what is and isn’t a “valid” loop later, but for now check out this example of the mixer cycling dynamically through 8-bar mixes.

You can see which loops are playing at any time, and when a mix comes to an end instruments crossfade between loops as necessary to switch between the different mix setups. Note that one bar before the new mix ends, a “transition” sound is played; this is a short wav that’s there to give some anticipation to the change, as well as smooth things over nicely.

An example of a transition

Gameplay States

That about covers the core concept of the mixer: crossfading between loops on the fly. But how does the game know which loops to choose at any one time? This brings us to our tagging system and how that interacts with the gameplay.

Each mission in All Walls Must Fall has the player infiltrate a nightclub with a secret agent, in order to achieve some objective, like recruit a potential informant or gather intel on a smuggling operation. As you explore and uncover the club, you’ll find yourself transitioning between different gameplay states. Each loop is tagged with compatible states, and only those that are tagged as for the current state can be played at any time. On top of that, we also increase an “intensity” value every time the player completes a sub-objective. Loops are also tagged with a range of intensities, and only play when the current intensity is within their range.

Paused/Unpaused

As mentioned above, the game switches between paused and unpaused when the player is interacting with it. When thinking about what to do, the game is paused; when an action is playing out, the game is unpaused, and everybody in the club all moves together. This is one of the key pieces of information that we want the music to convey to the player - if the game is currently paused, waiting for them to act, or not. Most (but not all) of the tracks represent this by adding some very noticeable percussive element when the game is unpaused.

In Synaesthetic, this is achieved simply by bringing in the Kick Drum. All other loops are tagged as being valid for both the paused and unpaused state, but the Kick loop is tagged as unpaused only. You can see (and hear) that in this video - when Kai, our player character, walks into the dancefloor, the state (on the left) changes to Unpaused, and a new kick loop comes in (on the right). When he pauses between moves, the kick goes away again.

Exploration and Combat

There are two main states the game can be in while the player is in a mission: Exploration and Combat. In exploration mode, you’re mostly walking through the club, chatting up bartenders and checking the place out. Everybody’s happy to see you. In Combat mode, however, you’ve been spotted by some people who don’t like your face, and it’s fight or flight time. The core gameplay mechanics of giving your agent orders and having the game paused while you make your decisions are shared between both modes, but we add some extra UI to the combat HUD to give you more tactical info in Combat mode.

For the music, these two modes are reflected through the tagging system. While the game can toggle between paused and unpaused very often, as the player gives orders, combat mode can last for quite some time, allowing the composer to create a different composition for this mode. We want to imply additional tension as the stakes are high: Kai may be a time-traveller, but he’s still very mortal.

Here, Kai has to hack into the club’s computer systems, but they’re guarded. The guard isn’t very impressed when Kai tries name-dropping Zoltan and pulls his weapon. As the combat transition sound plays and the UI changes, the supersaw and some percussion come into the mix.

The Drop (and Rewind)

In combat, you use the pause mechanic to plan out your strategy and take down your opponents. While you have the luxury of time manipulation, for everyone else on the ground, this intense firefight has only lasted a few seconds. Once you take out all your opponents, the game rewinds back to the beginning of the combat sequence and replays the whole thing in real time. This sequence is called The Drop; the music gets turned up to 11 and all the gunfire, shouts and explosions happen in time with the beat.

This is achieved with another two tags: Rewind and Drop. When the player is in control, that’s Planning mode, but once the combat ends, we do two things: first Rewind to the beginning of the combat sequence, and then replay through it in real-time, with the Drop tag applied. In Synaesthetic, some otherworldly glitch loops come in for the rewind, and finally, during the drop, we hear the key melody loop.

As you might guess, we're inspired by the concept of a Drop from Electronic music, when a Break serves to build up tension and anticipation, before the bass comes in and the dancers go crazy. The Planning mode where the actual combat gameplay happens serves to create the tension, the quick Rewind is a thinning out of the track in anticipation, and the Replay is the Drop, with the most intense loops kicking in and the gunfire providing much of the beat.

Intensities

As mentioned, during the mission we increase an arbitrary intensity value at a few points, and new loops come in and out as we do. One of the key moments this happens is when first entering the club. Outside, the intensity level is only 1. When that door opens, we immediately push the intensity up to 2.

As the player goes through the mission and the intensity keeps rising, the music gets more complex with additional loops being added, both new instruments and more ‘intense’ variations of existing ones. By the time you’re at intensity 5 and making a mad dash back to the car, the music is at its most powerful and exciting. Each time the intensity increases, we play a transition designed specifically for the new intensity.

The transition used when entering intensity 2.

The transition used when entering intensity 5.

Tagging and Testing

Putting together the tagging for each track is quite a bit of work - we normally take an initial setup from the composer that gives us the information about what they intended for each mode and intensity, and then play through the game a few times to see how it feels, before making small tweaks and adjustments to get it just right. To help with that, we have a visual display of the mixer that allows us to play with all the settings in real-time to check out all the possible mixes that it can throw out.

It's a system with a lot of moving parts, but without it we wouldn't have been able to build the game we have. We've very grateful to all our composers for putting together such awesome tunes and working with us as we've ironed out the kinks in the system! Check out the game - and the Original Soundtrack - on Steam when it enters Early Access on August 8th.