Composing Adaptive Music for Games - Advanced Concepts in Elias

Composing Adaptive Music for Games - Advanced Concepts in Elias

“The days of pure pencil-and-paper music composition are gone for good.  If we wish to compose music in modern times, we must be able to navigate the wonderful world of computers and technology.”  - Winifred Phillips, “A Composer’s Guide to Game Music”

Introduction

In my last article, I introduced some of the basic concepts of writing adaptive music using a middleware tool called Elias from Elias Software.  In that article I promised to go into more detail and depth about the advanced features of Elias, and so let’s dive in!  If you did not have a chance to read it yet, it would be a good idea to familiarize yourself with it here.

Elias, like a good video game, is easy to use but difficult to master. That means some detailed explanation is necessary to understand some of the advanced capabilities of the tool. This is a new type of tool with some new features, and therefore there are some terms that need more explanation, such as Objective Mode, Exploration Mode, Agility, Rendezvous, and Urgency.  I will also go into Reverb and Reverb Tails; Fades and Crossfades; Agility and setting Custom Agility Beatpoints; Changing Keys in Elias and understanding the “Rendezvous” setting; and the concept and usage of Pickup Beats for stingers. I will also discuss some of the optimal settings for reverb and fades.  

I take a fairly deep dive into the inner workings of Elias, but I also have created some videos for you to watch that will help to show the principles with real world examples, including a song I wrote specifically for this blog called Blade Revisited. By the time you finish this article, you should understand all the finer points of Elias and be able to use it like a pro!

Let's check out Blade Revisited and an overview of some of the features I will be going into in this article:

Objective and Exploration Modes - In Theory and Practice

Objective mode is used in scenarios where the music needs to go from one phase of gameplay to another. In the score I created for this article, imagine you are a futuristic bounty hunter wandering through the techno-slums in a world similar to Blade Runner or games like Deus Ex.  There is some tension building, but primarily you are just exploring the city.  Then, out of the corner of your eyes you see your target, the evil Gothmong.  Your heart quickens as you begin to chase Gothmong through the underbelly of the city. Eventually, you catch up to him and he is surrounded by his loyal goons, and a battle ensues.

In order to score this sequence of events, you would first need an exploration theme to play as you wander the city in search of Gothmong.  Then you would switch to an objective mode for the chase leading into the battle mode.  The battle would grow more and more intense. The intensity of the music could be correlated and controlled by game parameters such as your health and also Gothmong’s health or number of henchmen left.  It would need to be able to play different outcomes based on your success or failure.  Finally, if you succeed, then you would need the music to settle back down into the calmer, exploration mode theme.

Let's take a listen to the three "scenes" of Blade Revisited I created using Elias for this article:

Exploration mode plays different layers in the tracks randomly.  One of the most annoying things about looped music in video games is that if you spend a lot of time in a particular area, you will hear the same music over and over again until you are sick of it. One of the big advantages of the exploration mode is variety.  If you had 6 tracks with 6 variations (drums, bass, melody, harmonies, etc.) then you would have 6 to the power of 6 variations, or 46,656 variations!  One of the most important things to remember when writing exploration themes in Elias is to make sure all the layers on the tracks play well together.  To accomplish this, it is wise not to have too much difference in dynamic range between the different layers.  For example, you can use different instruments for the melody or bass, but make sure they are all played around the same volume.  Also, within the track, you would not want wild variations in the timbre or note ranges of the instruments.  Try to keep them in similar ranges you would for vocalists of similar range: bass, tenor, soprano, alto etc.

Objective mode is great for music that needs to go from one activity or emotion to another in the game. In our example, the chase theme and battle theme are similar, but vary in intensity and in emotion characterized by the music.  When the character is moving from one scene to the next, or one type of activity to the next is when it is best to use the objective mode.  You have a lot more control over the type of music that will be played when, and Elias becomes a kind of conductor or arranger who follows the player around making sure the right music is played at the right time, just as you would hear in a film.

Here is a video showing Exploration mode for the Blade Revisited "Exploration" theme and comparing it to the same theme in Objective mode:

Reverb and Reverb Tails

The brain’s ability to detect reverb is an amazing thing.  We evolved this ability to be able to tell the size of the space that we were in, and what objects are nearby by us.  Dolphins ability to ecolocate is in some sense just a reverb superpower.

For example, if I were to play the sound of a snare drum in a closet and then again in a large cave, our ears would clearly perceive the difference. The brain would be able to detect the size of the room it was played in with amazing accuracy.  Sound designers know this and most 3D games take reverb very seriously to ensure realism, and there are even some cool plugins developed by Impulsonic that do very accurate realtime modeling of rooms and spaces in games.

I find it useful to visualize these concepts, especially when writing about them, so here is an example of a simple snare drum hit without reverb (top) and with a long reverb tail (bottom.)

Composers who ignore reverb do so at their own peril.  Well, maybe not peril, but it is very important to understand and employ it correctly.  

In games that do not employ audio middleware, composers have to pay attention to reverb tails and handle them manually.  We have to copy the reverb tail from the end of the loop and mix it into the beginning of the loop.  This is a lot of extra work, and it is not a perfect solution since it can sound very strange the first time the loop is played to have some unknown reverb source in the mix (a fade in can help here.)

Elias handles this tricky situation beautifully by including a reverb feature that is applied to the audio at the end of the effects chain.  What this means is that when the song loops back to the beginning, or even when it changes from one layer to the next, the reverb tail is added back in by the software engine, and you don’t have to do anything top the source files!  Let’s check out an example of how that works in Elias in this next video:

Note that when you start a new theme, the default settings in the theme settings dialog have the reverb set to ON, as seen here:

 If the mix is set to 0, the reverb is completely disabled irrespective of what the other settings are.This is probably a good idea IF you plan to set the reverb track by track, as shown here:

Note however, that the more tracks that have different reverb applied will have an impact on the CPU usage during playback.  To help overcome this limitation, Elias automatically employs submixes for reverbs that have the same settings.  For example, if you have 6 tracks in your theme, but only two different reverb settings, then only 2 reverbs will be used for all the tracks.

Think of the mix as the percentage of signal that is made up of reverb (If you had it set to 100 it would be almost all reverb) and the size as being the size of the room.  As for optimal settings for reverb, in my experience with Elias, a pretty natural sounding room reverb is Mix = 20, Size = 20 and Damp = 35.  If you were to set size to 10, mix  to 10 and damp to 60, it would be like playing in a small carpeted room. If you set size to 100, mix to 30 and damp to 60 it makes a very convincing cave reverb.  I give several examples in the video that you can listen to, and I encourage you to experiment on your own with simple sounds, like a snare, to see how the parameters affect the output.

In general it is best to include dry or fairly dry stems when importing into Elias and let Elias handle the reverb for you. Also, I have found that the music tends to sound most natural when there is at least some reverb on the mix. Finally,  try not to have too many different reverbs in the theme - it is more optimal to use no more than two or three different reverb settings- say one for the drums, one for choirs, and one of the other instruments.

Fades and Crossfades

In general fades should be so clean that they are not noticed at all.  If a fade is too obvious, it really can destroy the immersiveness of the music and our sensitive hearing can easily pick up a bad fade.

In Elias, you could  just use the default settings and possibly never have any issues with fades, but as you will see there are some types of instruments you will definitely want to tweak the default settings to get the best results.

Another area of Elias that can be controlled with a great deal of specificity is fades. You can set ranges for fade ins, fade outs and crossfades.  

To see how all this works, check out this short video here:

Here is the default setting in the global fade settings, in milliseconds:

The only time you will hear the fade ins is when a track starts playing, or when the track goes from a layer with silence to a layer with music. To see this visually, check out what happens to a kick drum when there is 0 ms fade in (top) and a 100 ms fade in (bottom.)  Therefore, on tracks that have a quick attack, like drums, pizzicato strings or mallets, you would be tempted to turn the fade to 0, however, in practice this is not a good idea.  It is better to set the fade between 5 and 10 ms for these cases in order to avoid any possible pops or dc offsets from happening during a layer change.

Conversely the only time you will hear the fade out is when the music on a layer goes to a layer with silence on it. To see this, check out the snare drum with reverb on it - the top track has no fade out and is not nearly as smooth as the bottom track which has a 500 ms (or half second) fade out.

Crossfades, however, are happening all the time and deserve some looking into. Different types of instruments require different crossfade settings.  Let’s look at this example of the kick drum crossfading with a snare hit. The first two beats are the kick, and the second two are the snare.  There's reverb on the track, so you can also see the reverb tails here. In the top track, there is no crossfade and the snare plays without any of the kick’s reverb tail.  In the bottom track, the crossfade is set to 500 ms and you can see how the reverb tail of the kick drum colors the sound of the snare in the third beat, and is mostly gone by the second beat.  This is a more natural effect, and in general you will want to use crossfades.

What are the minimum and maximum settings you see in the dialogs?  I will go into those more when I discuss Urgency later on, but for now you can think of it like this:  when Urgency is set to 100, or most urgent, then the minimum fade setting is used.  When the Urgency is set to minimum, or 0, then the maximum fade setting will most likely be applied.  Why do I say most likely?  That is because the engine uses heuristics to determine the best settings for each type of track depending on the tracks amplitude and transients.

Now that you understand the three types of fades and how they are applied to the music, what are some optimal settings?  Here you can see the default settings that Elias will use if you do not set them yourself.

For the fade in, you will want to be sure to set both min and max to 0 for kick drums, and anything that has a sharp attack, as mentioned above. For legato instruments, choirs, and pads you will want to set the Min to at least 50.  A 50 ms fade is not perceptible in a large mix, and therefore is a very safe setting, but sometimes it is not enough, especially for music that is not tightly quantized.  For this, you may want to set the Min between 100 and 250.  If it is too long, say 500 ms, then you will definitely notice it and probably should be avoided unless you want to apply that effect to the music.

For crossfades, the default settings are pretty good for most cases, but you can experiment with much quicker crossfades. You can add  longer crossfades on slow legato passages or even on tracks that have a lot of dynamic range between each of the layers.  For example, if you have some soft strings on one layer followed by loud horns on the next layer, you may want a longer and smoother transition there to keep from startling the listener.  Stingers, which will be discussed in more detail below, can help to smooth these kinds of transitions as well, but are not required when you have the ability to fine tune fades so carefully.

For fade outs, you see they are defaulted to 500 ms.  This half second fade usually works quite well and makes the transitions to silence (in that track) very smooth.  The only time you would want to set it to 0 would be if you absolutely don’t want to hear any of that track when the next layer starts, but as I mentioned above, it would be better to set it to 5 or 10 ms in these cases to avoid pops.  Longer fade outs can be useful in cases where you would want to have a lead instrument ring out into the start of the next layer, especially if it has reverb on it.

Agility and Custom Agility Beat Points

Agility is another key concept in Elias.  It is what sets it apart from other middleware solutions I have used for music, and it requires some playing with it to understand how it works.  Basically, it is the setting which tells Elias when it is OK to switch from one layer to the next. Usually, you want this to happen in time with the music, and very often you want that to happen on the first beat of a new measure.  However, with Elias, you can set that to any beat of any measure you want!  Moreover, by adjusting additional settings, you can give Elias more or less control over the times at which it will switch.

Let’s say, for example, you have a 4 bar melody line, but the harmony (chords) are changing every two beats.  You can set the melody track to only change every 4 bars, but tell the chords it is ok to change every 2 beats!  If you have the melody being played by a violin on one layer and a flute on another, you would not want the melody line to change in the middle of an important phrase.  However, it would be great to have it change on the next phrase and even be able to do a call and response style interchange between several different instruments, all while the harmonies and rhythms can be as agile as they like underneath.

There are two places where agility can be set, in the Theme Settings dialog, which will be the global settings for all tracks:

and the Settings dialog for each individual track:

In the image above, on the left hand side you can see the default setting which is 1 bar.  This means that Elias will default to switching every bar on the first beat of that bar unless you override that setting.  On the left in the image above, you can see that Agility has been turned on, and a custom agility input box is shown by selecting “Custom Setting” from the drop down menu.  Custom beat points can be added manually by clicking the “Add” button, but can also be added by starting playback (F5) and then just tapping the spacebar when you want to set a beat point.  

This can be done in either the Theme Settings or the Track Settings, and here is an example of how it might look:

Check out the video to see this in action:

As you saw in the video, the notion of “strict” agility means that Elias will strictly adhere to the exact beat points you set in these dialogs.  This all obeys the logic you indicate. For example, if you click the “Strict Agility On?” radio button in the Theme Settings Dialog, then Elias will obey those settings.  If you then, on a track by track basis click the “Override Strict Agility in Theme Settings for this Track?” radio button, then, for this track, Elias will use the settings you made here.  If “Strict” is never set, then Elias will use its best judgement to adhere to the agility settings in those dialogs, but it may choose to switch at a musically better time to do so.  What is an example of this? Well, in the case of the kick drum with no reverb, for example, there is a silence before each beat. Elias can determine this and know that it is OK to switch at any time there is a silence before the beat. Elias apparently uses many other factors when determining when it is appropriate to switch, and that is part of its magic special sauce.  

Changing Keys and Rendezvous

It is very common in video game music to play the same themes in different keys.  Each key has its own feel to it, and it is one way game music can keep the player from getting bored with the music.  Elias has a built in feature that allows for key changes provided you have exported your song in different keys.  It does not do a warping or transposition of the music for you, but most modern DAWS have this capability.  Rather than relying on the DAW to transpose, you can also just record in MIDI and then bounce to different audio files in different keys.

Once you have the new set of stems in the new key, just change the key in the Track Settings dialog (as shown in the bottom image) and then load them into Elias.

Then, in the player (F5) you can switch between the two keys very easily by selecting the key you want.

Notice the check box above Key which says Rendezvous?  This is an important concept in Elias and it is directly applicable to key changes.  When Rendezvous is checked, all the tracks in the theme will switch at the same rendezvous point on a level change.  In the image below, I have just switched from Level 1 to Level 2 in the player, and the tracks on Level 2 are blinking.  They will all switch at exactly the same time when Rendezvous is checked.

The reason this is important is that when you are switching from one key to another, you want all the tracks to change at the same time, otherwise you would be getting some very dissonant sounding music!  Unless you are going for a Free Jazz style effect in your music, then this is very desirable. 

Now let’s take a look at this short video on Key Change and Rendezvous:

There is another place where you can set the Rendezvous which is the final tab in the Theme Settings dialog, as shown here:

In the example above, you can tell the theme exactly when it is OK for a rendezvous to take place.  Here, the music will only change on the first beat of bars 1,2,5, and 6, or the third beat of bars 3,4,7 and 8. Note this is NOT the same as Agility in that this will only be used if you have the rendezvous button clicked on the player, or if you are doing a key change.  One final note - do you see that Fade of 75 ms on the bottom of the Rendezvous dialog?  That is a global setting and it will apply to all tracks anytime a rendezvous occurs, even if you have set the fade out to 0 in the track...so beware of this.  

Stingers and Pickup Beats

One of the coolest things to add to video game music are called stingers.  You hear them all the time in game soundtracks, and they are used in genres like EDM and more and more in pop music as well.  Stingers can be of many types, but the most common are risers, cymbals, orchestra hits, drum hits or rolls (including timpani), and SFX.  These are often used to help hide or smooth transitions, and can also be used to signal a big change in the music.  Key changes are often accompanied by stingers as well.  Here are some examples:

Here are some hits:

And here are three different risers, which culminate in a hit at different number of bars (I used a great program called Rise and Hit by Native Instruments which you can use to create some of these with specific crescendo points:

In the second example, above, you can see that it would be important for Elias to know when to start playing the stinger so that the hit occurs at just the right moment.  Well, that is where they have implemented the concept of pickup beats, which is located on the stingers page under the Track Settings.

As you can see in the image above, the first track, Hits, has no pickup beats because the cymbal crash, for example, would play right when the transition happens.  For the 1 Bar Risers, where the Hit occurs after the first four beats, the PickupBeats radio button is clicked and the number of beats is set to 4, or 1 bar of the 4/4 song.  This works wonderfully, as you can hear in this video example:

Urgency

The final thing that ties together everything you have learned so far is the concept of Urgency. Unlike other settings, there is no global setting for this parameter.  It’s default value is 100.  Urgency takes two things into account, the fade settings and the agility settings.  As I described earlier in the description of fades, if Urgency is set to 100, then the shortest fade will be used. If it is set to 0, then the longest fade will be used.  But that is not all that Urgency does.  If you have it set high, to 100, then the engine will try to make the nearest possible Agility setting that it can.  So if you have Agility set to every beat for example, then Elias will try to make sure it switches on the nearest beat.  However, if you have the urgency set to 0, then Elias can be more picky about when it will switch.  It will eventually ALWAYS make the switch to a new level, basically as soon as it can find a good place to do so.

Here is how Urgency works in action:

Conclusion

As you can see from this article, the designers of Elias have put in a tremendous amount of thought and care into making sure that you have a great deal of control over the playback of the music.  As composers, it is important to understand all these capabilities so that when you are writing music for the game, you will know what the engine can do to make the most out of your hard work.  

Prior to Elias and other middleware, game composers had to settle for writing loops that got to be very monotonous for the players.  This caused ear fatigue to the point where many of us would just turn the music off rather than listen to another loop through of the same theme, no matter how good it was.  

Now, we can offer gamers a tremendous amount of variety and we can also score their actions just like they were in a film.  This does take extra time and planning on the part of the composers and programmers, but in the games I have worked on using this tool the benefits are clearly apparent.  In the next article, I will show you a real world example of how this works in the VR game Stampede being developed by Black Matter Labs.  It will show how Elias has been integrated into a project using FMOD and UNITY, and I look forward to seeing you back here then.