Composing Adaptive Music for Games with Elias – Part 3 – VR Game Stampede and Unity Integration

The history of the music industry is inevitably also the story of the development of technology.

- Edgar Bronfman, Jr.

Any sufficiently advanced technology is indistinguishable from magic.

- Arthur C. Clarke

Introduction

Welcome to the third article in this series - “Composing Adaptive Music with Elias.”  If you missed the first two articles, they are available here:

Part 1 - Composing Adaptive Music for Video Games with Elias

Part 2 - Composing Adaptive Music for Video Games with Elias - Advanced Topics

In the first article, I gave an overview of the Elias studio and introduced the importance of using adaptive music in video games. In the second article, I went into detail about the specifics of Elias and some of its advanced features, and how to get the most out of it when composing adaptive music. In this third article, I am going to show you a real world example of how adaptive music is used in a game, how simple it is for programmers to get Elias up and running in Unity, and discuss the performance and memory usage of Elais, which turns out to have negligible impact on the CPU or audio memory (less than .2% CPU impact and only .2 MB of audio memory no matter how large the theme is.)

The game I am going to show you is called Stampede, a super fun virtual reality VR game being developed by Collin Parker of Black Matter Labs. Collin is an indie game developer based in Los Angeles, and  Stampede is a game being built for the upcoming launch of the Oculus Rift and other VR headsets.  A demo level is already available for you to check out for free if you have an Oculus headset here. Those who have played the game are having a blast with it, as you will see in some of the live gameplay videos.  Stampede is being developed in Unity, and we are using FMOD for the SFX, Ambients and VO and Elias for the adaptive music in the game.  FMOD and Elias play very nicely together as you will see in the implementation and performance video below.

Elias has just announced that the Elias Engine is FREE for development budgets under 150K USD, and so there is no reason not to download it and try it out in your game.  I know my experience with it has been incredibly positive, and you will be glad that you did!

Black Matter Labs

I spoke with Collin, founder of Black Matter Labs, about the game development, how the music was implemented,  and about his experience in working with Elias in this video interview here:

Collin and I met at Indiecade in Los Angeles last year, and have been working together on this game since then.  He has been a pioneer in trying out new technologies, and has been an avid supporter of the Oculus and other VR headsets such as the HTC Vive and the Sony Morpheus.

Stampede

Stampede is a VR tower defence game inspired by the movie Jumanji in which crazed animals are attacking your village. The game is set in Africa, so the animals are all pretty fierce when they get riled up - you will be battling elephants, hippos, zebra and hawks among other animals on the first Savannah level.  Other levels include a wide variety of animals from land, air, trees and the sea as you lead up to the final boss level, which will be fought in a volcano.

Since this game involves waves of big game animals, I asked Collin if he has anything against elephants, zebras and hippos.  He assured me that he is an avowed animal lover, and is not promoting hunting or killing of any animals in any way. In fact, he created this fictional world where the animals have become crazed by some unknown source, which you will have to discover and remove as part of the game’s backstory.

There are tons of great reviews of the demo level of the game, and I put together a video with excerpts from four of them to give you an idea of the gameplay and how much fun people are having with it:

Here is the official trailer:

Music in Stampede

When I first met Collin at Indiecade LA last year, I got to sit down and play an early version of the game. I was all smiles and sweating bullets by the end of my first attempt (it was hot that day too...)  It was one of my favorite games from the show, and I was thrilled that he approached me soon after to work on the SFX, VO and Music for the game. I had just discovered Elias, and I sent him an early version of the music in Elias, which he loved.  However, the Unity plugin had not been developed at that time, so we decided to proceed implementing the music in FMOD for the time being since we were using it for the SFX and VO already.  As soon as the Elias Unity plugin was available, we switched back to Elias and now are using it for all the music in the game.

This gave me a unique perspective to compare the difference between my FMOD implementation of the music and the Elias implementation.

Walkthrough of Elias Theme for Stampede:

FMOD, WWISE, Fabric and Elias

Elias works perfectly well with FMOD in Unity, as it would with any other audio middleware such as WWISE or Fabric, and they actually have different purposes.  The other popular audio middleware products were developed first as sound effect playback engines, and now also support music, but Elias was developed by composers for composers and is a sophisticated music only solution. Though the others can be used to play back music adaptively, they do not have any built in templates. Therefore, you need to learn how to use the tool, then learn how to create adaptive music solutions from scratch, and then implement it yourself.

These other middleware solutions are  best at playing back music horizontally, meaning that you can switch from one loop of music to the next depending on the requirements of the game, and you can fairly easily add things like stingers to smooth the transitions.  However, if you want to attempt to do something as sophisticated as Elias can do out of the box, you are going to run into some severe limitations.  These same limitations would apply equally to WWISE, Fabric or any other middleware that is designed primarily for sound effects.  Elias uses the vertical resequencing method for playing back adaptive music, and as you will see, this is by far the smoothest and most advanced way to implement adaptive music in games.

Another huge benefit is the incredible variety that the vertical resequencing method gives over the more simple horizontal method.  In Elias, an exploration theme with 6 tracks and 6 variations can result in 6 to the power of 6 variations, or 46,656 variations!  In the horizontal approach, if you have 6 tracks - you have exactly 6 variations and this can lead to boring repetitive music and ear fatigue that usually results in the player just turning off the music.

For example, my early FMOD music player implementation looked like this:

It was basically a horizontal approach which simply played pre-rendered musical scenes depending on the wave of animals in the Savanna level. There are 10 waves in the Savanna level, and there were 6 “levels” of pre-rendered music, so Collin distributed these more or less evenly across the different waves, with the most intense music playing in a loop at the most difficult final 2 waves. This simple implementation took about a week to set up in FMOD. There are other ways to implement adaptive music in FMOD and other middleware, but to do what Elias can do out of the box would take me weeks or months to set up a similarly complex music player, and it probably would not have all the features explained in my second article.  If you are concerned with performance, as you should be, then as you will see with the performance section it turns out that Elias can offer huge benefits here as well.

Finally, if you want to use other middleware in conjunction with Elias and still be able to control the volume of the music playback, then it is a simple matter just to run the audio stream through that middleware so you can do things like sidechains for dialog or sound effects.

Elias Theme for Stampede

The theme for the Savanah level has 15 levels, and can be heard here:

And here is what it looks like in Elias:

There are also a bunch of stingers, as you can see here:

Implementation

Integrating Elias into Unity is very simple, and according to Collin, once you know where the music theme should go (the streaming assets folder by default), it only takes a matter of minutes.  However, it does take the usual amount of time time to set up the game logic to playback the music in the way you want it to react in the game.  For example, in Stampede, the first 3 levels are idle music which plays before and after a wave of animals start “coming to destroy the village!”  After that, the levels gradually increase and decrease depending on the number of animals attacking or remaining.  For example, in the early waves, the theme may reach up to level 5 or 6 of intensity and then slowly go back to level 1.  However, on the later waves of the game when animals are coming at you from every angle, even the air, the intensity can reach the top level before heading back down to level 1 as you defeat more and more of them.

Though this is not intended to be a complete tutorial on the process of implementing Elias in Unity, I did try it myself and found it to be incredibly easy.  Just download the Elias Unity Plugin here, and then double click on it (you have to have Unity installed on your computer, and it works with version 4.6 and above.)  Once you have done that, Elias is installed!  They also included a demo theme that is automatically included in the proper directory, the /Assets/StreamingAssets folder as shown here:

You can put your own themes in the same directory following the instructions in the ReadMe, which state: “The folder structure for a theme called 'demo' would thus be:

Assets/StreamingAssets/ELIAS_Themes/demo, where the directory demo contains demo.epro and /audio“

There are also a number of exquisitely composed themes available by searching for Elias on the Unity Asset Store, such as this gem that was first demoed at GDC 2015:

Elias Unity Integration and Performance

Here is a video where I interview Collin about the adaptive music engine, integration of Elias into Unity (at 4:15), the performance of Elias in Unity (7:20), and the target platforms (10:00):  

Performance and Memory Usage

There are 8 music tracks and 5 stinger tracks.  There are 52 separate “stems” for the music, and 25 stingers, and the total file size for this theme is 43 MB in OGG files.  However, when the music is played back in Unity, as you can see in the Elias Unity Integration and Performance video, only .2 MB of audio memory is used at any time!  In addition, only .2-.3% additional CPU is used - so the impact on the overall performance is negligible. This is because Elias uses a configurable streaming buffer which defaults to 400 ms.  That means only a small amount of the entire theme is present in memory at any given time!

Elias Demo Player

I have recorded a video showing how to put your own theme into the Unity Demo player along with screen capture of the Demo Player working.  I go into where to find the very well written documentation and other Elias assets in some detail here:

Conclusion

My thanks to Collin and to the amazing programmers and composers at Elias Software for letting me pick their brains in order to write this in depth series of articles.  It has been a long journey, and if you have read all three articles in full, pat yourself on the back!  You are on your way to having some of the best sounding adaptive music in your game that is available, and you won’t be sorry you tried it out!

Related Links

Black Matter Labs
Elias Software
Gryphondale Studios

Winifred Philips Blog on Elias

Special Thanks to these Youtube, Oculus Rift Reviewer's Full Videos

Bored Gamer
UK Rifter
MrKingOfMetal
Panderath