Taking live soundtrack improvisation to the next level

I've previously written about how Rene Derks and I did a live musical improvisation to someone playing Journey and Ori And The Blind Forest. This was a pretty unique experiment already, but recently we kicked it up a notch with an even bolder performance: instead of knowing what was coming, we let players choose whatever game they wanted to play and we had to improvise live music to that. Since this was an experiment not all of it worked well, but I think it was a success: a bunch of games turned out pretty awesome and we learned a lot. Today I'd like to explain how we approached each game and why some of them worked, and others turned out not to be a great fit for this format. I've included a couple of short videos from the performance, check 'm out!

The basic idea for these performances is that we mix live improvisation with gameplay. Instead of playing the original game's soundtrack we come up with new music on the spot and try to respond to whatever is happening in the game. The result is a truly adaptive soundtrack, by the magic of spontaneous improvisation. As always with improvisation, some of it sounds bad, a lot sounds okay, and some moments are pure magic that can only be experienced there and then.



For the previous performances we had chosen specific parts of the games Journey and Ori that would have a nice emotional arc to add music to, and we thought beforehand about the kinds of emotions and melodies we wanted to do for the various sections of those games. While that makes for better music, it's also limiting: we can't do a whole lot of games that way and experimentation on the spot is somewhat limited. So when we got a chance to try this again at Abunai in August 2017 we chose a different approach: we went in deliberately unprepared. We had a Switch and a laptop with some 40 Steam games installed and let players choose whatever they wanted to play. We improvised to whatever they did.
 

Beforehand we thought that if 25% of our performance was awesome, 25% was okay and 50% sucked, we would consider it a successful experiment. We may have done a little bit better than that, but not much: some games indeed didn't work, and for some games we only figured out what kind of musical approach worked after a few minutes of struggling.

Another change we made compared to the previous performance is that we played in a small hall in a corridor. During our previous performance at Animecon was had a big stage. That's great if a performance is a big success, but it creates a lot of distance to the audience and you need a lot of audience to make it work. At Animecon on average we had maybe 20 viewers, which made the hall look like a big empty failure. So at Abunai we requested something much simpler: a spot that people walk past on their way to other events at the con, and some couches. This reduces the distance to the audience and makes it easy for people to watch for a few minutes and then walk on. In total we reached way more people this way, and it was much more fun to do. There was even room for musically fooling around with the audience a bit.



The first player of the afternoon chose Limbo and this was also the most successful game for us. During the afternoon we could easily see whether we were doing well by checking whether passersby stayed to watch and listen. They did for Limbo.

The reason Limbo is so great for this kind of musical improvisation is that it has a slow pace with a strong emotional arc. First a player walks for something like 20 seconds, which is a nice duration to set a mood musically. Then he approaches a trap. We know the game so we can start increasing the tension by playing something eerie. Then the player interacts with the trap and might even die a couple of times, which are great opportunities for doing something special musically. In this case I played 2 high shrieking notes at every death, combined with a few hard hits on the djembe by Rene. By doing the same thing at every death the whole performance gets glued together.

Limbo is slow enough that Rene even saw room to play something similar to sound effects on his djembe. For example, he played taps that mimic footsteps, drum rolls during slides and a deep bass when the player lands after a slightly deeper drop.



The second game that was chosen was Ikaruga, an insane bullet hell shooter. This was very difficult for us to react to: the screen is filled with bullets practically all the time so there's isn't really any kind of arc that we could latch onto. We ended up just playing a fast beat and only really reacting during boss fights. It worked somewhat, but the whole idea of the music reacting to the game was kind of lost here and cello+djembe isn't the best possible combo for a fast beat.



The worst of the day and the only complete failure was Tekken 7. A player brought a laptop to show his skills at beating the final boss on the hardest difficulty. He was incredibly good at the game so that was impressive to see, but making music to this was practically impossible. The action was too fast to follow and the camera constantly jumped around to emphasise specific scenes. For a short bit we tried to play sound effects on the punches, but the game is so fast that we totally failed at seeing them coming.

Figuring out what to do with the next two games was really interesting. Spelunky is a game with a rather constant state during gameplay: every five seconds you kill an enemy, grab some loot and do some platforming. Individual events are too fast to react to and the flow of events is pretty much constant. However, Rene noticed that during each level the player is always going down. I tried following this progress by starting each cave on the highest string of my cello and then dropping to the next string as the player drops. While this idea is nice, I feel it was too subtle to be picked up by the audience. In hindsight it might have been better to react to how much health the player had left. Spelunky is a rogue-like game so the tension of being low on health is really high.



During The Binding Of Isaac we found a better trick. For each floor Rene chose a base rhythm that he expanded as the floor progressed, building towards the boss fight. During empty rooms he went back to the floor's basic rhythm. While Rene kept playing this beat on his djembe, I stopped playing altogether. Whenever the player cleared a room, I played a short melody; the same one for each room. I had another melody for opening a chest. And then as the player reaches the final boss of a floor, I started actually playing and driving up the tension. I think this worked really well and it shows a very different approach compared to what we did during the other games.

Next up was Broforce, in 3 player coop. Sjeez, that game is chaotic! Each level is just 40 seconds of constant gunfire and explosions. I had no idea where to look or what to respond to. This was almost as bad as Tekken 7. Ugh.

So I shivered when the next choice was Ultimate Chicken Horse: another super chaotic multiplayer game. However, here we found a fitting approach. A match of Ultimate Chicken Horse is a series of short rounds, where each round starts with everyone modifying the level a bit, and then everyone playing it. Each phase lasts around 15 seconds. Reacting musically to the actual events is impossible in this chaotic game, but instead we simply did something special for the phases. During the building phases I did some goofy plucking on my cello, deliberately playing melodies that sound a bit childish and dumb. And then during the action phase I played a faster, more aggressive variation with the bow. Rene made this contrast even stronger by not playing at all during the building phase and playing a really fast djembe rhythm during the action phase. I think this captured the back and forth of the gameplay phases quite well, and was a good fit for the wacky art of Ultimate Chicken Horse.



We ended with two slower single player games: Zelda and ABZÛ. The Legend of Zelda: Breath of the Wild is a great fit for improvising music. Most things the player does take long enough that you can react to them. Walk for half a minute, fight for half a minute, look at your loot for a bit, climb a hill, swim some water, another battle. Each of these steps takes long enough that you can really react to them without turning the music into a chaotic mess that goes all over the place.

ABZÛ is an even slower game. Like Journey it's a gorgeous game that's mostly about looking and experiencing, with very little challenge in the gameplay. The serene under water gameplay is a great fit for setting a mood through music, but I think our instruments weren't the best fit here. Djembe and cello are interesting for many things, but underwater moods are difficult with this setup. Also, after playing for around 2.5 hours the sound got a bit stale for us.
 

Nevertheless, during the final round of ABZÛ another interesting thing happened. The player had overlooked that he needed to free a little robot before a door could be opened. Instead, he repeatedly pressed a button on the door, hoping it would open. I played two simple notes each time to accentuate that it didn't work, and since the player kept trying, I kept playing those same two boring notes. This went on for way too long and the audience started laughing at how stupid that sounded. This was one of the nicest and most direct bits of audience interaction we had. It also taught me an important lesson: sometimes it's okay if it sounds bad musically if the interaction is fun or interesting in some other way.

Finally, after the performance we had dinner at Rene's parents. They happen to own a piano and Rene also plays that, so we started jamming some more, despite having played so long during the day already. This sounded quite wonderful and it taught me the biggest change we should do if we do this kind of performance again: we need a more diverse set of instruments. I can only play the cello, but since Rene plays both keyboards and percussion we should just bring both to diversify the sounds and moods we can play. Or maybe we should even get a third musician next time!
 

In the end I think this was a really interesting experiment. The quality of the music varied from really awesome to really bad and that's okay: the whole point of improvisation is that you don't know what's going to happen and the moments where it's awesome are true magic, happening right there and only then. The key lesson we learned is that different games require vastly different musical approaches and the trick is to find whatever arcs the game has going and respond to those through music.

For more blogposts on development of Awesomenauts, Swords & Soldiers, Cello Fortress and Proun check my dev blog at www.joostvandongen.com.