What we learned improvising a live game soundtrack

Recently I've done a couple of music performances that were quite different from any live game music performance I've ever seen. We let someone play the games Journey and Ori And The Blind Forest live on stage and we improvised a completely new soundtrack on the spot, reacting to whatever the player was doing. This was a unique experiment for us and we learned a lot about what works and what doesn't, so today I'd like to share our approach and experiences.



Usually when game music is performed live the original soundtrack is replicated by musicians on stage. We however completely ignored whatever the original soundtrack had sounded like and improvised based on what we saw. The resulting music is different with each performance and sounds nothing like the original soundtrack. The fun of improvisation is that it's entirely in the moment. You don't know what's going to happen and it will never happen in the exact same way again. A truly live experience! Sometimes we might make mistakes, but sometimes we might also improvise the most intensely awesome music. Doing this in response to a game that's being played live is really exciting! Here's a video compilation of our performance at Animecon in the Hague last month:

Some people might know my game Cello Fortress. While there are similarities between Cello Fortress and our live game soundtracks, they really are very different. In Cello Fortress I control the game by playing cello. The cello is a game controller and the game is designed around this unique input device. On the other hand, the live game soundtracks in this blogpost are for existing games: the music doesn't control the game at all. We just respond musically to whatever happens on the screen. This is a bit less extreme than Cello Fortress, but as far as I know it also hadn't been done before and it's a different, interesting approach.

Before I continue, a short word about who 'we' are. The original idea for this performance came from Bram Michielsen and Zuraida Buter, who asked us to improvise live music to Journey at the Screenshake festival in Antwerp in 2015. After that we did a couple more performances and added Ori And The Blind Forest to the mix. I did these performances together with Rene Derks, who played djembe, keyboards and pads. Rene is a lecturer in game design and production at the NHTV in Breda and previously worked on Loco Roco 2 and Gatling Gears. I myself played cello and in my daily life I'm lead programmer at Ronimo Games, mostly known for our games Awesomenauts, Swords & Soldiers and the original version of the student game De Blob. We're not professional musicians but music is a rather serious hobby for both of us.

Improvising interesting music is difficult, and doing so together with other musicians even more so: whatever you're playing needs to not just sound good on its own, but also sound good with what the other musicians are playing. This means you can't just suddenly change the rhythm or mode unless you have a lot of experience improvising together. The specific combination of djembe and cello helps a lot here. It's only two instruments and they both carry a very clear responsibility: the djembe is in charge of rhythm and the cello is in charge of melody.

That gives me total freedom to change the mode or chord without needing to worry that this will sound bad with the other instrument. This kind of freedom is extra important here because we need to respond not just to each other and the audience, but also to the game. It also leaves room for great accidentals: a fast djembe rhythm can work great with slow cello notes and vice versa, so as long as we change intensity at the same moment it's okay if we don't always agree on whether we're increasing or decreasing the intensity.

When preparing for our first gig we started by choosing specific parts of each game, looking for areas with enough of an arc to make them interesting musically. The part of Journey we chose starts relaxed in the desert, then goes down into a darker valley with machinery, then has a very fast descent and finally goes through a dark, scary area that ends in a climax as the player flees from flying monsters. This gives us plenty of room for variation and for having an emotional arc in the music.



The next step was to figure out the general mood we wanted in each area and to come up with some ideas for the kinds of things we could do there musically. We looked up some longplay videos of Journey and improvised over those. We didn't actually write any music but we did end up with a good idea of what kind of things would work were. We also decided where Rene would play djembe and where he would use atmospheric synthesizer pads. In some cases we defined rather specific musical ideas, but in most it was just general directions. For example, in the haunted caves in Journey we didn't have more than this idea: "the caves get low dark dissonant cello and the djembe keeps silent in the beginning and joins after a while."



We wanted to make it very clear to the audience that we were truly responding to the game and not just playing pre-composed music, so we looked for some player actions we could respond to. Journey turns out to be great for this: jumps can take as long as 10 seconds so they are long enough that we can respond to them musically. During a big jump I might for example play an upward flurry on cello, hold a long sliding note during the fall and then get back into the rhythm as the player lands on the ground. Musically it feels slightly childish to play an upwards line as the player jumps upward, but audiences responded very well to this. We could clearly hear in their responses that these were the moments where they truly 'got' it.

To tie it all together musically and to have some kind of theme that the audience can recognise and remember we choose a couple of events for which I wrote small melodies. This is very much like in Zelda, where a short melody plays whenever you open a chest. In Journey we chose the finishing of each level as the key moments and in Ori And The Blind Forest the opening of doors. I quite liked the melody I came up with for Journey so I've also reused it as an attack melody in Cello Fortress.
 

This kind of performance isn't just about the music, but also about the person who sits on stage playing the game. Figuring out what's best here was a challenge. On the one hand the most pure expression of the concept would be to have a random player from the audience play the game without any preparation. However, this is risky since the player might not know the game and get stuck or need to retry an area a dozen times. That's going to be really boring to watch. On the other hand a prepped player feels like faking it. So we ended up making sure we had a player who knew the basics of the game, but who hadn't played with us before so that we both didn't know what was going to happen exactly.

This worked well in general but during one performance it backfired. The player got stuck in a dark cave in Journey and walked around aimlessly for minutes. It's difficult to improvise anything good for that and I saw some people from the audience leave during that section. I think next time it might be better to look for a player who knows the game better and accept that the player might rush through some sections. That's probably more fun than watching someone be stuck, trying to figure out what the jump button is or looking in every nook and cranny to find every last item...



While Journey is a slow game in which we could respond to individual actions by the player, Ori And The Blind Forest is way too fast for this kind of approach. For example, combat often takes only 2 seconds. We tried switching to combat music every time an enemy was encountered, but the music alternating so much just didn't sound good. Music needs to flow. So instead we decided to respond to individual events very little and mostly just have different types of music per area. The player can quickly go from place to place but generally stays in an area for at least 30 seconds so it's okay if the music changes whenever you go in or out of an area.



In Ori We also took a more melodic and less improvisational approach: Rene played keyboards here and we defined most of the chords and arpeggios beforehand. This works, but it felt a bit too forced to our liking. Our goal with this project was to truly improvise and after doing a couple of performances especially Ori felt a bit stale.

Doing these performances was really exciting and we've learned a lot about what works and what doesn't. Our next experiment with this idea is going to have a different structure. At the Abunai convention we want to try a smaller, more cosy setting where we don't decide beforehand which games are going to be played. Instead we're just going to bring a list of suitable games and let the audience decide whatever they want to play and we'll improvise to that. If someone happens to be carrying their own Switch they could even hook that up and play their own games! This means we can't prepare beforehand, which will probably result in less polished music, but at the same time it will be much more spontaneous so I expect more fun, weird things will happen. I also hope that not being on a stage will make the interaction between the audience, the players and us more direct. I'm really looking forward to trying out this new experience!

For more blogposts on development of Awesomenauts, Swords & Soldiers, Cello Fortress and Proun please check my dev blog at www.joostvandongen.com.