"Sound is Magic" - Insights for the Game Music Composer

From May 19th to the 20th of this year, the British Broadcasting Corporation's Research and Development department presented a two-day conference to explore the future of immersive sound.  Called "Sound: Now and Next," the event featured a distinguished speaker list that included accomplished audio engineers, producers, educators, inventors, researchers, musicians and composers.  The event offered a wealth of fascinating presentations on the future of audio, and I recommend visiting the site and checking out the awesome video resources from the event, which include complete session videos made freely available for streaming from the site.

For game composers and sound designers, one of the best sessions was presented by Nick Ryan, an award-winning audio engineer/composer/audio consultant who is best known in the game industry for his sound design work on the Papa Sangre, Papa Sangre II and The Nightjar audio games for iOS.  These three games utilize binaural sound to immerse players in an audio-only interactive environment, which Nick Ryan calls "inhabitable audio."

Nick's presentation at the "Sound: Now and Next" conference was entitled "Sound is Magic."  According to Nick, audio has a unique power to bring about an emotional and perceptual impact by virtue of the collaborative relationship between the sound source and the listener.  When a sound is separated from its original source (i.e. when it's not possible to see the source that's emitting the sound), listeners will instinctively use their imaginations to supply the nature of the sound's origin.  This imaginative contribution on the part of listeners has the potential to draw them more fully into the experience.  "I profoundly believe that we are co-authors in everything that we listen to," Nick tells us.

Nick Ryan, sound designer for Papa Sangre, Papa Sangre II and The Nightjar

Early in the presentation, Nick introduces us to his initial work in binaural / spatial audio by describing a project he produced in 2002 for BBC Radio 4. "The Dark House" was a popular interactive radio drama: a ghost story recorded on location in a large house.  The actors wore baseball caps with microphones embedded in the brims.  While the project was ostensibly a traditionally linear radio drama, it was structured so that the audience could decide from which character's perspective the story would be told, and the audio mix would switch to the perspective of the character who had received the most votes.  In this way, the audio mix of the program changed drastically as the audience cast their votes during the broadcast.  The entire program is available for listening here:

Nick stresses that this project illustrates the power of adding interactivity to an audio experience.

Moving on to his work in video game development, Nick launches into a discussion of his work on Papa Sangre, a game set in a completely "non-sighted" realm of the afterlife, inhabited by vicious unseen monsters.  Sharing a few observations about gamers' experiences in Papa Sangre, Nick points out that visually-impaired players would usually breeze through the game in an hour, whereas sighted players found it to be crushingly difficult.  Also, Nick describes a phenomenon whereby sounds associated with personal movement (such as footsteps) stimulated the motor cortex of the brain to be active, making listeners feel as though they were actually in motion. This motor cortex stimulation contributed to the immersive qualities of the gaming experience in Papa Sangre.  The effects of sound on the brain are tremendously fascinating, and I explored some of the effects of music on brain activity in my book, A Composer's Guide to Game Music - so I was especially interested to hear more about that phenomenon in Nick's talk.  To learn more about Nick's work on Papa Sangre (and another audio-only game titled The Nightjar), check out this sound design mini masterclass that Nick gave for the British Academy of Film and Television Arts:

Continuing with his presentation for the BBC "Sound: Now and Next" conference, Nick described a collaboration with Volkswagen and the famous electronica duo known as Underworld to allow a car to essentially drive a piece of music, associating an interactive musical composition with the turning, braking, acceleration and de-acceleration of the vehicle.  While it isn't a game-related project, it is fascinating when considered in terms of the interactive music possibilities that could be translated into gaming applications.  Here's the final video result of "Volkswagen Golf GTI Play the Road."

And here's a behind-the-scenes video that explores the making of this interactive music system for driving:

Finally, Nick brings the entire concept of "Sound is Magic" to a culmination by describing his collaboration with John Matthias to create a four movement piece for string orchestra entitled "Cortical Songs."  A computer simulates the way in which human neurons naturally behave, sending these signals to tiny flashing lights on the music stands of the string players.  The musicians respond to these flashes as they would respond to a conductor issuing cues - as though the simulated neural activity was leading the orchestra.  The magic of the human mind is now expressed through sound, expressing Nick's concept of Sonification -- the aural expression of silent phenomena.  Here is an excerpt from a performance of the composition:

The "Sound: Now and Next" conference offered an abundance of inspiring ideas from top practitioners in their fields, and I urge everyone to check out the site and see some of the other presentations that are available online.  Also, be sure to check out the complete presentation given by Nick Ryan -- "Sound is Magic."

Winifred Phillips is an award-winning video game music composer whose most recent project is the triple-A first person shooter Homefront: The Revolution.  Her other credits include five of the most famous and popular franchises in video gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims.  She is the author of the award-winning bestseller A COMPOSER'S GUIDE TO GAME MUSIC, published by the Massachusetts Institute of Technology Press.  As a VR game music expert, she writes frequently on the future of music in virtual reality video games.

Follow her on Twitter @winphillips.