Subversive Sound

Watch this short scene from "Stranger than Fiction":

Describe to yourself what you saw.

I've spent most of my short career creating and implementing sound effects for games (Antichamber, Capsule, Orcs Must Die 2}.  Humans are visual thinkers, and to at least some degree we critically analyze what we see. What I find so exciting about sound design is that nobody's thinking about it - they don't even realize it's there. That means sound designers can inform the player's mood and thoughts in a particularly subversive way. The clip above is my favorite example of an established subversive technique called "synchresis." This is a trick used by sound designers all the time when they want to lend the associations of one object to another. The construction truck in the clip above reminded you of a T-Rex because it was layered with the sound of one¹.

These perceptual tricks get me really excited. Synchresis is a particularly well studied method (the term was invented by Chion for his wonderful book Audio-Vision), but there are lots of others and we're inventing new ones all the time. In the interactive arts, we can go even further and deeper in how we trick your subconscious!

You can forget I told you anything about synchresis now if you like. It'll work out better for both of us this way...

SoundSelf's non-representational audio is a playground for me as a sound designer to experiment with using novel techniques to get under the player's skin. Some of these techniques are on the scientific fringe - areas of research that haven't yet been rigorously tested, but have either a rich tradition in ritual or show anecdotal promise for healing.

Unfortunately it's just not realistic to tape a rigorous scientific study on top of our game design process. That said, by maintaining an open minded and a casually empirical approach to development we're finding a cocktail of techniques that draw the player into a trance state for SoundSelf. Right now we're experimenting with a combination of binaural beats (playing slightly different tones in your left and right ear which produce a 5hz - 15hz beat when summed by your brain) and shamanic drum rhythms to help guide the player into and out of the experience. Whether and how well these work remains to be seen, but our early tests are very encouraging.

There's one technique though that's clearly very potent for intimately connecting the player with the experience: SoundSelf's music is made up of long drones whose pitch is determined by the player's own voice. By matching the player's pitch with a sound that is at least not drastically different from a human voice, it feels to the player like the game's audio is emerging from their body. This allows us to do some really magical things! By associating timbre change (say, from a cello to a synth) with a visual change, the player feels like their own voice and body is morphing in step with the game. It's quite trippy.

Try chanting along to the video below, or trying out the Kickstarter build of SoundSelf to see for yourself.

¹ We all know what a T-Rex sounds like because Gary Rydstrom made it up for Jurassic Park.