Functions of Sound in Games

One of the wonderful things about sound is that it can accomplish many different things. Additionally, the same sound used in one context can have a completely different meaning in another. This is true from an emotional, informative and clarity standpoint. In my game audio classes at DePaul University, I always point out moments when we hear the same thing in games but have a very different response or reaction to them. 

Sound is powerful – and if you work in games, you should think about its capabilities for your project or work with someone who understands what effects it can have and all the ways it can be used. A single sound can be doing many different things at once!

So with that in mind, here are eight ways sound can be used in games:

Contextual/Narrative Sound 

This is probably the most straightforward entry on this list. When an action happens such as a character moving, using an ability, or the player selecting something in the UI, we need to hear something that seems “appropriate” concurrently. 

If we don’t hear something when expected, it can be one of the most immediate ways to lose that sense of suspending disbelief or “buying” into the experience. These sounds need to be present but also need to be choreographed to the visual gesture. Starting or stopping “out of sync” is just as much of a glaring error as not having the sound at all.

Pretty much every game is chocked full of sound filling this role, but for an even more visceral example, check out the game, Perception.

The premise of Perception is everything we see is based on sound reflected from the world. Think of it as similar to echolocation. If something isn’t generating sound on its own, the only way we see it is if sound travels out and bounces off surfaces in the environment, and returns to the listener. 

Everything we see is based off of sound, so if we see anything, it is because that action/object/event has an intrinsic sound with it. For some, seeing all the sounds that populate our game worlds can help make it clear how vital the sounds are.

Focuses Attention 

A very powerful intent from a design perspective is what our player is focusing on. Are they marveling at the art or environment of a new area in an RPG with a massive world? Will they be able to make the jump from one level to another in a platformer? Do they need to be ready to dodge an enemy attack?

Most times, the auditory and visual cues work in conjunction with one another. This makes it very persuasive in telling the player that something is important and deserves their attention. However, having separate visual and auditory cues can be very powerful and have incredible effects on the player. Look at this sequence from Amnesia: A sense of danger is communicated through an invisible monster splashing through the water chasing you as you jump from box to box.

Can you imagine how boring hopping between boxes would be without hearing the splashing footsteps coming after you? Are the boxes the real focus this whole time? No, not at all!

That constant auditory reminder of impending doom is so strong! So strong that the player doesn’t need to see the footsteps of the monster in the water to be utterly terrified of it.

Defines Space

We are used to different spaces sounding differently. If you yell in a small room, it sounds very different than yelling in an empty sports arena. Not only does it take longer for sound to reach a listener’s ear in a larger space, but when we are in large spaces, most of what we hear is reflected sound as opposed to direct sound. 

The sound of our voice goes out in every direction, with very little of it going directly to a listener’s ear when we’re in a large space like an arena. A listener may still hear this sound even if it doesn’t travel directly to their ears, but not after it’s bounced off a number of surfaces. This is what’s called reflected sound, and it’s most of what we’ll hear in a large space.

In a small space, our listener will be closer to us. This means more of our voice will go directly to their ear, and the reflected sound will take less time to reach their ear. This gives a very different character to everything we hear in a small space as opposed to a large space.

Additionally, the materials present in these spaces play a huge part in what sounds we hear. We hear certain types of sounds more when hard flat surfaces are present as opposed to curved cloth couches.

If we don’t acknowledge and emulate these sound characteristics, our game worlds will never feel real.

Game audio folks spend a lot of time ensuring game worlds feel real. Here is a portion of the implementation used in Hitman 2 to ensure this happened:

Creates Atmosphere/Mood 

This is referencing the emotion a player feels while experiencing your game. While the previous point was pertaining to making a space feel right in terms of physics, this point is in terms of emotion.

A large space can be awe-inspiring, majestic, threatening, magical, exciting, intense, etc. – the list could go on forever! Every sound and note we hear has the ability to give an emotional impression, but only if we want it to - and know how to execute it well.

Watch the opening to Bioshock on mute. It looks creepy, but this experience lacks any sort of visceral emotional response.

Now, play the opening with the sound on and close your eyes. Pay attention to how much emotion you feel with no visual component!

Of course, the real impact is achieved when we have both the visual and working together, but pay attention to where the emotional part of the experience is coming from.

Emphasizes/Intensifies Action 

My favorite example of this is Doom as well as any Tarantino movie. More than having a sound be present for a gesture, audio can add a layer of intensity that isn’t naturally there. Any team creating an experience that is highly “stylized” will have given significant thought to how the audio contributes to the overall aesthetic because it is that important!

Case in point – if it hadn’t been incredibly thoughtful about what the nature of Doom was, the end experience could have missed the whole point of their world! Don’t believe me? Listen to this:

A far cry from the intended experience of Doom, right? Heres the correct audio.

This effect can be true for entire soundscapes like in Doom, but it can also be true for single ability or action sounds. There are certain sounds that just aren’t appropriate for a healing spell, right? How can we communicate something positive, negative, disorienting, dangerous, all in a single sound?

Here are a number of ability sounds I created for the game Card Chronicles: Sentinels that all needed to communicate the nature of the ability being used by the quality of its corresponding sound. What makes the healing spell sound feel appropriate for a healing spell?

Promotes Immersion (VR) 

This is very different than the contextual/narrative and space-defining audio we looked already. While you could describe both these capabilities of audio as making something in your game feel “believable,” immersion is the sense of the player actually being in that space.

Immersion is getting someone to lose the sense of their physical self and feel like they are actually in another space or occupying a body other than their own. Your attention shifts from controlling something in a digital world using your physical body to feeling as if you are actually occupying the digital world.

That is a huge jump. 

Advancements and accessibility of technology have made techniques such as spatial and ambisonic audio integral to the VR experience. More than hearing something to your left or right, we can accurately simulate that qualities of sound emulating from different points in 3D space in different sized spaces, with different materials, and when you’re looking in one direction versus another.

But the ultimate audio sensation of immersion is through binaural audio experiences. Not only do we hear the qualities of sound being affected by different spaces, but we can experience how the human ear perceives audio in that space as well. While ambisonic and spatial audio are incredible experiences, they are ultimately taking “believability” to a higher level. Ambisonics audio, in particular, has many applications to VR since it is a format of an entire sphere of sound around a point in physical space, and can be converted to playback in headphones.

However, for the most immersive audio experience, nothing beats binaural audio. This audio format makes you feel like you are actually there. If you want to better understand what this difference in experience is, grab some headphones and listen to this:

Pretty amazing, isn’t it?

Now the drawback with binaural audio is that it is generated relative to where a microphone or listener is. Since we usually move in our game environments, it’s not easy to replicate accurately, and it can cause motion sickness if not used correctly. That’s how powerful this stuff is!

Sets Pace as Gameplay Function

The most common example of this is used in rhythm games. If you’ve ever seen a serious Dance Dance Revolution tournament or players, you know how fast and intense these games can be! A big reason the player is able to quickly and accurately time their feet to the visual cues is because of the music giving them a constant frame of reference.

A different example of this is any sort of timer that has an associated audio cue. Many games have a timer that is ever-present, but when we get to our last 10 seconds or so, the timer’s audio either becomes audible or is louder in the mix. It definitely gets the point across that you need to complete an action/puzzle/objective sooner rather than later!

Smooths Transitions

There are a couple of different flavors to this one and many more than what I can speak about depending on the genre and mechanics of your game. But, the two that I can touch on with good certainty that they’ll be relevant to you are: transitioning between story/cinematic and gameplay as well as loading screens.

Especially in many AAA titles, we are potentially switching between linear story elements and gameplay sequences regularly. In the playthroughs I’ve had with recent military shooters, 15 minutes of the game can have 2-3 moments of linear story. Using audio in conjunction with a visual effect or shift can make this transition feel effortless and seamless – almost like playing through a movie as opposed to pausing your gameplay experience.

Another flavor of smoothing transitions is during loading screens. Developers have come up with a ton of great ways to make loading screens less of a “drag” on the experience such as Namco having the Star Blade mini-game. 

But sometimes a traditional loading screen is inevitable, and audio and music can help make these moments much more interesting. Mute the video below if you want to see how much a loading screen with audio can be a complete bore.

Let’s Wrap Up!

We’ve looked at eight different ways audio can enhance your game, but there are many others as well! In order to ensure your game has an engaging experience, the audio needs to hit on all of these dimensions and be purposeful in its execution. Time and thought need to be given to what you’re trying to accomplish and how audio can help achieve it. Without that, a game will be missing an entire dimension of an effective and engaging experience.

Be sure to check out all the game audio awesomeness at Unlock Audio!

To reach out, [email protected]