Story in Games Pt. 2: Sound Design and New Leaves

Originally posted on the Team Junkfish blog.

We're on to the second proper part of my little series of blogs covering the use of audio in storytelling for games. Previously on I spoke about the use of music in games, and I also did a writeup how we're going about those ideas in Monstrum. This time I'm going to talk a bit more about the use of sound effects and how they build up the world and everything in it.

Before I start, here's a Gamasutra article by Andrew High on game music that is very much worth a read.

The IEZA Framework

Let's begin with some theory first and explain some terms that I'll be using later. I'll try keep it short! The IEZA Framework can be used to describe the sonic environment in a game, and is a good starting point when conceptualising the sound of a game. It looks like this:

Basically, it breaks down game audio along two axes: “Diegetic” and “Non-Diegetic” which we've covered before (primer: diegetic when something originates within the game world, such as footsteps, non-diegetic: something outside the world, like music with no in-game source) and “Setting” and “Activity” which relate to, in basic terms, the overall environment and mood of the game and the actions that can happen in it respectively. From there it gets broken down into four quadrants:

“Interface” - Cues that are used to provide information to the player, such as health pickups or menu selections. It lies on “Non-Diegetic”, as your character won't be able to hear you changing menu settings as it's disconnected from the in-game world, and “Activity” as the player is performing an action in order for the sound to happen.

“Effect” - Relates to things like footsteps, gunshots and other sonic responses to player actions. As such it lies on the “Activity” axis, with the sounds falling on “Diegetic” axis due to their in-game world origins.

“Zone” - Relates to the sonic space that the player currently inhabits and should be designed to replicate that environment. This lies on the “Diegetic” and “Setting” axes, as it has in-game origins, but it's not being driven by player actions. Think environmental sound effects, traffic, wind, that sort of thing.

“Affect” - This one is a bit nebulous, as it relates to audio that somehow “affects” the player, such as setting a mood via music. This could be something like an orchestral sting when a monster jumps out at you, or addressing an area you happen to be in (like bagpipes in Scotland, for one stereotype). This sits on the “Non-Diegetic” and “Setting” axes.

By following this framework we can define different types of sounds, and from there we can discuss how they assist in world building and story telling. For a more in-depth read check this article by Richard van Tol and Sander Huiberts.

So now that that's out the road, let's get down to business. I'm going to try break down the audio from a few games into these categories and explain how they help with world building and storying. So without much further ado...

Animal Crossing: New Leaf

Probably not a game that people would expect give that Team Junkfish are currently working on a horror title, but "Animal Crossing: New Leaf" (I'll use Animal Crossing in reference to this title from here on it) has some rather interesting and detailed sound design choices that help build up the world and the characters in it. In the game you play as the new mayor of a town, and it's up to you to bring in new residents and drive its development forward. The game rotates through a day night cycle in real time, and also has a weather system in place, both of which have an effect on the sonic background of the game. There are a couple of points that I quite like about the game, so I'll break them down into the IEZA sub groups where applicable in this.

Effect

Starting with some of the more obvious ones: footsteps! The easiest way to give somewhere a sense place is through footsteps (and reverb). Changing the sound for each surface is an obvious thing, but it it's still rather effective. Going from town's dirt ground, down a stone ramp to the beach and then across the shoreline with the sound changing beneath your feet just feels right. Even the carpets in your house sound different depending on their texture. Similarly, when the rain drifts into town the footsteps naturally change to reflect this, but they aren't alone in this regard. If you go digging in the rain the mud takes on a denser, wet sound as you throw it about and pack in holes. It's small, it's subtle, and yet it makes the game world seem a bit more plausable, a bit more believeable, for it.

They do!

Another interesting thing in Animal Crossing is the speech. When you talk to characters in the game they start “speaking” in a seemingly garbled language which, if you listen closely, seems to replicate what's in the text box! While any sort of chatter could have done this vocalisation, in addition to giving the townsfolk some degree of variation with their “voices” (although they aren't completely unique) adds to their own character. When Timmy or Tommy Nook are trying to sell you something and dip into their little quiet sales pitch, which is reflected in the text by a smaller font as well, you can believe that they're talking to your player. This also ties into...

Interface

A further nice little touch in the game is the text entry. As you enter each letter your character seemingly reads it out for you, giving them their own voice among the crowd. It's particularly fitting when another character asks you for an answer or you send out a message and you get to vocalise what is said (to some degree).

Zone

The town itself has some obvious zone related sounds effects, namely the beach area and when you stand near the river as you hear their respective flowing waters. But it also makes use of ambient sound to alert you to collectable insects, giving certain families or bugs their own calls so that you can not only tell where an insect is but what type it may be. This is a nice little touch, as the developers could have went with one stock insect noise but instead populates the world with a variety of life. It also makes it easier to try and find insects you haven't caught for your collection too. There are also items that float into the town on balloons, which tend to duck the music to allow a wind sound to play as the balloon comes into view. It is a small way to attract the players' attention to a focal point by utilising environmental audio.

On a side note, the 3DS actually has a surprisingly decent HRTF (Head Related Transfer Functions) implementation with its speakers that doesn't get enough love or attention. Nintendo really went all in on the 3D! But implementation is another key part of game audio and is often overlooked by developers. This is, however, a blog post unto itself and has a fair amount of tech considerations too!

It's not the Old Grey Whistle Test, but still.

There are also a few instances of diegetic music in the game. Most obviously is Club LOL, the nightclub. It occasionally features the guitar playing canine troubadour K.K. Slider, on top of general club nights which he DJs under the psuedonym DJ K.K.. On Saturdays K.K. Slider will perform various songs, which you have to actively become the audience for by taking a seat, take requests and pass copies of tracks on to you to play in your house. Similarly, when travelling to and from the island you'll also be greeted with a little sea shanty from the boat's helmsman, Kapp'n. While the music remains the same, he tends to change the lyrics quite a bit, singing along via his synthesised voice.

Affect

Lastly, the use of music in Animal Crossing in terms of world and character building is pretty intriguing too. There is a “town theme” that each potential villiager has their own variation of. This little jingle plays whenever you talk to someone, and manages to put across some aspects of their personality. If you don't like the original you can actually change the tune that's played, allowing you to impart some of your own personality back into the town too! This is limited to 16 notes and the scale of C Major, but is still a nice feature that plays on the game's customisation and town building motif. If you play the game different times of the day you'll also get different background music, reflecting how active the town is.

There's also a little throwback/Easter egg in the Nook brother's T&T Mart, which riffs off of the instrumentation and themes from the Wii Shop Channel's music. Of course, no Animal Crossing game would be complete without “Tokata's Song”, a notable musical 19-note Easter egg by Nintendo sound designer Kazumi Totaka that appears in a number of different titles which he has worked on. He's also the source of K.K. Slider's name and the voice of Yoshi. The more you know.

These are only a few examples of Animal Crossing's well thought out and crafted sound design, and only from one game too. Another excellent example of using sound effects to build up the game world and story is ThatGameCompany's “Journey”. Sound designer Steve Johnson wrote a detailed breakdown of his process which you can read here. Visually the world of “Journey” is stunning, but the sound from both it and its inhabitants make really bring the game to life. The article also has sound examples, so give it a read! Similarly, the sound design in "Abe's Oddysey" and "Abe's Exoddus" allows players with little or no vision to navigate through and complete the game. I'll talk about that in the next major part of this series, which will have look at accessability and Audio games, and might even touch on the implementation side of things too.

Cheers,
Jaime

Listed and Further Reading

Caleb Bridge - Creating Audio That Matters (Gamasutra) - Touches on audio immersion

Andrew High - Is Game Music All It Can Be? (Gamasutra) - Excellent, lengthy article on game music

Steve Johnson - The Sound Design of Journey (Gamasutra)

Darren Korb - Build That Wall: Creating the Audio for Bastion (GDC Vault)

Kirk Hamilton - How To Compose A Great Town Theme Song In Animal Crossing: New Leaf

Richard van Tol and Sander Huiberts - IEZA: A Framework For Game Audio (Gamasutra) - Breakdown and examples of the IEZA framework

IZEA Framework (Wikipedia)

Kristine Jørgensen - Left in the dark: Playing Computer Games with the Sound Turned off - Talks about how audio provides information and how a lack of sound changes players' perceptions. It's part of "From Pac-Man to Pop Music", a collection of studies covering a variety of different game audio topics.

Karen Collins - Playing With Sound - Another of Collins' books, this time on how players interact with game audio as opposed to listening to it, and how it can alter players' experiences.

Andy Farnell - Designing Sound - A technical book on sound design, and practically a bible for anyone interested in PureData and DSP.

David Sonnenschein - Sound Design - A more general sound design book, but covers a lot of the theory behind it very well.

Use Distinct Sound/Music Design (Game Accessibility Guidelines) - More of a segue into the next topic, it has examples of a blind player playing through "Abe's Oddysey" and beating Ed Boon at "Mortal Kombat" through sound alone.