The Positive Future of (Spatial) Audio for Games

(this blog post was first published on the gameaudiomix tumblr here.)

Dolby Atmos (for Home Theatre, and for Headphones) and a whole slew of other emerging ‘spatial’ or ‘3D’ audio technologies are offering a “new” dimension into which game sound designers can now work in terms of bringing depth and immersion to players. Now that the technical hurdles of getting these formats into game development pipelines and into people’s homes has been overcome, we are only left with the question of what exactly can we do with it?

Of course, in reality, surround sound formats, and 3D sound have been around almost as long as games consoles and PCs themselves (Sega Q-Sound et al), and that these newer immersive formats offer something revolutionary is easy to dismiss. However, having mixed a game in Atmos very recently (see this earlier post), I have some positive thoughts about how this heightened focus on sound, and sound placement in the 3D space, is going to lead to greater emphasis on sound in games and therefore lead to a deeper appreciation of all aspects of sound, music and voice, from players.

VISCERAL SOUND IN THE SHARED FICTIONAL SPACE

Traditionally, surround sound (in cinema at least) has always been about SPECTACLE, particularly in a theatrical setting, from innovative 5.1 surround movies like ‘Apocalypse Now’, to full Atmos mixes like ‘Gravity’, all have celebrated the VISCERAL qualities, effects and impact of sound on an audience. And, by moving sounds through space around the audience, enveloping them inside the world of the film. With surround, the audiences seemingly share the same acoustic space as the characters in the film, hear the same things they do, and react in a similar visceral way to how the characters react (A loud gunshot offscreen, or a T-Rex roar). The physical nature of sound, is emphasized and felt on an audience. By physically moving the air around them, most notably in the case of sub and low frequency effects, but also through positioning or moving sound around the audience, sound brings something physical and real (that has spatial dimensionality) to a cinematic experience that is simply not possible through the screen (light) alone – the more physical, impactful and convincing that sound ‘image’ is, the greater the potential emotional reaction.

“sound brings something physical and real (that has spatial dimensionality) to a cinematic experience that is simply not possible through the screen (light) alone“

SPATIAL SPECTACLE

With Atmos, the biggest innovation is, arguably, the inclusion of height, or ‘overhead’ speakers into the playback array, which creates a much more immersive ‘dome’ or ‘sphere of sound’ effect around the listener. So, now the audience no longer hears a flat disc of sound (7.1) around them, they hear positional sounds in a full 3D arc above them too.

This allows for the audience to hear what is above the characters onscreen in this shared fictional space. However, height brings a lot more to the table as it completes the sphere around the viewer, so we really feel like we are missing nothing. If a sound comes from above us, this is a specific hot zone in which we have some very hard-wired instincts – we REALLY want to ‘look up’ – exactly the same is the case for a sound behind us, we want to ‘look behind us’ – and this is the fundamental difference between video games and cinema - as a gamer, you hear a sound behind or from above, you CAN move the camera and look up or behind you. In cinema, you cannot. (You can, of course, but all you see is a cinema ceiling, or the exit sign, or someone eating popcorn).

In games, once you’ve established that sounds are above the player, and that they can look up at them, the ‘novelty’ effect falls away and eventually the new 3D spatial reality becomes quite normal, until you hear something that doesn’t belong or attracts your attention to a specific PLACE in the game space.

In this sense, games are far more free to go really far with adding sounds with gameplay, story, or exploration meaning to the overheads above the player, as well as surrounds, in order for the player to be encouraged to investigate the space itself, and also if a louder more threatening sound is heard, that sound can be more accurately pinpointed by the player and neutralized or safeguarded against (Yes, spatial audio actually has gameplay value!).

“In games, spatial sound has deep meaning, it serves gameplay, story, and encourages exploration.”

CINEMATIC CHALLENGES

We can certainly see some challenges to this kind of spatial sound approach when done in cinema, here is a small extract of Randy Thom’s notes on Alfonso Cuaron’s Roma mix…

(above: Roma)

“I assume that Cuaron was going for a certain kind of “immersive realism” by using this aggressive approach to surrounds, but for me it sometimes backfires, and actually makes scenes less “immersive” by yanking the listener out of the water every few minutes so that an unseen and unmotivated cat can yeowl, car horn can honk, or a dialog line can come from the surrounds.” – Randy Thom Wordpress Blog 12^th March 2019

Now, Cuaron, is almost synonymous with the Atmos format in that his earlier movie Gravity, was such an exceptional showcase for what the spectacle of the medium could be. I certainly agree that by having this new playground for sound available in film, it is easy to go a little too far in a focused storytelling moment, or try some things out that maybe later feel distracting. Though a distracting loud sound in Atmos, would still be a distracting loud sound in a stereo fold-down of that Atmos track (if loyal to the director’s vision), and therefore would still be a loud distracting sound – regardless of the format. The question is - would that sound still have been placed at that level in just a stereo or mono mix? (As perhaps the majority of the audience will currently experience it on Netflix) Did the availability of a spatial audio format, encourage this decision to paint with sound in that space outside the screen? Certainly. I feel Roma is such a completely different film, and vision, from most Hollywood cinema, that this kind of experimentation with sound should be applauded and celebrated as a break from the safe norms. While maybe not bringing more ‘immersion’ to the movie, that may not have been the goal. Spatial audio may bring more ‘ambiguity’ and even change how people remember the film. Because of the sound, I almost think of Roma as a truly 3D film, something it technically isn’t, but the experience, certainly is.

“I see video games as a major driver of this new aesthetic of vertical, immersive and spatial spectacle.”

In this sense then, games are so much freer to explore the creative spectacle of spatialization, given that the player can move the camera around to look at the cause of that sound behind or above you. And though this is something I think most audio designers working in surround have known for some time, with these newer more immersive formats, the point becomes even more vivid.

Emphasizing what is above and behind and around the player at specific moments, or in specific environments, unlocks a lot of new storytelling tools. Being able to collapse a mix from full Atmos 7.1.4 down to a mono centre-channel mix and back again, in a specific moment of a game, to exaggerate the feeling of claustrophobia, for instance, is something that is extremely effective and, now, fully achievable.

“It is precisely through this notion of the SPECTACULAR and the IMMERSIVE, that video games continue to market themselves, and through sound, we can now play perhaps one of the most important roles in truly DELIVERING those promises to our audiences.”

FOR GAMES, THE FUTURE IS SPATIAL

I see video games as a major driver of this new aesthetic of vertical, immersive and spatial spectacle. And in cinema, I think there is an audience who, perhaps being more used to video game mixes along these lines, are already open to film mixes placing things around the space, even if initially a little more distracting from the story or the character POV than ‘traditional audiences’ may be used to. Once the aesthetic is established, the ‘novelty’ will soon settle down.

(Shadow of the Tomb Raider’s Spatial mix was all about celebrating Vertical Spectacle)

Importantly, I believe that, as these various spatial audio formats gain traction (and fidelity) and become more widely embedded, the audience for games, gamers themselves, will usher in a new era of appreciation for the artistic and technical achievements of sound.

It is precisely through this notion of the SPECTACULAR and the IMMERSIVE, that video games continue to market themselves, and through sound, we can now play perhaps one of the most important roles in truly DELIVERING those promises to our audiences.

This is why I believe that spatial audio will find its strongest aesthetic expression of spectacle and immersion through video game sound.