By Winifred Phillips | Contact | Follow

Hello there!  I'm video game music composer Winifred Phillips.  Lately, I've been very busy in my production studio composing music for a lot of awesome virtual reality games, including the upcoming Scraper: First Strike first person VR shooter (pictured above) that's coming out next Wednesday (November 21st) for the Oculus Rift, HTC Vive and Windows Mixed Reality Devices, and will be released on December 18th for the Playstation VR.  My work on this project has definitely stoked my interest in everything VR!  Since the game will be released very soon, here's a trailer video released by the developers Labrodex Studios, featuring some of the music I composed for the game:

Scraper: First Strike is just one of a whole slew of VR games I've been working on over the past year.  Last year, when I was just starting to get really busy working with VR development teams, I wrote an article here that offered a bunch of informative resources connected to the field of VR audio.  The article I posted in 2017 took a general approach to the role that audio plays in Virtual Reality experiences.  Since we're well into 2018, I thought we could benefit from expanding that topic to include the state-of-the-art in VR headset platforms.  Taking a look at the hardware platforms that are currently available should give us video game composers a better idea of the direction that VR audio is currently headed.

For one thing, VR is now broadly considered a part of a larger category that also includes AR (Augmented Reality) and MR (Mixed Reality) devices.  Those two categories are often considered synonymous, although that's certainly debatable.  Since there's no clear expert consensus at this point on what characteristics separate AR from MR, let's just consider them as one category that we'll call AR/MR for now.  In this article I'll be focusing on resources that are specific to each of the competing platforms in VR and AR/MR.

Let's get started!

Audio for VR and AR/MR devices

A wide variety of head-mounted devices now exist that can immerse us in imaginary worlds, or bring fantastic creatures to life in our living rooms.  While many of these devices share common underlying technologies in regards to audio creation and implementation, there are differing tools and techniques that apply to each of them.  I've included links in the discussion below that may be helpful in understanding how these technologies differ.

When virtual acoustics meets actual acoustics

The newly-released Magic Leap One is an AR/MR device.  This means that it allows the wearer to see the real world, while superimposing digital images that seem to exist in reality, and not just within the device.  For instance, an AR/MR device can make us think that a miniature toy dinosaur is toddling across our coffee table.  With this in mind, creating audio for AR/MR becomes a little tricky.

For instance, let's say that we want our tiny dinosaur to emit a ferociously-adorable little roar as he climbs on top of our coffee table books.  That sound won't be convincing if it doesn't seem to be happening inside our actual living room, with its unique acoustical properties.  The real-life room has to be mapped, and acoustic calculations have to be factored in.  This isn't an issue when developing sound for virtual reality, since the sound sources emit within an environment that exists completely within the virtual world.

It's a fascinating problem, and one that the Magic Leap folks have considered seriously, using a system they've dubbed 'Soundfield Audio' to apply physics calculations that can produce appropriate acoustics based on the environment.  They've also patented a spatial audio technology that uses the wearer's head movements to calculate the position of virtual sound sources.  Here's a video that shows off a video game music visualization application for Magic Leap called Tónandi:

The Hololens is also an AR/MR device, and therefore faces a lot of the same issues as the Magic Leap One.  To address these, Hololens uses a spatial audio engine that calculates the position of sound-emitting sources combined with personalized Head Related Transfer Functions or HRTFs (a concept we discussed in an article from 2015).  These HRTFs help to localize all the aural components of the virtual soundscape.  In addition, the Hololens creates a room model to match the user's location so that sounds seem to reflect from real-life walls and travel convincingly to the player's ears.  We should expect this technology to improve when Microsoft releases their next generation of Hololens early next year.  Here's a video produced by Engadget that goes into more detail about the audio experience delivered by Hololens:

Spatial sound for mixed reality

While we're waiting for the next generation of Hololens to be released, Microsoft has been keeping busy in the traditional VR space with its Windows Mixed Reality platform, which allows third party equipment manufacturers to create VR headsets based on its existing VR reference designs and software.  While the Mixed Reality platform shares common elements with the Hololens, the VR devices under the Windows Mixed Reality banner offer standard VR experiences, without any AR/MR elements.  Both the Hololens and the Windows Mixed Reality devices use the Spatial Sound software development kit for the design and implementation of positional audio.  This allows audio developers to create soundscapes for a large number of devices using the same tools.  While the convenience factor is certainly attractive, Hololens and Windows Mixed Reality offer very different experiences, so audio developers will certainly need to keep that in mind.  Here's a short video that reviews the capabilities of the Spatial Sound SDK:

Positional audio inside the virtual machine

Now let's move on to discuss what's happening with the current VR devices.  As we know, unlike an AR/MR headset, a VR device cuts us off completely from the outside world and plunges us into an environment existing entirely within the machine.  There is currently a healthy and varied crop of VR devices from which to choose.  

The two most popular and famous VR headsets are the Oculus Rift and the HTC Vive.  Both devices rely on positional audio technologies to deliver great aural experiences, and each company has worked diligently to improve the technology over time.  

In June 2018 the HTC Vive introduced a new Software Development Kit (SDK) for immersive audio.  The new SDK allows for more sophisticated audio technologies like higher order ambisonics, higher resolution audio, more refined spatial acoustics (pictured right), and HRTFs based on refined real-world models to improve the accuracy of positional audio.

The Oculus Rift has upgraded their existing audio SDK to improve the positional accuracy of sounds emitting very close to the player (a technology they call Near-Field Head-Related-Transfer-Function (HRTF).

They have also provided the option of implementing sound that originate from large sources (an ocean, for instance, or a forest fire).  

Using the Volumetric Sound Sources technology, large sound-emitting objects can project their aural content across an assigned radius consistent with their scale.  

Here's a video from the Oculus Connect 4 conference, demonstrating the Near-Field HRTF and Volumetric Sound Sources capabilities of the Oculus audio SDK:

The PlayStation VR, as the only console-specific VR device, does not share the same market with such devices as the Vive or the Rift and therefore is not faced with the same competitive pressures.  Nevertheless, improvements continue to be made to the PSVR's technology.  The newest model of the PSVR (released late last year) is a revised version with small but valuable improvements.  Among the changes, Sony added built-in stereo headphones to the headset (pictured right), removing the need for players to hook up separate headphones in order to experience VR audio.

Standalone VR audio

Now let's take a quick look at the standalone VR devices (i.e. those devices that don't need to be hooked up to a computer or console, and don't need a mobile phone installed in order to work).  These VR headsets offer untethered, cable-free virtual reality exploration, but they're also usually a bit less powerful and full-featured.  The five best-known standalone headsets are the Oculus Go, the Oculus Quest, the Lenovo Mirage Solo, the HTC Vive Focus, and the Shadow VR.

The Oculus Go and Lenovo Mirage Solo both hit retail this May.  The HTC Vive Focus and the Shadow VR both became available for consumers just this month. The Oculus Quest was recently announced and is expected to hit retail in spring 2019. All five use a Qualcomm smartphone processor chip from the Snapdragon line, so in that respect they've essentially adopted the internal mechanism of a high-end mobile phone and simply incorporated it into their on-board hardware.  In fact, the Qualcomm Snapdragon 835 (used in the Lenovo, Vive, Oculus Quest and Shadow VR devices), is also the same chip that's at the heart of the Samsung Galaxy S8, the Google Pixel 2, and many other smartphone models.  Since all five untethered VR devices use the Snapdragon technology, developers can choose to avail themselves of the Qualcomm Snapdragon VR Software Development Kit, which includes a 3D Audio Plugin for Unity (designed to provide high-performance audio on Qualcomm Snapdragon devices).  Qualcomm also offers a suite of 3D Audio Tools for use in conjunction with a Digital Audio Workstation such as Pro Tools.  While these are by no means the only choices, since they were developed by the company responsible for the Snapdragon processor, it stands to reason that some Snapdragon insights may have influenced the design and function of these tools.  Here's a video interview with Hugo Swart, head of IoE Consumer Electronics at Qualcomm, as he discusses the virtual reality capabilities of the Snapdragon 835:

Audio for mobile VR

If you do a quick Amazon search under the phrase "Mobile VR Headsets," you'll see that there is now a dizzying plethora of headset models based around the "insert your mobile phone here" philosophy.  These headsets all rely on the processing technology of the phone inserted into them, and there are so many varying models that I won't be attempting to delve into that topic.  Generally speaking, if there is a SDK specific to a particular VR headset model, then it should be considered.  For instance, the Oculus Audio SDK makes sense for the Samsung Gear VR and the Oculus Go and Oculus Quest, since all three are Oculus-designed VR systems.  Likewise, the new Resonance Audio SDK from Google is a good choice for any of the Google headsets (Daydream View, Google Cardboard, Lenovo Mirage Solo with Daydream).  Here's a brief video produced by Google that demonstrates the Resonance Audio SDK:

Conclusion

That's our discussion of where things currently stand with regards to VR platforms!  In the next article, we'll be focusing on tips and tools for game audio folks working in VR.  I hope you enjoyed the article, and please let me know your thoughts in the comments section below!

Winifred Phillips is an award-winning video game music composer whose projects include the triple-A first person shooter Homefront: The Revolution. She has composed music for games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. Her latest video game credits include numerous Virtual Reality games, including The Haunted Graveyard, Audioshield, Scraper: First Strike, Bebylon: Battle Royale, Fail Factory, Dragon Front, and many more.  She is the author of the award-winning bestseller A COMPOSER'S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games.

Follow her on Twitter @winphillips.