Game Music and The Theory of Flow: How Game Composers Can Enhance Virtual Presence (Pt. 1)

By Winifred Phillips | Contact | Follow

Hello there!  I'm video game composer Winifred Phillips.  At this year’s Game Developers Conference in San Francisco, I was pleased to give a presentation entitled How Music Enhances Virtual Presence (I've included the official description of my talk at the end of this article). The talk I delivered at GDC gave me the opportunity to pull a lot of ideas about virtual reality together and present a concentrated exploration of how music can increase a sensation of presence for VR gamers.  It occurred to me that such a discussion might be interesting to share in this forum as well. So, with that in mind, I’m excited to begin a four-part article series based on my GDC 2019 presentation!

Over the past couple of years, a lot of development studios have hired me to create music for virtual reality games. It’s fascinating work! In my GDC talk, I discussed virtual presence in connection with seven of the virtual reality games and experiences that I've scored, which have either released within the past year or will be released within the coming months.  These include the PSVR version of the Scraper: First Strike VR shooter that was released just last week by Labrodex Inc, and numerous other VR titles including  Audioshield (Audiosurf LLC), Bebylon Battle Royale (Kite & Lightning), Fail Factory (Armature Studio), The Haunted Graveyard (Holospark), Life Hutch VR (Next Stop Willoughby), and Shattered State (Supermassive Games).

Gaming in virtual reality involves an engulfing 360 degree spherical environment and a sense of physical agency never-before possible with traditional gaming. Unlike other forms of video gaming, players in VR have the best chance to feel bodily present inside the virtual world. This sense of Virtual Presence offers an awesome opportunity for expert development teams to create powerful gaming experiences.

During this four-part article series, we’ll explore what Virtual Presence is. We’ll take a look at some creative composition strategies we can use to induce Virtual Presence.  We’ll discuss some of the possible drawbacks of Virtual Presence, and what we video game composers can do to support players and keep the fun going during their adventures in VR. I hope you’ll find these creative strategies to be useful in your own VR projects. Keep in mind that, while all these ideas are primarily designed to make gaming more fun in the popular VR gaming platform, they are essentially meant to get players more viscerally involved in any kind of gameplay, which means they can be applied to traditional game projects too.

Let's start with the basics.

What is Virtual Presence?

According to Professor Thomas B. Sheridan of the Massachusetts Institute of Technology, Virtual Presence is defined as the sensation of being “present in the environment generated by the computer.” In order to feel virtually present, we have to accept the in-game location as a tangible place. According to Professor Sheridan, there are lots of methods that VR designers can use to accentuate the reality of their virtual environments. They can shower players with sensory stimuli. They can encourage players to move about, changing their relative viewpoint and altering their binaural soundscape. They can place objects in the environment that can be manipulated and changed by players, leading to a great sense of material engagement.

Here’s an example – one of my most recent projects, The Haunted Graveyard, from developers Holospark. As a Halloween-inspired VR experience designed primarily to be accessible to a wide audience in top VR arcades, The Haunted Graveyard focuses on an involving, atmospheric landscape for players to explore. Virtual Presence is extremely important, so the game provides the necessary sensory stimuli, player movement and usable objects that Professor Sheridan described. However, the game also includes lots of music to keep players emotionally stimulated. Here's a ten-minute gameplay video of The Haunted Graveyard:

In order for true Virtual Presence to be attained, players have to let go of their natural incredulity and get emotionally involved. They have to forget about the fact that they’re in VR, and just live the adventure.

As you saw from the gameplay video, the music in The Haunted Graveyard is an important part of the game’s design.

Music can be inspiring – it can make you feel more committed to what you’re doing, and more invested in the world around you. The question is – how does music enable Virtual Presence? And what tools can video game music composers use to make that happen?

In this four-part article series, we’re going to explore three ways wherein music enables Virtual Presence:

In this article, we'll start with the first (and most famous) mechanism on our list:

The Theory of Flow

First introduced in the book Flow: The Psychology of Optimal Experience, the Theory of Flow tries to explain what it means when we get so absorbed by some interesting task that we forget everything else – all distractions fall away and our minds sharpen to a laser-point of focus. Sometimes it’s called being “in the zone.” The feeling is an incredible rush. Moreover, it’s a useful tool when trying to instill the sensation of Virtual Presence in our players.

In a study published in the journal Computers in Human Behavior, researchers from Indiana University found that the Flow phenomenon, when experienced in a virtual world, can enable and enhance the sensation of Virtual Presence. In other words, when we’re feeling “in the zone” during VR, we also tend to feel more like we’re actually inside an alternate world.

With that in mind, how can we game music composers use this phenomenon to our advantage? Well, science has shown us that music has a direct correlation with the Flow state. In an experiment conducted at Brunel University, researchers studied athletes who listened to music during their exercise routines. The researchers found that music had a strong beneficial effect on the athletes’ flow state, helping them to get “into the zone."

Here’s an example – one of my most recent projects was the main theme for Audioshield – a music rhythm game for VR. Using a shield in each hand, players block glowing orbs that fly towards them in synch with the music. The game analyzes the unique qualities of the music to determine the type and frequency of orbs coming towards the player.

Because Audioshield analyzes music and then constructs gameplay around it, composing music for Audioshield was a real trial-and-error process. All during music production, the game designer and I both repeatedly loaded the music into the game and played through it, looking for any orb activity that felt jarring, or choppy, or rough – anything that would interrupt Flow. I’d go back into the music studio, make changes – then we’d play through the track in Audioshield again. Were the orbs showing up in ways that were too surprising? Or not surprising enough?

We were looking for a sweet spot, when Flow would grab hold, when the player would be firmly in the zone. And when that happened, it felt like everything came together. The gameplay was fun, the music was driving, and you really felt like you were there. Everything seemed tangible and real. Flow and Virtual Presence kicked in together. Here’s a video showing how gameplay worked during the main theme of Audioshield:

So we've now discussed the relationship between Flow Theory and the concept of Virtual Presence.  In our next article, we'll examine the second mechanism by which music enables Virtual Presence – psychological attachment.  Thanks for reading!

How Music Enhances Virtual Presence

(Game Developers Conference Session Description)

Virtual Presence is defined as a state in which gamers fully accept the virtual world around them and their existence within it. This talk, “How Music Enhances Virtual Presence,” will explore how highly effective game music can enhance the sensation of Virtual Presence in VR gaming.

The talk will begin with an exploration of both the Flow Theory of Mihaly Csikszentmihalyi and the research of Dr. Paul Cairns on psychological engagement in video gaming. By understanding how the mental activity of players interacts with the way a game is designed, composers can create music intended to induce psychological states conducive with the formation of Virtual Presence.

The talk will include a discussion of techniques aimed at drawing attention to mission objectives, facilitating effective concentration, enhancing emotional empathy and intensifying player focus. The discussion will also include an exploration of some inherent drawbacks to Virtual Presence, including its fragility when exposed to negative emotional states, and its possible susceptibility to inducing the “event boundary” phenomenon. Musical solutions to these problems will be explored.

Phillips' talk will offer techniques for composers and audio directors who seek to employ music as a tool to enhance Virtual Presence for their players.

Takeaway

Using examples from several games, Phillips will explore how music can influence the mental states of players through specific effects documented in scientific research. Study data will be discussed in regards to the interaction between music and cognition. Phillips will offer strategies and tips for composers seeking to use their music to influence the player’s mental state, thus facilitating the formation of Virtual Presence.

Intended Audience

This session is intended to inspire and stimulate composers seeking to employ their music towards enhancing player engagement and enjoyment, with a particular emphasis on VR games. Includes overview of Flow Theory and the psychological components of Virtual Presence, which may be useful to other disciplines within game development. Talk will be approachable for all levels (advanced composers may better appreciate the specific composition techniques discussed).

Popular music from composer Winifred Phillips' award-winning Assassin's Creed Liberation score is currently being performed live by a top 80-piece orchestra and choir as part of the Assassin's Creed Symphony World Tour, which kicked off in June 2019 with its Paris premiere. As an accomplished video game composer, Phillips is best known for composing music for games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims.  Phillips' other notable projects include the triple-A first person shooter Homefront: The Revolution, and numerous virtual reality games, including Sports Scramble, Audioshield, Scraper: First Strike, Dragon Front, and many more.  She is the author of the award-winning bestseller A COMPOSER'S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games. Phillips' is a sought-after public speaker, and she has been invited to speak about her work as a game composer at the Library of Congress, the Game Developers Conference, the Audio Engineering Society, the Society of Composers and Lyricists, and many more.

Follow her on Twitter @winphillips.