Horizontal resequencing and dynamic transitions for game music composers

By Winifred Phillips | Contact | Follow

Welcome! I'm video game composer Winifred Phillips, and I'm happy you've joined me for the second installment in this series of articles based on the content of a lecture I gave during the Game Developers Conference 2021. My talk was entitled, "From Spyder To Sackboy: A Big Adventure in Interactive Music."

In my presentation, I compared and contrasted the interactive music design of two of my video game projects from the previous year. Both projects were developed by one of the game industry's top development studios – Sumo Sheffield. Both projects included ambitious dynamic music systems, using similar techniques and approaches. However, there were lots of differences in the execution of those techniques. This series of articles explores the similarities and differences between the dynamic music design of the popular Sackboy: A Big Adventure game for PS5/PS4, and Spyder for the new Apple Arcade (a great platform for supporting larger-scale game development for iOS). (Above you'll see a photo from one of the sections of my GDC 2021 lecture in which I'm discussing the music systems of both of these projects). If you missed the first article in this series, you can find it here.

In the previous article, we took a look at how the music design of Sackboy: A Big Adventure employed horizontal resequencing to introduce musical interactivity into my cover version of Madonna's famous Material Girl track. For those of us who could use a refresher, here's an excerpt of my discussion of horizontal resequencing from my book, A Composer's Guide to Game Music:

"In music, we mentally picture the ephemeral concept of time as a horizontal phenomenon, moving inexorably from left to right (like the notes on a musical staff). Likewise, most audio editing software visually presents audio as a waveform moving horizontally from left to right. The fundamental idea behind horizontal re-sequencing is that when composed carefully and according to certain rules, the sequence of a musical composition can be rearranged. This process occurs while the music continues to move forward on the horizontal axis of time, allowing a continuous free-flowing transformation of musical content...

"In horizontal re-sequencing, a musical composition is constructed as a collection of segments, each representing a discrete chunk of the overall composition. The duration of these segments can range from a scant few to a large number of measures. Each segment of music may contain digital markers that indicate convenient and logical points at which the game engine may switch between that music segment and another in the collection. These markers may be positioned on each individual beat, at the beginning of each measure, or at designated points in the composition where switching to another music segment would be feasible." (Chapter 11, page 188)

The audio team's expert dynamic implementation of this song focused on dividing it into segments. Since Material Girl adhered faithfully to traditional song structure, the song sections could be used as horizontal segments, and this yielded a highly-effective result that depended on the sharp delineations between those different portions of the composition.

The interactive system triggered the music to advance through song-structure components such as verse, chorus, bridge, and interlude, and this progression rewarded players by revealing new content as they proceeded through the level. Around the same time, the developer, Sumo Digital, was also asking me to compose horizontally-resequenced music for the Spyder video game, and that was a very different approach that employed more distinct segmentation and dynamic transitions. So let's now move to that technique in our list of dynamic music strategies:

Spyder is a pretty stealthy game – it’s not about running and jumping – it’s about being a secret agent, so the music had to be sneakier, and it had to leave space for players to think and plan. For puzzle solving and exploration, the music I composed for Spyder is horizontally resequenced, but those segments are designed to ebb and flow, and even settle into silence periodically.

There are also a lot more segments involved. Every level of the game had over thirty music segments. This provided the greatest flexibility for enabling divergent and unpredictable music during puzzle-solving gameplay, and the order of those segments is randomized so that players can’t anticipate what they’ll hear. Let’s check out an example of this system in action:

So you can see, this interactive system isn’t really indicatingprogress through the level the way the Material Girl music did. Instead, the focus of this music system in the Spyder game is to create musical variety, and avoid predictability. A system like this can keep game music from coming across as repetitive. However, the system also reveals its segments in an overt way, without making any attempt to hide its dynamic transitions.

So the Sackboy and Spyder approaches to horizontal resequencing are very different. However – there are similarities here.

Like the Material Girl example from Sackboy: A Big Adventure, this system in the Spyder game is taking an overtly segmented approach. We’re aware that the music has different sections that transition into each other (such as those pictured left), and the dynamic music system structures itself around that awareness.

But what if that isn’t the goal? What if we want the horizontal resequencing system to completely hide its dynamic transitions? To get a taste of how that approach would work, let’s look at one of the original tracks I composed for Sackboy: A Big Adventure: "Shoo-Bee-Doo Hullabaloo."

This is the boss battle music at the end of the awesome "Thar She Blows Up" level. Like Material Girl, this boss battle track is structured in horizontal segments – in this case, three progressively more dramatic chunks of music. But the implementation strives towards a seamless experience, with the goal of hiding the dynamic transitions rather than showing them.

So let’s see how that worked:

As you can see – pure horizontal resequencing can be a very flexible approach to injecting interactivity into our game music, and there's more than one way to use it. We can aim to make our music feel seamless, or we can present our music segments in an overt way. And both approaches can be effective.

So now we've explored how these two games have adapted the pure horizontal resequencing technique in different ways. In the third article of this series, we'll approach the next item in our list of dynamic music techniques:

When we resume our discussion, we'll be exploring how pure vertical layering can be adapted to gameplay circumstances, using several examples to demonstrate how this dynamic technique can be molded and shaped to accomplish different tasks. Until then, thanks for reading!

About the author

Winifred Phillips is a BAFTA-nominated video game composer whose whose most recent project is the music of the latest blockbuster release in the Lineage series - one of the highest-grossing video game franchises of all time. Popular music from Phillips’ award-winning Assassin’s Creed Liberation score is featured in the performance repertoire of the Assassin’s Creed Symphony World Tour, which made its Paris debut in 2019 with an 80-piece orchestra and choir. As an accomplished video game composer, Phillips is best known for composing music for games in five of the most famous and popular franchises in gaming: Lineage, Assassin’s Creed, God of War, Total War, The Sims, and Sackboy / LittleBigPlanet. Phillips has received numerous awards, including an Interactive Achievement Award / D.I.C.E. Award from the Academy of Interactive Arts and Sciences, six Game Audio Network Guild Awards (including Music of the Year), and three Hollywood Music in Media Awards. She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As one of the foremost authorities on music for interactive entertainment, Winifred Phillips has given lectures at the Library of Congress in Washington DC, the Society of Composers and Lyricists, the Game Developers Conference, the Audio Engineering Society, and many more. Phillips’ enthusiastic fans showered her with questions during a Reddit Ask-Me-Anything session that went viral, hit the Reddit front page, received 14.9 thousand upvotes, and became one of the most popular gaming AMAs ever hosted on Reddit. An interview with her will soon be published as a part of the Routledge text, Women's Music for the Screen: Diverse Narratives in Sound, which collects the viewpoints of the most esteemed female composers in film, television, and games. Follow her on Twitter @winphillips.