Horizontal Resequencing and Song Structure for Game Music Composers (From Spyder to Sackboy: GDC 2021)

By Winifred Phillips  |  Contact  |  Follow

Hey everybody!  I'm video game composer Winifred Phillips, and I was excited this year to present a talk at the Game Developers Conference.  GDC is one of the top conferences in the video game industry, and it was a fantastic event this year, full of expert sessions and lots of opportunities to learn and network.  As in previous years, I thought it might be best if I included the content of my GDC lecture in my articles here, so I'm now kicking off a six-part series of articles based on my presentation in July!  I've included the substance of my GDC presentation, supported by some of the multimedia materials I used to illustrate concepts during my lecture.  I've also enlarged upon most of those topics with a bit of further explanation that couldn't be included in my original GDC presentation (due to time constraints).  So now without further ado, let's get started!

Back in the summer of 2019, I started working with Sumo Sheffield on music for two different games. It was pretty intense work over the course of many months on two awesome projects with very different musical needs.  The list of musical requirements was quite long for both projects, and I spent a lot of time ping ponging back and forth between them. After they both hit retail in 2020, I realized how eye-opening that experience had been. Both games required complex musical interactivity, but each met that goal in very different ways.

In this article series, we’re going to be taking a look at those two projects: Sackboy: A Big Adventure for the PS5/PS4, and Spyder for Apple Arcade.

Both projects used some common dynamic implementation models. Over the course of these articles, we’ll discuss how each project incorporated these implementation strategies in the list to the right.

As we know, dynamic music differs from linear music in fundamental ways.  Linear music is a single contiguous unit, like a straight line moving in one direction, with a beginning, middle and end. But dynamic music is more like a maze that can move in many directions with lots of divergent possibilities.  The art of interactive music creation and implementation is not just about understanding and deploying interactive music systems (such as the ones on the pictured list.)  It’s also about looking at the nuts and bolts of these systems and seeing all of those divergent possibilities.

Sackboy: A Big Adventure and Spyder both did a great job in taking dynamic music techniques and expanding on them in different creative ways. So let’s explore that!  But before we jump in, I’d first like to introduce you to our two heroes.

Of course, everyone is well acquainted with that international superstar, Sackboy! His incredibly popular escapades in Craftworld have become the stuff of legend. Sackboy’s latest Big Adventure is a smash hit action game that pits him against the evil Vex, who’s bent on destroying Craftworld with his terrible Topsy Turver machine. Of course, as Craftworld's greatest adventurer, Sackboy must act fast to save the day (as he always does).  Let's watch Sackboy in action!

While Sackboy is famous, many of you may not have heard of our other hero –– that amazing covert operative, that groovy agent of mystery, that globetrotting defender of truth and freedom... Agent 8!

It’s understandable to overlook him – he’s less than an inch tall.  Spyder is an Apple Arcade exclusive in which Agent 8 undertakes stealthy missions to save the world many times over – though the British Spy Agency would disavow any knowledge of his actions, should he be captured… or stepped on.

Working on music for these two projects at the same time was an intense and rewarding experience.

For Sackboy: A Big Adventure, I joined a diverse music team, and was assigned the task of composing original music for Sackboy’s underwater escapades. I was also one of the team members responsible for creating cover music to fit into Sackboy’s world.

For Spyder, I was the sole composer, writing original 1960s and 70s-inspired music for Agent 8’s sneaky spy missions.  Both games were developed by the experts at Sumo Sheffield, and both included ambitious music systems using similar dynamic strategies.  There are a lot of differences that are interesting to us as game composers – and that will be the subject of our discussion.

Coming back to our list of implementation procedures, let’s start with horizontal resequencing.  Both games used the fundamentals of this technique, so we’ll now focus on a straightforward example that's connected to a profoundly familiar musical technique – song structure.

Like the rest of the games in the franchise, Sackboy: A Big Adventure is a celebration of music – incorporating original compositions, licensed pop tracks, and cover songs.

Along with the original music tracks I composed for Sackboy, the team also asked me to create a cover song – a brand new version of Madonna’s Material Girl.  My version reimagined Material Girl as a classical orchestral composition, while also keeping enough elements to make sure it remained recognizable.

The orchestral arrangement was recorded at Air Studios London, and then the final mix was separated into segments for implementation.  That’s where Horizontal Resequencing comes in. When we compose music in segments that can be triggered in different orders, that’s Horizontal Resequencing.  Shuffling things around makes the music more responsive to the game, and also helps the music feel less repetitive.

Material Girl is broken into segments according to its song structure, which is a pretty straightforward approach.  Songs typically have some form of an introduction, followed by a verse that states some of the core ideas of the song.  After that, we get the emotional chorus section that gets right to the heart of the matter.  After the chorus, we may have a section that leads us to a new thought (sometimes called a bridge).  As the song starts approaching the end, we may get an interlude section.  Then, finally, we'll hear an outro section that winds up the song and provides the listener with a sense of closure.  For this interactive version of Material Girl, we get all these song-structure components implemented as segments in a Horizontal Resequencing system. They’re all separate fragments with their own trigger points.

First we get the intro, which plays once.  This transitions into the verse and chorus – which is combined into one segment that’s essentially the main loop. Let’s take a look at that:

After a while we get to the bridge – it’s implemented as a short loop that triggers right before Sackboy goes flying across a chasm.

Once he lands, the music jumps right into the interlude section. So let’s see how that part worked:

Following the interlude, we go back to the main loop for awhile, and then we finally get the outro segment at level completion.

So let's check that part out:

You can see how the music provides a sense of reward.  As Sackboy progresses, the music advances to reveal new content.  A dynamic music system like this one has the primary goal of indicating progress through the level.  In the case of Material Girl, the advantage of song structure is that it provides well-defined segments that are inherently dissimilar, so jumping from one segment to the next feels very dramatic. We’re meant to notice what’s happening – it’s part of the reward.

So, we've taken a look at one of the dynamic music implementation strategies from Sackboy: A Big Adventure.  In the next installment in this article series, we'll examine how the Spyder video game approached traditional horizontal resequencing using dynamic transitions designed to keep the musical texture fresh for long-term gameplay.

After that, we'll contrast this dynamic music approach against a more context-sensitive implementation strategy in Sackboy, to see how flexible the horizontal resequencing technique can be.  Until then, thanks very much for reading!

Winifred Phillips is a BAFTA-nominated video game composer whose most recent project is the music of the latest blockbuster release in the Lineage series - one of the highest-grossing video game franchises of all time.  Popular music from Phillips’ award-winning Assassin’s Creed Liberation score is featured in the performance repertoire of the Assassin’s Creed Symphony World Tour, which made its Paris debut in 2019 with an 80-piece orchestra and choir. As an accomplished video game composer, Phillips is best known for composing music for games in five of the most famous and popular franchises in gaming: Assassin’s Creed, God of War, Total War, The Sims, and Sackboy / LittleBigPlanet.  Phillips has received numerous awards, including an Interactive Achievement Award / D.I.C.E. Award from the Academy of Interactive Arts and Sciences, six Game Audio Network Guild Awards (including Music of the Year), and three Hollywood Music in Media Awards.  She is the author of the award-winning bestseller A COMPOSER’S GUIDE TO GAME MUSIC, published by the MIT Press. As one of the foremost authorities on music for interactive entertainment, Winifred Phillips has given lectures at the Library of Congress in Washington DC, the Society of Composers and Lyricists, the Game Developers Conference, the Audio Engineering Society, and many more. Phillips’ enthusiastic fans showered her with questions during a Reddit Ask-Me-Anything session that went viral, hit the Reddit front page, received 14.9 thousand upvotes, and became one of the most popular gaming AMAs ever hosted on Reddit.  An interview with her will soon be published as a part of the Routledge text, Women's Music for the Screen: Diverse Narratives in Sound, which collects the viewpoints of the most esteemed female composers in film, television, and games.  Follow her on Twitter @winphillips.