Video game music systems at GDC 2017: tools and tips for composers

By video game composer Winifred Phillips | Contact | Follow

Welcome back to this three article series that's bringing together the ideas that were discussed in five different GDC 2017 audio talks about interactive music! These five speakers explored discoveries they'd made while creating interactivity in the music of their own game projects. We're looking at these ideas side-by-side to broaden our viewpoint and gain a sense of the "bigger picture" when it comes to the leading-edge thinking for music interactivity in games. We've been looking at five interactive music systems discussed in these five GDC 2017 presentations:

In the first article, we examined the basic nature of these interactive systems. In the second article, we contemplated why those systems were used, with some of the inherent pros and cons of each system discussed in turn. So now, let's get into the nitty gritty of tools and tips for working with such interactive music systems. If you haven't read parts one and two of this series, please go do so now and then come back:

Ready?  Great!  Here we go!

Tools and tips

To implement his interactive music system for the ocean exploration game ABZU, sound designer Steve Green relied on the popular Wwise middleware application in order to best facilitate transitions from one piece of music to another. "This idea of changing music on the fly rather than bringing instruments in and out was a good example of how to do ABZU," Green says. In order to implement these transitions, Green relied primarily on simple horizontal resequencing, with some limited vertical layering to ease transitions. At times, synchronization points were placed (usually at the beginning of each measure) in order to enable the music system to switch from one track to another. Sometimes, this approach was too abrupt.  "Transitionary pieces are basically to help two tracks that are just not going to flow well together," Green explains.

For instance, when the player is traversing a particular portion of a level (i.e. stage) the audio engine is able to keep track of the player's progress."We used a parameter that would gauge how far along you are in this stage of the level," Green says. "Once you reach the 90% range, (the audio engine) would call a transition." This musical transition would play simultaneously with the previous track, allowing the previous track to fade out and facilitating either a more gentle culmination into silence, or a more graceful segue to another piece of music.

During educator Leonard J. Paul's GDC presentation, he discussed his work on the audio team of the Vessel platforming game, but he also shared insights into several other projects from his repertoire, along with lots of general advice that's worth considering. "When working in games, of course, you've got to think about code," Paul says. "You've got to think about RAM – how much space this is all gonna to fit in – streams, if you’re streaming. You’ve got to think about what your RAM window size is and how fast you can get information in and out – how many streams you can do at once, what kind of compression you’re going to use," Paul continues, listing areas of technical concern for the game audio expert to think about when working on a project like Vessel. "Because if you don't," he warns, "then you’re going to run into some issues."

Paul also mentions some of the more famous software tools available to audio pros, including Wwise, FMOD Studio, Fabric, and Pure Data, along with a number of general words of advice. "Technical tips – use spreadsheets, be organized, use tools to refine your process, embrace prototyping," Paul urges. "Do audio sketches and get this stuff done early."

For Final Fantasy XV, audio programmer Sho Iwamoto created an audio engine he dubbed MAGI (Music API for Gaming Interaction). While the audio engine accomplished tasks similar to the functionality of other applications such as Wwise and FMOD, the MAGI system also included some creative and original specializations. "There are a lot of time signature variations or changes in JRPG music, so I designed MAGI to support them," Iwamoto says. Among its other capabilities, MAGI has the ability to adjust to changing tempo and time signature values across the length of a composition, allowing flexibility in the placing of synchronization points within the track. This capability was born out of both a spirit of innovation and pure necessity. Because Iwamoto was hired by Square Enix just two years earlier, much of the music of Final Fantasy XV had already been composed and recorded before he joined the team. "Many of these (compositions) were not planned to be interactive," Iwamoto admits. Because of this, the musical compositions included variable tempos, dramatic pauses, and frequent shifts in time signature.

In order to make these tracks interactive, Iwamoto used the MAGI system to insert custom synchronization points into each composition so that the transitions would be pleasing in a musical way. "All you have to do is just move or delete (sync points) when you think the transitions are not musical, or add the sync points when you think the transition is musical," Iwamoto comments. "Sometimes these sync points can be hard to find, and you may have to wait sixteen bars or more to (find a workable) transition." This problem became a frequent issue during combat music, so to solve this problem, Iwamoto devised a strategy that he called the pre-end section. This was so named because it is always used as a preparation for the ending of the combat track. "The pre-end section is designed to have more constant and dense sync points," Iwamoto explains. "That makes for really quick transitions." Existing as a separate music file, the pre-end section would always be in the same style as the main body of the composition, so the music could switch from the main body to the pre-end section smoothly. Designed to be more consistent in terms of tempo, key and time signature, the pre-end section could accommodate lots of sync points, allowing the music to transition to a finishing flourish with almost instantaneous precision in accordance with the end of combat. Despite its seeming complexity, Iwamoto meant for this system within the MAGI engine to be intuitive by design. "We used very simple and basic approaches," Iwamoto says. "I wanted to make (MAGI) very easy and versatile."

For audio director Becky Allen, simple and basic approaches are important for very different reasons. "We are a mobile studio and we have to be very mindful of our size because of the devices we’re played on," Allen says, describing her strategy for the MIDI music system of Plants vs. Zombies: Heroes. "We’re always thinking about size, and ways to be smart." Part of that strategy involved a plan for limiting the amount of memory required for the MIDI files and their associated library of instrument sounds. At first, those limitations were meant to be sensible but not extreme. "80 megs," Allen says. "40 for music, 40 for sound effects and VO." But then, the audio development process hit a snag when another mobile game came along and made a big splash in the marketplace. "Along came Clash Royale, and that was a quick game, it loaded quickly," Allen says. "We had some pressure to come down to 30 megabytes."

In the end, the team held to 60 megabytes, but even this required sacrifices. "We switched some things from stereo to mono but not everything. We streamed all the WAV music and we streamed larger WAV assets." After making these and other modifications, the newly streamlined music system was ready to show how interactive it could be. For instance, by associating the player's relative health with a RTPC (real-time parameter control) in Wwise, the music system could adjust MIDI data on the fly in accordance with the player's success or failure. "If you’re up by five points all the MIDI music went up a whole step, and if you’re up by ten points all the MIDI music went up two whole steps. And it worked, it was awesome," Allen enthuses. "Utilizing your tools and an open mind and a flexible mind and a flexible team," Allen says, "you can make these changes all the way along through the process."

If flexibility is the ultimate ambition of an interactive music system, then a procedural music system could be considered the definitive realization of that goal. While procedural music can be viewed as the height of interactivity, Paul Weir had built such systems for games before, and his viewpoint on the technology is much more down-to-earth. "Essentially they’re glorified random file players with a little bit of logic stuck on top," Weir admits. Together with the development team for the space exploration game No Man's Sky, Weir set about to see what procedural music could bring to the project. First, Weir hired a drum-driven electronica band called 65daysofstatic, then asked them to start creating music for their science fiction game. "We were kind of really adamant that as a band, I did not want to interfere creatively with their process," Weir says. "So we always said, write an album, just write us an album, and we’re not going to start telling you how to do that, because you know how to do that – because you’re the band. So we let them go off and write a very traditional album, but in the knowledge that we were going to come back to it later and just rip it all apart."

Using Pulse (his self-made procedural music generation tool) Weir set about disassembling the elements of the submitted music so that it could be incorporated into the procedural system. He quickly learned that it would require more than just a retooling of the original content. "So it wasn’t a question of saying just give us the stems and we’ll do it," Weir says. "It was like, no no no, go right back and do us more performances, take out bits, give us more drum loops, perform new guitar riffs, create new stuff – almost like kind of remixing the original tracks." With these musical fragments, the procedural system could then perform calculations and assemble the library of elements into new combinations in accordance with the player's locale and occupation. For instance, the music might be triggered depending on such in-game variables as proximity to buildings and time spent walking, utilizing what amounted to a large library of musical 'soundscape sets' that randomly alternated depending on how long any previous soundscape set had been playing. "In the game you don’t get the album," Weir insists. "The game soundtrack is bits of what appear on the album. It’s lots of bits that aren’t on the album, but it feels relatively cohesive."

Conclusion

So, there we have it!  We've compared the viewpoints of five game audio pros discussing the interactive music systems of their projects during their presentations at GDC 2017.  Interactive music is a fascinating subject, ranging from the simplest of solutions to the most complex and intricate of designs.  With ingenuity and patience, these audio developers have introduced their own creativity into the implementation of music in games.  Their strategies and solutions can help us to broaden our minds when we're thinking about ways to make our own music interactive!  I hope you've enjoyed this three-article series, and please feel free to leave your comments below!

Winifred Phillips is an award-winning video game music composer whose most recent projects are the triple-A first person shooter Homefront: The Revolution and the Dragon Front VR game for Oculus Rift. Her credits include games in five of the most famous and popular franchises in gaming: Assassin’s Creed, LittleBigPlanet, Total War, God of War, and The Sims. She is the author of the award-winning bestseller A COMPOSER'S GUIDE TO GAME MUSIC, published by the MIT Press. As a VR game music expert, she writes frequently on the future of music in virtual reality games.

Follow her on Twitter @winphillips.