Adaptive Music

Author: by Kurt Akeley

What is music?
In a freshman "Introduction to Ethnomusicology" course I attended at the University of Toronto, the entire first class was dedicated to attempting to agree on a definition of "music." Easier said than done! Luckily, since this paper is a solo effort, I get to pick the definition without argument. Music: structured patterns of sound in time.

What's in a name?
In many ways that was a terrible definition. For one thing, it relies heavily on the fact that you already know what music is. (The word "music" is a symbol with which you already have years of personal experience.) Really, its sole purpose is to focus our attention on some fundamental elements of music, without getting distracted with sticky issues like art, creativity, culture, human involvement, and intent.

Sound Structures in Music
Part of the reason that music is so appealing is the number of levels of patterns that are going on at once. Sound comes from very fast air pressure changes at rates of roughly 60 to 23000 times a second. Phrases involve the arrangement of these sounds on a much different temporal scale. The sounds become melody and rhythm, and harmonic motion. Structure is a higher level organization of phrases. (E.g. classical Sonata form, or the ordering of verses and choruses in a pop song.) Very different parts of the brain process and appreciate these different elements simultaneously when you are listening to music.

Equivalent structures in language
A non-musical structural equivalent of a sound would be a word in language, e.g. the word "sound". The equivalent of tone (the quality or "color" of a sound) might be font. This sentence is a good example of a phrase. This paragraph could be thought of as a substructure, and the essay as a whole as a much larger architectural structure or form.

Structures and Adaptability in Language

This article explores adaptive art in the context of language. This is because language is much easier to demonstrate and discuss in the written medium than music is! So, despite the title "Adaptive Music," this article doesn't talk about music at all - except by analogy.

An Adaptive Poem
I used to write video game music for a living. It was a very exciting job, but it also had its share of frustrations. For instance, part way through one particular project, I discovered that the lead programmer hadn't bothered installing a sound card in his new computer. As one might expect, this was making it kind of hard for him to test the interactive score I was writing. (True story! The next part is made up though.)

I decided that it might be a good idea for me to write some interactive poetry for him to work with temporarily. (We could even keep it as an option in the release version of the game, for the hearing impaired.) Like the music, the poetry would change in real time in order to reflect the changing action of the game and to enhance the atmosphere. It would be of extremely high quality, and interesting enough that the player wouldn't get sick of it after a mere eight hours of game-play.

An adaptive poem?!!
Writing adaptive poetry sounds impossible, doesn't it? Well, it's essentially the same challenge that an adaptive music composer faces… but more on that later. For now, let's try to prototype an adaptive poem and see what happens. The first step will be nailing down the tone and style, and what we want the piece to communicate. The following example is a linear poem that will help us figure out what we want the interactive version to do. (The game, by the way, is called Seal Hunter and features the heroic Captain Brave-O.)

Example: Linear Poetry

Adaptability at the word level
The action and mood of Seal Hunter can change quite suddenly. Captain Brave-O could be walking around being brave and heroic one minute, and then diabolical seals could leap out at him the next! Ideally, our accompanying poem should be just as dynamic. Unfortunately, since the game player will be controlling Captain Brave-O's actions, we have no way of knowing in advance what is going to happen, when. One way to try to deal with this is to compose a poem that is flexible at the word level, so that it can change moods at any time.

The next example randomly selects words from one of four word lists. Each list contains only words that are appropriate to a particular game mood. When the action of the game changes, the source word list is also changed, thereby changing the mood of the poem on the fly.

Example: Adaptive Word Texture

That was surprisingly effective and very dynamic; however, aesthetically it might be considered somewhat lacking. It was more like word texture than traditional poetry -- there was no real attention to rhyme or meter. Let's see what happens when we try a less random ordering of words in an attempt to address this:

Example: Adaptive Word Texture with Meter and Rhyme

The second example was essentially the same as the first, except for two small changes. This time, the word list for a particular mood consisted only of words sharing the same foot. Additionally, I added special lists of rhyming pairs. Every fourth word was selected from this list, in order to form rhyming couplets. The result was a kind of pseudo phrase structure (but without any real sentence-level meaning).

This more structured version came at a slight cost. Notice that when the sudden seal attack happened, it broke the rhyme and rhythm scheme of Brave-O's strut poem. If we'd waited for the line to finish, the attack poetry would have been late and much of the effect would have been lost. Having to make this kind of compromise makes this solution less dynamic than pure word texture. On the other hand, patterns of rhyme and meter do have an undeniable attraction.

Patterns

The human brain is fantastically adept at recognizing all kinds of patterns. The human brain is fantastically adept at recognizing all kinds of patterns. The human brain is fantastically adept at recognizing all kinds of patterns. (I bet you skipped at least that last sentence without even reading it.) Patterns in any medium involve some form of repetition, and have a certain aesthetic appeal. Unfortunately, with over-repetition comes boredom. Repetition provides unity and cohesiveness; variation provides interest. This is always a delicate balance!

What is interesting about these examples is how important scale is when balancing cohesiveness and interest. For example, when your brain looked at the boring pattern, it saw repeating "t"s. Boring. When it looked at the less boring pattern, it saw the same repeating "t"s; however, its interest was held by the larger grouping patterns that began to emerge. Spend some time with each of the patterns and think about the meaning of "t" in each one. I'm serious! There is a lot to be learned from these simple examples.

Boring Adaptive Poetry
Think way back to "Adaptive Word Texture" examples. Which adaptive poem would get boring faster? Most definitely it's the "Adaptive Word Texture With Meter and Rhyme" version. In that example, two patterns (meter and rhyme) repeat constantly without variation. By the second couplet, the pattern has become recognizable. By the third, it has become predictable. By the fourth, it begins to become tedious.

One might argue that the "Adaptive Word Texture" poem's structural organization is in fact more repetitious and therefore more tedious. After all, it's essentially just word, word, word, word, word… Why isn't this the case? In the first version the simplicity of the pattern works in its favor. It's unobtrusive -- it's basically a non-pattern. It doesn't distract from the interesting unpredictability of what the next word might be. By contrast, the higher level structural patterns in the second version engage the part of the brain that pays attention to higher level structural patterns -- and becomes bored with them. In this case, the (repetitious) structure dominates the (non-repetitious) content. This unbalance is due in part to the fact that our content only works at the word level, while the patterns (and the brain) are working at the phrase level.

Adaptability at the Phrase Level
The problem with word textures is that they are extremely limited in terms of actually expressing ideas. Compare "danger fear shadows lurking seal evil danger" (from "Adaptive Word Texture") to "the enemy could attack at any moment." The first "sentence" is a pseudo-random collection of symbols. The second uses syntax to combine symbols into a coherent thought. It actually says something. (And you thought syntax was only important for programmers!)

What happens when we try to write an adaptive poem that actually communicates something at the sentence (phrase) level? A logical place to start might be with a sentence texture. This will work in much the same way as the word texture example, except that we will use four lists of complete sentences.

Example: Adaptive Sentence Texture

An important thing to consider is the production cost of sentence texture compared to word texture. A word list that is twenty elements long requires only twenty words. A sentence list that is twenty elements long requires more than one hundred words! Plus, the production of a sentence involves a process of composition.

Variety at the sentence level requires a bigger budget than the same amount of variety at the word level. On the other hand, a single sentence covers more "ground" in terms of words and time. The "Adaptive Sentence Texture" example used far fewer elements than the"Adaptive Word Texture" example, but was just as long. In terms of pure variation over time, perhaps fewer sentences are required.

Another issue is that complete ideas demand more attention from the player. The danger is that instead of enhancing gameplay, we're going to start distracting the player from it. There's an old saying among film makers: "The best soundtrack is the one you never hear." (You should be feeling it instead.) The idea is that if you notice the music, it's because the composer's done something wrong. Well, the same goes for adaptive poetry. We want the player to respond on an emotional level, not an intellectual one. Complete sentences are dangerously high level.

Lastly, sentence texture is a lot less flexible than word texture. It takes longer to state an idea than it does to state a word. Our "Adaptive Word Texture" example could switch moods literally in midstream. However, when dealing with adaptive structures at the phrase level, we're suddenly faced with logistical and creative concerns about what to do when the game mood changes in mid-sentence. How do we make a smooth and effective transition?

Transitions

That was a pretty good transition, wasn't it? It was even at a higher structural level than mere sentences, or even paragraphs. I changed sections almost seamlessly! OK, not so impressive. I already knew where I was going, and timing my arrival wasn't critical. So, I was able to compose my sections so that they fit together just so.

In adaptive poetry, timing is critical. In order to enhance the mood, changes have to happen with the mood. If cues are delayed, even by a second, they aren't contributing to the emotional experience of a game event. They become their own distinct event, separate from the associated game event - which the player's brain will already have had time to process. Poetry mood changes have to happen on time!

In the "Adaptive Sentence Texture" example, this wasn't a big deal. When the seals sprung their surprise attack, it made sense to suddenly interrupt the previous mood. Ending a combat mood might be more complicated, though. Imagine that the sentence "When will the carnage end!?" is one of our combat list elements. Imagine that Captain Brave-O strikes down his last opponent midway through the playback of this particular sentence. We want to switch to victory poetry, and we don't want to be late. We also don't want to be awkward. One way to address this situation is to compose transition versions of the combat sentence. If the mood changes midway through the statement, we continue smoothly with the transition version.

Immediately, the cost of this approach should be obvious. Each element of the sentence list will require similar treatment if the transition to victory is to be possible at any time. Additionally, transitions to other moods may be necessary. ("When will Captain Brave-O be deader than this?") On the other hand, it does what it's supposed to de very well. What more could we ask for?

Going for Broke

Example: Adaptive Sentence Texture with Meter, Rhyme, and Transitions

Knowing Your Arsenal
As these examples get progressively high-level in terms of structure, I am reminded a little bit of the Seal Hunter weapon selection options. The UltraNinja2000 (a light, fast club) is perfect for lightning strike strategies, but doesn't do very much damage. The more expensive Crusher is slower and difficult to wield effectively, but makes an impressive impact when it connects! Each weapon has its place, and is useful in different game situations. In fact, choosing the right tool for the job is a big part of winning the game.

The same thing is true of the different adaptive strategies that we have examined. Each has advantages and disadvantages. There is no one perfect choice that is right for every game context. The best approach is to use all of them as necessary, applying the most effective solution on a case by case basis. "What do we want to hit the player over the head with for this part?" Choosing the right tool for the job is a big part of winning the game. It also helps to have a secret weapon handy…

Adaptability at the Letter Level
Those of you who have been paying attention to the structure if this paper will have been expecting either A) an example of adaptive poetry combining multiple approaches or B) "Adaptability at the paragraph level." Instead, in a surprise move, I am unveiling a secret weapon that's been hiding under our noses all along!

It's easy to overlook adaptability at the letter level because we don't normally think of packing much useful meaning into a single letter. If we could, it would offer incredible advantages in terms of flexibility. (Even faster than the UltraNinja2000!)

Example: Adaptive Letter Texture

Shockingly, that little example of pure adaptive letter texture was packed full of meaning! How could this be? Well, don't forget that the game is its own narrative. The player already knows what's going on. Our job as adaptive poets is to merely to enhance the mood of the story -- telling it is somewhat redundant. Still, adaptive letter texture on its own is somewhat limited in terms of what it can express.

(HEROIC) The real power of adaptive letter texture is its ability to com-(CREEPY)-bine with other techniques. It is fully com-(COMBAT)-PATIBLE WITH ALL OF THE OTHER APPROACHES THAT WE HAVE DISCUSSED. IT COULD even BE USED TO add AN ADAPTIVE element to PRE-COM-(CREEPY, INJURED)-posed material… and even com-(NEUTRAL, INJURED)-pletely unrelated prose!

Notice too that with this technique we've added an "analogue" element to our mood swings. Instead of limiting moods to either ON or OFF, BLACK or WHITE, we've added some shades of gray (and red…). For instance, the frequency of red characters (or completely red words) can indicate the status of Captain Brave-O's health meter, without interfering with the content or structure of other levels.

Seal Hunter: The Final Battle

Now that I've introduced my secret weapon, I'll leave you with one final example. The final example demonstrates how one might combine some of these approaches into a complete adaptive game poetry solution. Don't forget that the purpose of this exercise has been to learn something about adaptability in structured art forms, so that we can apply it to making interactive video game audio soundtracks. Meanwhile, think about the example that follows. What techniques are being used? Why did the game's Poetry Director make the choices he did? How much would this cost to implement and produce?

Example: Seal Hunter, Level 10

Annotated Bibliography

Hofstader, Douglas R. Gödel, Escher, Bach - 20th Anniversary Edition: An Eternal Golden Braid. New York: Basic Books, 1999.

Hofstadter, Douglas R. Le Ton beau de Marot - In Praise of the Music of Language. New York: Basic Books, 1997. Both Hofstader works explore symbol and meaning, form and content. The former has cognition and self-awareness as its central theme, while the latter is about communication and relationships. Hofstader draws deep and engaging analogies between his themes and the systems and structures that he explores (math, art, and music in the former; poetry, translation, and language in the latter). Frequently (particularly in GEB), the very structure and/or form of his writing demonstrates the content - an impressive device that adds an extra dimension to his exposition. Both of these books contributed enormously to the form, content, and even tone of my little paper.

Jourdain, Robert. Music, the Brain, and Ecstasy - How Music Captures Our Imagination. New York: Avon Books, Inc., 1998. A very broad survey of the phenomenon of music, from the physical and biological to the psychological and the sublime. Not an academic reference work, it's very readable - yet obviously exhaustively researched across many disciplines. Especially interesting to me were the explorations of how different elements of music (particularly patterns at different structural levels) are processed in different parts of a listener's brain, and the incredibly complex relationships that are involved in even the perception of music. OK, put it like that and it sounds boring, but seriously - anyone who is even remotely interested in music or audio will thoroughly enjoy this book. I'm not kidding, buy it! The takeaway is universal, despite a fairly western-classical-tradition-centric focus. Room for a sequel I guess…

Schoenberg, Arnold. Fundamentals of Musical Composition. London: Faber and Faber, Ltd., 1970. You wouldn't think that one could learn how to compose from a book, but something special clicked into place for me with this one. Partly the timing was just right - years of intensive training probably had something to do with it… Whatever it was, I really did understand music differently after reading this book. Unexpectedly (considering Schoenberg's prominent role in the deconstruction of tonality), the book limits itself strictly to traditional classical composition. It doesn't discuss or challenge tonality at all; it merely explains how and why the structures work. Despite the specific focus, there are lessons to be learned here about cohesiveness of form, repetition, and variation that can be applied to structures in any genre (…any medium?). Kind of an enigma is that the structure of the book itself is confusing at some points almost to the point of loosing the point entirely. Also, unlike the other works cited in this bibliography, it's extremely dry - unless you're really interested in classical music theory. Nonetheless, for me it was a gem. Grab it… if you can find it!

________________________________________________________