Interactive Music Sequencer Design

Interactive music sequencer design for games is a real mouthful -- or rather an earful -- and a pretty large topic to boot. I'll begin with a summary of the motivations for making interactive music and then get in to the design discussion. To make the design discussion a bit more manageable, I will assume a lot of familiarity with music and synthesizer details and put greater focus on design options and programming methods. First, I will talk about the design issues in making a computer music language. Second, I will consider the controls available with audio synthesis. Third, I will add the issues related to providing interactive music functions for game control. Throughout, I will focus on concepts that directly influence interactivity, and point out approaches that make the complexity more manageable and help to make implementation practical.

Motivations for Interactive Music

Most forms of electronic entertainment include music. Games, movies, and TV shows are seen as incomplete without music. We hear music at the introduction to news programs, on our mobile phone rings, in department stores, and at coffee shops.

Music is its own form of entertainment. We listen for styles, attitudes, technology, improvisation, composition, and skilled performances. Our memories associate music with past situations, friends, places. We associate music with love and hate.

Games include music for many reasons: To identify with a particular audience. To establish attitude, tension, and mood. To let them hear the pride and glory of success or shame and ridicule of defeat. To march them off to the drums of war. To take them to a magical place. To set them in the past. To set them in the future. To take them to alternate worlds. To bring them back to reality. Music instantly adds definitions and associations beyond what the visuals can do.

Games are interactive. This means a player has control over the game in some way and the game asks the player to interact in some way. This control and interaction are the basis for how a game becomes immersive and entertaining. The quality of control and interaction are the basis of successful games.

Therefore, it is natural to want to mix the immersive quality of control and interaction in computer games with the immersive qualities of music. How do we control music? How can we create musical interaction? This is the motivation for this article.

The reasons for developing your own interactive music sequencer code are the same as the reasons for any code development. You may want standard features across many platforms. There may not be systems available that meet your needs. You may want an implementation that you can optimize for delivering the particular features you need. You may want control over your own code to provide improvements, enhancements, and reliability in line with internal scheduling requirements.

Making a Computer Music Language

Event Blocks
Music can be described as events or commands occurring over time. We can build our music command language with "event blocks" composed of three elements: time, event type, and event details. The time is stored relative to previous events and is called a delta-time, the event type is identified with a number, and the event details are zero or more parameters that are defined by the event type.

Figure 1 - Event Block

With this basic design we can describe any groupings of events and time intervals. Building our computer music language is now a task of choosing how to store the delta-time, what event types, and what details are needed for each event type.

MIDI (Musical Instrument Digital Interface) Music
The MIDI specification has been around for a long time. Most composition and sequencing software provides compatibility with MIDI input and output ports, and these products also provide options to save and load MIDI files.

Since the MIDI specification defines a delta-time storage format and a useful set of event types and parameters, we should start with this model as a reference. This will make it easier to convert the MIDI language to our custom music language.

Keep in mind that our goal is to create music directly on a machine with audio capabilities, so we want to take what is useful from the MIDI design, but not restrict ourselves to its features. We are also likely to get music from composers in the MIDI file format, so we will want to convert the MIDI file music language to our own custom language.

MIDI Events
The MIDI specification defines several events called the channel voice messages, and there are also meta-events defined in the MIDI file format 1.0 specification. Some of the meta-events simply contain text string data. I summarize the MIDI events and meta-events we are interested in Table 1, Table 2, and Table 3.

Table 1 - MIDI Channel Voice Message

Table 2 - Basic MIDI Event Types

Table 3 - Text MIDI Event Types

Of the MIDI channel voice messages, the control change event is unique in that it specifies an additional event identifier, the controller ID. Examples of common controller ID numbers are 7, which refers to volume and 10, which refers to pan position. Many controller ID numbers are not commonly used and we can have musicians insert control change messages with controller IDs to mark the music in special ways.

The basic meta-events listed give us ways to get time organization details and tempo changes. The text meta-events give us ways to get custom data embedded in strings. Composers can put special track settings in the track name strings or special kinds of playback commands in lyric strings. These methods will save us from having to write custom tools for composers when deadlines are looming.

MIDI Channel and Tracks
The "channel" in MIDI channel voice messages refers to the fact that these messages are sent with a channel number from 1 to 16 (0 to 15). The channel number is used by MIDI devices to route commands. We are not routing commands to MIDI gear, so we won't talk about channels anymore. Type 1 MIDI files have any number of tracks where each track contains a series of event blocks. It is the track concept that is useful for our data organization.

Sequences, Tracks, Events, Instruments, and Voices
This is where we leave the world of MIDI and talk about our custom music sequencer language. A quick summary of the terminology used in the rest of this paper is in order: A Sequence is a collection of Tracks that run simultaneously. Each Track is a sequence of Events that control the current Instrument. Certain types of Track Events will turn on and off Voices of the current Instrument. Some types of Track Events will modify all active Track Voices. You can think of an Instrument as the default settings given to a Voice that is turned on with a NoteOn Event. I will present a list of class definitions that display this structure in the implementation section.

Event Type Possibilities
Table 4 lists many event type ideas so you can get a taste of the possibilities. There are a lot of details that could be discussed ast to how these event types might be implemented, but I will just mention the options here and move on to the other parts of this discussion.

Table 4 - Event Type Ideas

Sequencer Data Structures
The main data structures that we could use for our music sequencer are listed in Code Example 1.

Code Example 1 - Music Sequencer Data Structures

typedef list< Sequence * >  SequencePtrList_t;
typedef list< Track * >        TrackPtrList_t;
typedef list< Voice * >        VoicePtrList_t;

class MusicSequencer_t {
     MusicSequencerState_t  State;
     SequencePtrList_t           ActiveSequencePtrList;
     SequencePtrList_t           FreeSequencePtrList;
     TrackPtrList_t                  ActiveTrackPtrList;
     TrackPtrList_t                  FreeTrackPtrList;
     VoicePtrList_t                  ActiveVoicePtrList;
     VoicePtrList_t                  FreeVoicePtrList;
};

class SequenceState_t {
     Tempo_t    Tempo;
     Volume_t   Volume;
};

class Sequence_t {
     SequenceState_t State;
     SequenceState_t BeginState;               // Interactive feature
     SequenceState_t EndState;                  // Interactive feature
     SequenceInterpolator_t Interpolator; // Interactive feature
     TimeUnit_t         TimeElapsed;
     TimeUnit_t         TimeStep;
     CallbackFunc_t *pCallback;                 // Interactive feature
     TrackPtrList_t    TrackPtrList;
};

class TrackState_t {
     Volume_t     Volume;
     PitchBend_t PitchBend;
     Pan_t         Pan;
     Effect_t      Effect;
};

class Track_t {
     TrackState_t      State;
     TrackState_t      BeginState;               // Interactive feature
     TrackState_t      EndState;                  // Interactive feature
     TrackInterpolator_t Interpolator;      // Interactive feature
     Sequence          *pOwner;
     char                    *pEvent;
     Instrument_t    *pInstrument;
     VoicePtrList_t   VoicePtrList;
};

class VoiceState_t {
      SynthVolume_t   Volume;
      SynthPitch_t        Pitch;
      SynthPan_t          Pan;
      SynthEffect_t      Effect;
};

class Voice_t {
     VoiceState_t                CurrentState;
     VoiceState_t                BeginState;     // Interactive feature
     VoiceState_t                EndState;        // Interactive feature
     VoiceInterpolator_t    Interpolator;   // Interactive feature
     Track_t                        *pOwner;
     char                               nKey;
};

Event Data Structures
To implement the event type commands we can have the event type command numbers correspond to an array lookup that holds the relevant function pointer and the byte length of the event type and parameters. Code Example 2 shows this code.

The function pointers give us a quick way to get to the code associated with each event type. The byte lengths give us a quick way to step to the next event block.

Code Example 2: Event Type Data Structures

// Example Note Off Event Block
typedef struct {
     char nEventType;
     char nKey;
     // no release velocity
}NoteOff_EventBlock_t;

void NoteOff_Function( Track_t *pTrack )
{
      // the pEvent is pointing at our event block
     NoteOff_EventBlock_t *pNoteOffEB = (NoteOff_EventBlock_t *)pEvent;

     // walk through this track's voices and turn off
     // any that have pVoice->nKey == pNoteOffEB->nKey
}

     // Example Note On Event Block
     typedef struct {
     char    nEventType;
     char    nKey;
     char    nVelocity;
}NoteOn_EventBlock_t;

void NoteOn_Function( Track_t *pTrack )
{
     // the pEvent is pointing at our event block
     NoteOn_EventBlock_t *pNoteOnEB = (NoteOn_EventBlock_t *)pEvent;

     // try to get a voice from the free list or
     // try to get a voice from the active list if possible
     // if we have a voice, turn it on with the pNoteOnEB->nKey
     // and pNoteOnEB->nVelocity and other state information
}

enum enumEventType
{
     EVENT_TYPE_NOTEOFF,
     EVENT_TYPE_NOTEON,
     .
     .
     .
     EVENT_TYPE_COUNT
};

typedef void (*EventFuncPtr_t)(Track_t *);

typedef struct {
     EventFuncPtr_t pFunc; // pointer to command function
     int nLength; // byte length of command
}EventTypes_t;

static EventTypes_t aET[EVENT_TYPE_COUNT] = {
     { NoteOff_Function, sizeof(NoteOff_EventBlock_t) },
     { NoteOn_Function, sizeof(NoteOn_EventBlock_t) },
     .
     .
     .
};

Timing
Different computer systems will have different ways of providing timing callbacks or threads that wake up at specific intervals. I will simply assume that we can have a function called at a specific interval that I will call the "audio frame callback". We can think of the time between the callbacks as the audio frame. During each callback we need to update our notion of how much time has passed and we need to send out all of the commands that have "timed out".

In the sequence data structure listed in Code Example 1 there is the TimeStep quantity that should be set based on the tempo, the delta-time parts per quarter note, and callback timing. We add the TimeStep to the TimeElapsed on each audio frame to keep track of the time. Since these time parameters are in the sequence structure, we can only change the tempo for the whole sequence. If we wanted to change tempo for each track individually we could put these parameters and the tempo setting in the track structure.

Audio Synthesis Control

Control
Certain interactive music effects are possible if sequencer commands can be tied to audio synthesis parameters. The SetTrackEffect event type could represent any effect parameter that the audio synthesis system provides. Some audio synthesis parameters are simply set before a note is played and may not respond to updates until the next note is played. Some audio synthesis parameters can be altered during notes.

The behavior of the audio synthesis system that we are using will determine what kind of control we can have, and music will have to be written with the control issues in mind.

By defining immediate or target state changes for sequences, tracks, or voices we can manage what could be many controls changes with simple ChangeState commands.

Connecting Synth to Sequencer
The key to connecting our audio synthesis to our music sequencer is the SetInstrument event type. This command looks up in to a table of instrument definitions and sets the pInstrument field of our Track data structure. When a NoteOn command occurs the parameters from the pInstrument are transferred to the voice to be started.

We will need to manage the available audio synthesis voices to provide good voice stealing logic. When we have used all of the available voices we may want to end an active voice because a new note on request is evaluated to have a higher priority. The priority system could use several kinds of weights. These weights could be based on the note ontime, time until note end, envelope stage, volume levels, count of this type of voice active, instrument priority number, track priority number, and sequence priority number.

Interactive Music

Interpolation
Game programmers are very familiar with interpolation. I mention it here because our ability to interpolate between sequence states, track states, and voice states provides us with interactive control of music.

Transitions
Transitions can be defined as one or more changes occurring over a time interval. A transition might mean an interpolation of some kind of state data done over a specific time interval. Or a transition might mean a new track section, musical cadence, key change or other compositional technique. A transition could be a combination of these things.

Transitions may be triggered by a combination of game logic and music language logic. It is useful to provide API functions for game code to directly set target states. These kinds of implicit and explicit controls over transitions are another key element of the interactive control of music.

Meanings
Some of the meanings that we might want to attach to the control of music are categorized in Table 5.

Table 5 - Music Meanings

Transition Types
Some of the many transition types are mentioned in Table 6.

Table 6 - Music Transition Types

Design Influences
There are four important factors in the discussion of interactive music: game design, game programming, music design, music programming. Music programming is influenced by the other factors in the following ways:

To point out these influences I will present some hypothetical examples.

Design Example #1

Game Design: Through player skill, a character can achieve a powered-up state. This state can last a very long time and a sound effect might get monotonous. We want to hear his energy in the music.

Music Design: Transition the melody and percussion track instruments to add DSP effects that add color and depth to the instruments.

Programming Design: Two sequence states are created and the game can choose when to set each target state.

Design Example #2

Game Design: When our player goes near a location in a level we want to hint that there is danger using the music.

Music Design: Fade down the main melody track and fade up the danger melody track.

Programming Design: Based on distance from the location, set the track target states for volume.

Design Example #3

Game Design: Lets say we have a game design where we change from day to night. Lets say that the player's role is more offensive in the day and more defensive at night. We want energy music during the day and tense and scary music at night.

Music Design: To keep it simple, we will describe three tracks of the music: melody, accompaniment, and percussion. We will define "energy", "mellow", and "creepy" versions of each of the three tracks. Again keeping it simple, we will define a "hard" and "soft" version of each of the instruments for each track.

Programming Design: We generate our control value based on the game time. This control value is used to interpolate the instrument states and initiate transitions for the track states.
So when our game time reaches 3pm, the melody instrument transitions to the "soft" state. When our game time reaches 6pm, the "energy" melody track fades down and the "mellow" track fades up.

Conclusions

This article covered many motivations and implementation ideas for interactive music, and presented some core programming concepts that can provide the flexibility and control needed for interactive music while also pointing out the design influences, meanings, and transition types. The next thing to do with all of these ideas is implement your own interactive music sequencer and make sure interactive music is part of your game's design.

________________________________________________________