Coding for Music

Preface

Using code in games to actually create music, note by note, on the fly, is an underutilized idea that has potential to add huge value to games that aren't even focused on music, by offering an extra sense of authorship to the player's experience. From the simplest implementation, like the subtle musical sound effects in the menus of Super Mario Galaxy, to the generative music systems woven throughout Spore, the potential is broad and exciting.

Back in 2009, as an excuse to get better at coding and to make something in the process, I started exploring a game idea that involved a man walking around and licking snowflakes that made music. I created a bunch of sound assets of different tones, and based on what sounded good, I'd choose what asset to trigger next. Over time, I refined what I thought was a game into more of a music tool, adding lots of features such as chords, changing keys, different scales, and so forth. The following ideas all come from that tool, which is called 'January.' Check it out at http://january.cc.

Generating Something

So (maybe) you want to generate some music! For simplicity's sake, we'll create a program that uses the notes associated with the white keys on the piano, starting with the middle note of the keyboard, C3, and going up a single C Major Scale: C3, D3, E3, F3, G3, A3, B3, ending with the octave, C4. We'll get a series of new note choices by figuring out what the last note played was. The program then chooses one randomly, plays it, and stores it as the new last note played. This creates a process that can be repeated to randomly generate a curated musical sequence. */

// For the purposes of these examples I'm using Flixel and AS3, because it's what I'm familiar with.

/** The last note played, stored after each new note is played. */

public static var lastNotePlayed: Class;

/** Play a note! */

public static function playNote():void

{

var noteToPlay: Class = getNote();

// This is Flixel's function for playing sounds.

FlxG.play(noteToPlay);

lastNotePlayed = noteToPlay;

}

/** Get a new series of note choices, based on what the previous note was. */

private static function getNote():Class

{

var noteChoices: Array = [];

    if (lastNotePlayed == C3) noteChoices = [D3, E3, F3, G3, A3, B3, C4];

else if (lastNotePlayed == D3) noteChoices = [C3, E3, G3];

else if (lastNotePlayed == E3) noteChoices = [F3, G3, C4];

else if (lastNotePlayed == F3) noteChoices = [E3, G3];

else if (lastNotePlayed == G3) noteChoices = [C3, D3, E3, F3, A3, B3, C4];

else if (lastNotePlayed == A3) noteChoices = [G3, B3, C4];

else if (lastNotePlayed == B3) noteChoices = [G3, A3, C4];

else if (lastNotePlayed == C4) noteChoices = [C3, G3, B3];

else    noteChoices = [C3, G3, C4];

var choice: int = Math.round(Math.random() * (noteChoices.length - 1));

var note: Class = noteChoices[choice];

return note;

}

Generating Anything

The approach above works if you intend to stay in one scale, playing over one chord, but what if you want to have the generated notes adapt to different chords, scales, and eventually keys? To accomplish these things, a more sophisticated approach will be necessary. Storing your note choices as strings instead of using the sound classes themselves is one way to do this that will work. By separating out the logic choices from the actual sounds, it will be easier to adapt your system to new musical contexts.

A musician playing along to different chords in a single key is more often than not utilizing the various 'musical modes' of that key. A brief synopsis of what these modes are may be necessary if you're not familiar with the concept. The key of C Major is comprised of 7 modes, and they all use the white keys on a piano. Each mode is essentially it's own 7-note scale starting from a different note of the C Major Scale (C, D, E, etc.). By starting the scale from a different place, you get a different sound, and each of these are called modes. If you start on C, you're in Ionian Mode: C D E F G A B. If you start on D, however, you're still in the key of C, but you're in Dorian Mode (D Dorian): D E F G A B C. This concept continues on up the C Major Scale for E (Phrygian), F (Lydian), G (Mixolydian), A (Aeolian aka Natural Minor) and B (Locrian).

In order to make different musical choices based on what mode we're currently using, we'll need to come up with a modular system. Directly referencing the sound classes is no longer the best option, because the various modes, while possessing the same notes, tend to sound best when they use the notes in their own unique ways. A 'C' note is going to sound quite different depending on whether you're playing it over a C chord or a D chord. In order to accommodate this fact, the code below creates a unique object variable for each mode. We store that mode's position in the scale (ie. Ionian is the 1st mode, Dorian the 2nd, etc) and use a 2D array of strings called 'logic' to represent our potential note choices as scale degrees. A scale degree is the numerical representation of a note in the scale. For instance, in C Ionian, the 4th degree is F. They let us talk about a note in a musical key or scale without saying what the note actually is (ie. C, D, etc). Here is what our old C Major Scale would look like in this new form (Ionian), plus a new mode, Dorian:

public static const IONIAN: Object = { position: 1 };

IONIAN.logic = [ /* 00 one  */ ['two', 'thr', 'for', 'fiv', 'six', 'sev', 'one'],

/* 01 two  */ ['one', 'thr', 'fiv'],

/* 02 thr  */ ['for', 'fiv', 'oct'],

/* 03 for  */ ['thr', 'fiv'],

/* 04 fiv  */ ['one', 'two', 'thr', 'for', 'six', 'sev', 'oct'],

/* 05 six  */ ['fiv', 'sev', 'oct'],

/* 06 sev  */ ['fiv', 'six', 'oct'],

/* 07 oct  */ ['one', 'fiv', 'sev'],

/* 08 else */ ['one', 'fiv', 'oct']

];

public static const DORIAN: Object = { position: 2 };

DORIAN.logic = [ /* 00 one  */ ['thr', 'fiv', 'sev', 'oct'],

/* 01 two  */ ['fiv', 'six', 'sev'],

/* 02 thr  */ ['one', 'for', 'fiv', 'six', 'sev', 'oct'],

/* 03 for  */ ['fiv', 'six', 'sev', 'oct'],

/* 04 fiv  */ ['thr', 'six', 'sev', 'oct'],

/* 05 six  */ ['one', 'for', 'fiv', 'sev', 'oct'],

/* 06 sev  */ ['thr', 'fiv', 'six', 'oct'],

/* 07 oct  */ ['one', 'thr', 'for', 'fiv', 'sev'],

/* 08 else */ ['one', 'fiv', 'oct']

];

This is all fine and dandy, but these strings need to be hooked up to the sound assets in order to work. As a way of dealing with different modes, we can create a for loop that will fill a 'loadout' object with the proper notes, assigned to the proper scale degrees.

/** An array of the scale degrees, used as a reference for populating the loadout object. */ 

public static const DEGREES: Array = ['one', 'two', 'thr', 'for', 'fiv', 'six', 'sev', 'oct'];

/** An array of all the notes needed to potentially form an eight note scale in any mode (Ionian is C3 - C4, Dorian is D3 - D4, Phrygian's E3 - E4, etc.) */ 

public static const C_MAJOR: Array = [C3, D3, E3, F3, G3, A3, B3, C4, D4, E4, F4, G4, A4, B4];

/** The loadout object, to be filled with the notes of the current mode/key. */

public static var loadout: Object = {};

/** The last note played, stored after each new note is played. */

public static var lastNotePlayed: Class;

// For the sake of the example, let's say the current key is F# Major.

public static varcurrentMode: Object = DORIAN;

/** Play a note! */

public static function playNote():void

{

var noteToPlay: Class = getNote(currentMode); // Pass the current mode into getNote(), whatever mode you want it to be!

FlxG.play(noteToPlay);

lastNotePlayed = noteToPlay;

}

/** Get a new series of note choices, based on what the previous note was. */

private static function getNote(modeToUse: Object):Class

{

/** Whether we've found the last note played in the current loadout. */

var found: Boolean = false;

/** The note choices that we'll use to decide the next note to get. */

var noteChoices: Array = [];

/** Make sure that the current scale we're using to make logic decisions starts on the first note of the current mode. 

ie. in Ionian, loadout['one'] = C3, loadout['two'] = D3, etc.

in Dorian, loadout['one'] = D3, loadout['two'] = E3, etc. */

for (var i:int = 0; i <= DEGREES.length - 1; i++)

loadout[DEGREES[i]] = C_MAJOR[i + modeToUse.position - 1]; // - 1 is to account for starting at Zero.

/** Iterate through the loadout, find which is the last note played, then store the associated set of choices to noteChoices. */

loop: for (var j: int = 0; j < DEGREES.length - 1; j++)

{

if (lastNotePlayed == loadout[DEGREES[j]])

{  

noteChoices = modeToUse.logic[j];

found = true;

break loop;

}

}

/** If the last note played was not found in the loadout, use the 'else' choices from the logic array. */

if (found == false)

noteChoices = modeToUse.logic[08]; // [08], or the last set of arguments in the logic array, used like an else statement.

/** The numerical choice from the array of choices that we will use. */

var choice: int = Math.round(Math.random() * (noteChoices.length - 1));

/** The note that we will return to the playNote() function and eventually play. */

var note: Class = loadout[noteChoices[choice]] as Class;

return note;

}

The system above will work great for moving through various modes in a single key, but what if you want to move through the other 11 keys, too? The most straightforward solution would be to create arrays for all of the keys. You also would of course need to create assets for all the new notes that you are using.

// Arrays of all the notes in needed to potentially form an eight note scale in any mode, in any key. s = #, ie. Cs3 = C#3 

public static const C_MAJOR : Array = [C3, D3, E3, F3, G3, A3, B3, C4, D4, E4, F4, G4, A4, B4];

public static const Cs_MAJOR: Array = [Cs3, Ds3, F3, Fs3, Gs3, As3, C4, Cs4, Ds4, F4, Fs4, Gs4, As4, C5];

public static const D_MAJOR : Array = [D3, E3, Fs3, G3, A3, Cs4, D4, E4, Fs4, G4, A4, Cs5];

public static const Ds_MAJOR: Array = [Ds3, F3, G3, Gs3, As3, D4, Ds4, F4, G4, Gs4, As4, D5];

public static const E_MAJOR : Array = [E3, Fs3, Gs3, A3, B3, Ds4, E4, Fs4, Gs4, A4, B4, Ds5];

public static const F_MAJOR : Array = [F3, G3, A3, As3, C4, E4, F4, G4, A4, As4, C5, E5];

public static const Fs_MAJOR: Array = [Fs3, Gs3, As3, B3, Cs4, F4, Fs4, Gs4, As4, B4, Cs5, F5];

public static const G_MAJOR : Array = [G3, A3, B3, C4, D4, F4, G4, A4, B4, C5, D5, F5];

public static const Gs_MAJOR: Array = [Gs3, As3, C4, Cs4, Ds4, Fs4, Gs4, As4, C5, Cs5, Ds5, Fs5];

public static const A_MAJOR : Array = [A3, B3, Cs4, D4, E4, G4, A4, B4, Cs5, D5, E5, G5];

public static const As_MAJOR: Array = [As3, C4, D4, Ds4, F4, Gs4, As4, C5, D5, Ds5, F5, Gs5];

public static const B_MAJOR : Array = [B3, Cs4, Ds4, E4, Fs4, A4, B4, Cs5, Ds5, E5, Fs5, A5];

// For the sake of the example, let's say the current key is F# Major.

public static var currentKey: Array = Fs_MAJOR;

Then, you would adjust the for loop that fills the loadout to check for the current key's scale:

for (var i:int = 0; i <= DEGREES.length - 1; i++)

loadout[DEGREES[i]] = currentKey[i + modeToUse.position - 1];

And voila! With these systems in place, you now have access to all 7 modes in all 12 keys. That's 84 different scales! Other applications, such as chords, can be generated in a similar fashion, using 2D arrays to store chord choices and the loadout system as before to make sure you're in the right key and the right mode. You could also use a timer to fan out the playback of the chord notes, if you want to be fancy. The possibilities are endless!

Considerations

If you start to experiment and your generative music system gets more complex, you will probably run into some 'musical bugs'. Namely, you will find dissonances (notes that don't sound good together) and other unwanted events creeping up from time to time. You'll want to consider preventing certain sequences from happening, such as repeated notes, 'trills' (hearing the same note twice with another note in between), and if you start changing keys and scales on the fly, you may want to limit what kind of notes play around the time that those events happen. These kinds of problems can often be solved with conditional checks, often against the lastNotePlayed.  If you want to check for trills, you could create a second variable, secondToLastNotePlayed, and continuously log the last two notes. This could also open the possibility space for what kind of choices you make about which notes come next, if you're up to the challenge.

At the end of the day, the most sophisticated music generating code won't amount to much if you don't have someone musical onboard, using their ears, making decisions about what sounds good. Maybe that's you, or maybe not, but hopefully either way this will be a decent springboard for thinking about using code to generate music, instead of the other way around.