Introduction

So, you want to make a rhythm game or you tried to make one, but the game elemtns and the music quickly became out of sync, and now you're not sure what to do. You've come to the right place. I've been playing rhythm games since high school, when I frequented the DDR machine at my local arcade. Today I'm always on the lookout for new takes on the genre, and with entries like Crypt of the Necrodancer or Bit.Trip.Runner, there's still a lot that can be done in rhythm gaming. I put some work into a few rhythm-based prototypes in Unity, ultimately devoting a month to creating Atomic Beats, a short rhythm/puzzle game. In this article I'll cover a few of the most useful coding techniques I learned in creating these games that were either not covered somewhere else, or I thought could be covered in more detail.

Firstly, I owe a huge debt of gratitude to Yu Chao's blog post 'Music Syncing in Rhythm Games'. Yu covered the core of syncing audio timing to the game engine in Unity, and made the source code for Boots-Cut available, which helped immensely in getting this project off the ground. You can take a look at his post for a quick guide to music syncing in Unity, but I'll be going over the core of what he outlines and more. My code derives heavily from both his article and Boots-Cut.

At the core of any rhythm game is timing. People are extremely sensitive to any deviations in rhythmic timing, so it's extremely important to make sure that any actions, movement, or input in a rhythm game is directly synced to the music. Unfortunately the traditional methods for tracking time in Unity, like Time.timeSinceLevelLoad and Time.time will quickly lose sync with any audio playing. Instead we'll access time according to the audio system using AudioSettings.dspTime, which relies on the actual number of samples the audio system has processed, and therefore will always remain in sync with the audio as it plays (This may not be the case with extremely long audio files as some sampling effects may come into play, but for normal length applications, it should work perfectly). This function will be the core of how we track song time, and we'll build our main class around it.

The Conductor class

The Conductor class is the main song managing class that the rest of our rhythm game will be built on. With it, we'll track the song position, and control any other synced actions. To track the song, we'll need a few variables:

//Song beats per minute
//This is determined by the song you're trying to sync up to
public float songBpm;

//The number of seconds for each song beat
public float secPerBeat;

//Current song position, in seconds
public float songPosition;

//Current song position, in beats
public float songPositionInBeats;

//How many seconds have passed since the song started
public float dspSongTime;

//an AudioSource attached to this GameObject that will play the music.
public AudioSource musicSource;

When the scene starts, we need to do a few calculations to determine some of the variables, and also record, for reference, the time when the audio began.

void Start()
{
    //Load the AudioSource attached to the Conductor GameObject
    musicSource = GetComponent<AudioSource>();

    //Calculate the number of seconds in each beat
    secPerBeat = 60f / songBpm;

    //Record the time when the music starts
    dspSongTime = (float)AudioSettings.dspTime;

    //Start the music
    musicSource.Play();
}

If you create an empty GameObject with this script attached, and add an Audio Source with a song and start the program, you can see the script will update with the time when the song started, but not much else will happen. You'll also need to manually enter the BPM of the music you're adding to the Audio Source.

With these values, we can now track the location of the song in real time as the game updates. We'll determine the song timing, first in seconds, then in beats. Beats is a significantly easier way to track the song as it let's us add actions and timing in time with the song, say on beats 1, 3, and 5.5, without having to calculate the seconds between beats each time. Add the following calculations to the Update() function of the Conductor:

void Update()
{
    //determine how many seconds since the song started
    songPosition = (float)(AudioSettings.dspTime - dspSongTime);

    //determine how many beats since the song started
    songPositionInBeats = songPosition / secPerBeat;
}

this gives us the difference between the current time, according to the audio system, and the time when the song started, which amounts to the total number of seconds the song has been playing, which we store in the variable songPosition.

Note that while music counting typically starts at 1, with a beat of 1-2-3-4-etc., songPositionInBeats begins at 0 and increases from there, so the third beat of a song will occur when songPositionInBeats is equal to 2.0, not 3.0.

At this point, if you wanted to make a traditional Dance Dance Revolution style game, you could spawn notes according to the beat you wanted them to be pressed, interpolating  their position towards a trigger line, then record the songPositionInBeats when a key is pressed and compare it to the intended beat of that note.Yu Chao goes through an example of to set that up in his blog here. Instead of repeating that though, I'll cover a few other potentially useful techniques that can be built on top of the conductor class that I used in building Atomic Beats.

Adjusting for Starting Beat

If you're creating your own music for your rhythm game, it's easy enough to make sure that the first beat starts exactly when the music begins, which will make the Conductor's songPositionInBeats align correctly to the song, as long as the BPM is entered correctly.

However, if you're using pre-existing music, there's a good chance that there's a small period of silence before the song starts. Without accounting for this, Conductor's songPositionInBeats will think the first beat occurred when the clip began playing rather when the first beat occurs. Anything you have aligned to subsequent beat numbers will no longer be synced to the music as the game plays.

To fix this, we can add in a variable to account for the offset. In the Conductor class, add the following:

//The offset to the first beat of the song in seconds
public float firstBeatOffset;

In Update(), songPosition changes from:

songPosition = (float)(AudioSettings.dspTime - dspSongTime);

to
songPosition = (float)(AudioSettings.dspTime - dspSongTime - firstBeatOffset);

Now songPosition will calculate the song position correctly relative to when the first beat occurs. You will have to manually enter the offset to the first beat however, as this will be unique to each song file. There is also a short window during that offset where songPosition will be negative. This may not affect your game, but some code that relies on songPosition or songPositionInBeatss may not be able to process negative numbers during this time.

Looping

If you have a song that runs from start to finish, just using the above Conductor class is sufficient for tracking how far through a song you are, but if you have a short track that loops around and you want to work within that loop, it's necessary to build loops into your conductor.

With a perfectly looping clip (for instance if the beat is 120bpm, and the clip you want to loop is 4 beats long, it will have to be exactly 8.0 seconds long at 2.0 seconds per beat) loaded into the Conductor Audio Source, check the loop box. Conductor will work the same way as before, providing the total time since the looping clip was first started as the songPosition. To determine the position in the loop, we'll have to provide a way for Conductor to know how many beats are in one loop, and how many loops have been completed. Add the following variables to the Conductor class:

//the number of beats in each loop
public float beatsPerLoop;

//the total number of loops completed since the looping clip first started
public int completedLoops = 0;

//The current position of the song within the loop in beats.
public float loopPositionInBeats;

Now every time the SongPositionInBeats is updated, we can also update the loop position in Update()

//calculate the loop position
    if (songPositionInBeats >= (completedLoops + 1) * beatsPerLoop)
        completedLoops++;
    loopPositionInBeats = songPositionInBeats - completedLoops * beatsPerLoop;

This will give you a marker for how many beats through the loop you are with loopPositionInBeats, which will be useful for a lot of other synced items. Don't forget to enter the number of beats per loop on the Conductor GameObject.

Something else to be careful with here, again, is the counting of beats. Music always begins at 1, so a 4 beat measure goes 1-2-3-4-, while in this class, loopPositionInBeats startss at 0.0 and loops at 4.0. As a result of this, the exact middle of a loop, which would be 3 when counting by music beats, would occur at a loopPositionInBeats value of 2.0. It's possible to modify loopPositionInBeats to account for this, but will carry through to all other calculations, so be careful how you insert notes.

It will also be useful to add two more things to the Conductor class for the remaining tools. The first is an analog version of LoopPositionInBeats, called LoopPositionInAnalog, which measures the location in the loop between 0 and 1.0. The second is an instance of the Conductor class, so it can easily be called from other classes. Add the following variables to the Conductor class:

//The current relative position of the song within the loop measured between 0 and 1.
public float loopPositionInAnalog;

//Conductor instance
public static Conductor instance;

In the Awake() function add:

void Awake()
{
    instance = this;
}

and in the Update() function add:

loopPositionInAnalog = loopPositionInBeats / beatsPerLoop;

Synced rotation

It can be really useful to have movement or rotation that is synced to the beat in order to ensure elements are in the correct location. In Atomic beats I used this to dynamically rotate the notes around a central axis. They were initially placed around a circle according to their beat within the loop, and then the entire playing area is rotated so that notes would align with the trigger line at their intended beat.

To achieve this, create a new script called SyncedRotation and attach it to a GameObject you want to rotate. In the Update() function of SyncedRotation add:

void Update()
{
    this.gameObject.transform.rotation = Quaternion.Euler(0, 0, Mathf.Lerp(0, 360, Conductor.instance.loopPositionInAnalog));
}

This will interpolate the rotation of the GameObject this game is attached to between 0 and 360 degrees, rotating it so it completes one full rotation at the end of every loop. This is useful as an example, but syncing animation loops can be more useful for any cyclical movement or frame animation that you want to remain perfectly aligned to the beat.

Synced animation

The Animator feature in Unity is extremely powerful, but not always precise. In the case of lining up animations to music reliably, I struggled with the Animator class and its tendency to drift out of sync from the beat over time. It's also difficult to adjust the same animations for different tempos as you change between songs without having to redefine the animation keyframes for that tempo. Instead, we can directly access the animation loop we want to target and set the location within that loop according to where we are in the Conductor loop. 

Firstly, create a new class called SyncedAnimation, and include the following variables:

//The animator controller attached to this GameObject
public Animator animator;

//Records the animation state or animation that the Animator is currently in
public AnimatorStateInfo animatorStateInfo;

//Used to address the current state within the Animator using the Play() function
public int currentState;

Attach it to a new or existing GameObject that you want to animate. For this example, we'll just move an object back and forth on the screen, but this could be applied to any animation, whether it was adjusting a property, or a frame-by-frame animation. Add an Animator to the GameObject, and create a new Animator Controller called SyncedAnimController, and an Animation Clip called BackAndForth. Load the controller into the Animator class attached to the GameObject, and add the Animation to the Animator tree as the default animation.

For example purposes, I set the animation to move the object first right by 6 units, then left to -6, then back to 0.

Now to sync the animation, add the following code to the Start() function of SyncedAnimation to initialize the information about the Animator:

void Start()
{
    //Load the animator attached to this object
    animator = GetComponent<Animator>();

    //Get the info about the current animator state
    animatorStateInfo = animator.GetCurrentAnimatorStateInfo(0);

    //Convert the current state name to an integer hash for identification
    currentState = animatorStateInfo.fullPathHash;
}

Then add the following code to Update() to set the animation:
void Update()
{
    //Start playing the current animation from wherever the current conductor loop is
    animator.Play(currentState, -1, (Conductor.instance.loopPositionInAnalog));
    //Set the speed to 0 so it will only change frames when you next update it
    animator.speed = 0;
}

This will position the animation at the exact frame of the animation relative to one complete loop. So for example, using the animation above, if you are half way through the loop, the position of the GameObject will be just passing 0. This can be applied to any animation that you create that you'd like to sync to the Conductor beat.

It's worth noting that in order to create a seamless loop of some animations, it's necessary to adjust the tangents of the inividual animation keyframes on the animation curve. Linear will get you a straight line coming out of the keyframe towards the next keyframe, while constant will keep the animation at that value until the next keyframe, giving it a blocky sudden movement.

While useful, this method will also interfere with any animation transitions as it forces the animationState to remain at whatever state it was in when the script was initially run. It's useful for objects that only need to use one synced Animation forever, but to do more complicated objects with different synced animations, you will need to include code that handles those transitions and adjusts the currentState variable to match the intended animation state.

Conclusion

Those are just a few of the items that I found useful for making Atomic Beats. Some were cobbled together from other sources, or created out of necessity, but most of it was not readily available in one place on the internet that I could find, hopefully you find some of it useful! Some of them may cease to be useful in larger projects because of constraints on CPU or the audio system, but they should work as a good start for any game jam game or hobby project.

Making a rhythm game, or game elements synced to the music can be difficult and require some tricky coding in order to make sure everything is following the same beat, but the payoff of being to play along at a steady beat is huge, and can really pull in a player. There's a lot more that can be done in the space than just the traditional Dance Dance Revolution style game, and hopefully this article helps you along the way to making some of them. If you have a chance, please also check out Atomic Beats. I made it over one month in Spring 2019, and it includes 8 short songs and is free, so take a look!