Rockstar's quest for the ultimate video game horse

'AI and Games' is a crowdfunded YouTube series that explores research and applications of artificial intelligence in video games. You can support this work by visiting my Patreon page.

The horse is a staple of many an action-adventure videogame, whether you're cantering through historic streets in Assassin's Creed, hunting your prey in The Witcher or drinking in the Hyrule plains in The Legend of Zelda, your noble steed will help you make your way across vast open landscapes; traversing terrain in a fraction of the time than you would while on foot.

But while they've become something of a commodity, building a good videogame horse is incredibly difficult. Horses are not cars - I mean I think that should be obvious - but they also don't move like regular vehicles. It's not just forward motion and steering, their footwork and lateral movement have a huge impact on how horses traverse a variety of conditions. But more critically, horses are living creatures and even videogame horses should move and behave in realistic ways. As players, we strive not just for a videogame horse that is fast and responsive but feels like a character in and of itself.

Today, we're taking a look at the horses in Red Dead Redemption 2 and the efforts made by developers Rockstar to make these creatures feel realistic. Ensuring they react to the world around them, the terrain they're riding across and the perils they face: such as clifftops, obstacles and even those idiot humans trying to ride them directly into trees.

The Horses of Red Dead Redemption 2

In Red Dead Redemption 2, players assume the role of Arthur Morgan: a member of the Var der Linde gang who are ramping up for one last heist before setting off into the sunset. Players can explore the highs and lows of approximately 29 square miles of the old west, and as such, your horse is critical to helping you navigate and survive the harsh terrain.

There are 19 breeds of horse in Red Dead Redemption 2, all of which have unique characteristics and different personalities. This means that the programmers, artists and animators involved had a big task ahead of them in building realistic companions for the player to ride - not just in how they move around the world, but how they react to terrain, as well as a lot of little behavioural quirks that add that attention to detail fans of Rockstar have come to expect. As detailed in a 2021 GDC talk by Lead AI and Gameplay Programmer Tobias Kleanthous, the goal for Red Dead Redemption 2 was for players to build emotional attachments to their horses. The functionality combined with bespoke behavioural tweaks would make them stand out as the true supporting roles of the game.

Now, this isn't to say that the horses built in the original Red Dead Redemption weren't great, but the development team reviewed a lot of that original functionality, plus their first pass at updating it for the sequel, and realised it wasn't going to suit their needs. The horses actually had fairly limited and stiff movement, their movement speeds were quite rigid and often the animations didn't quite line up as desired. Horses felt more like cars with legs, in much the same way that they feel in other games and that wasn't going to be enough. They needed to be more flexible to changes in the world and have greater freedom of movement. This is evident right from the start of the game, given there are many sequences in the opening chapter with the Van Der Linde gang riding together. Without this more flexible system, this would become a chaotic mess, and deny the cinematic vibe that Rockstar sought to attain.

So this led to a huge revamp aimed at injecting believability in the horses, with the team focussing on three core areas:

This is not a small task by any stretch, with the work on horse AI and controls handled both by a team at Rockstar India based in Bangalore, as well as the games' primary studio, Rockstar North, in Edinburgh, Scotland. To get started the dev team sought inspiration by observing real horses in the hands of professional riders. And this led to the first fundamental change to the horse behaviour: the introduction of lateral movement.

Introducing Lateral Movement

Lateral movement (or, y'know an object moving side to side) is not something we see in videogame animals. They're typically objects with a fixed forward motion and an ability to steer, with the turning angle becoming tighter the slower the animal is moving. But when standing still, a lot of videogame horses simply rotate on the spot. But to make it realistic, you have to add a lot more nuance. Horses often strafe when turning, getting more purchase on the ground, and then allowing them to turn quickly. Plus, when a horse turns on the spot, it doesn't pivot from the centre of the body, but rather the root of the rotation is actually aligned with its front feet.

So this can be added in just fine from an AI and gameplay perspective, but it adds a new problem for animators. Given it's adding a new dimension of movement, which complicates the animation systems you need to make it look realistic. Typically, an animation team will dictate which animations a character in a game can transition into, and then the exact points where the two meet - known as a blend tree. The blend tree can then say based on some input information, such as the speed of the character and their direction, what weight each of the possible animations has in the blend. So if you're looking at the speed of a character, you can tell whether that character should walk or run and if it's in that middle-ground where the character should be transitioning from walk to run, the weights dictate how much each animation is being used in the character model.

Blend trees work great for small sets of animations or with animations that easily align with one another. But of course, this gets incredibly difficult the more realistic you want the movement to be and the range of freedom the character has and the number of unique animations increases. That's because developers need to sit down and manually figure out where those animations should blend and what the rules are for making it happen.

So what's the issue here? Why is adding lateral movement such a big deal? Well previously, the horses only had two key considerations: the movement speed, and the turning angle. And that's already a huge problem, given you're dealing with a four-legged animal. If we consider the movement speed, or rather, the linear velocity of a horse, the gait or movement pattern of the legs, changes accordingly. Typically a horse moves with one of four key gaits: walk, trot, canter and gallop. In each case, you can see the physical exertion of the horse based on that gait, and while the speed may be within some definable range, it isn't fixed. On top of all of this, when you watch them move, you'll notice the intervals between their front and rear hooves hitting the ground are irregular.

Bringing this all to a point, you already have an animation system that has to figure out how to blend different animations for the linear velocity, then you add in the angular velocity (aka the speed with which you're steering the horse) and already that is a complex mesh of two dimensions being plotted in an animation space, within which you need to figure out where the blend points are for the animation tree. Now, adding in lateral velocity (which is the rate of strafing movement), you now have a third dimension, making it significantly harder to figure out the correct blend points. Rockstar originally implemented the system as before, but it wasn't working and more critically, getting the animation team to figure out those exact blend points was going to be a huge challenge. So the trick was to figure out how to automate it.

Now it's becoming more common for programmers to try and automate animation blends - and sometimes with a little bit of AI help. We saw this back in my previous examination of the modern DOOM series, with a bespoke system for calculating how to adapt animations at runtime, as well as my deep dive on Hitman, in which we looked at how IO Interactive have been using reinforcement learning to solve animation blends since Hitman: Absolution.

Rockstar's solution is a highly complicated animation framework that allows the horse to speed up, slow down, turn and the animations adapt automatically. The trick to this is a system for capturing the movement features of each animation. It could figure out how a horse is moving in that three-dimensional space, where its legs are currently placed, and based upon that, figure out what is the correct animations the horse should be using based on the three velocity values.

Taken from (Kleanthous, 2021)

The system would analyse each animation and record the acceptable rates of linear, angular and lateral velocities that it reflects and what are acceptable movement speeds for that animation. So for example, a given gait would have a minimum, ideal and maximum movement speed, meaning that the animation isn't linked to a fixed speed of the horse but rather reflects the exertion that the horse must be making in order to hit that speed range. With this analysis complete, at any point in time, the animation system could observe the current speed values of the horse, identify which animations are relevant and what blend weights to use between them.

This was a huge effort, but it yields two very important benefits: first of all, it meant that the developers could continue to add new animations and the system could automatically figure out the key features and how it can then be used in the blends. But also, it meant for blending the system could produce results better than humans could, or at least as good as what a human could achieve in significantly less time.

By introducing the lateral velocity, it allowed the horses to drift. It meant the horses more loosely turned, which had the added benefit of making them look more realistic, as well as conveying the notion that the player isn't directly controlling the horse, but rather… you're telling it where you want to go and the horse is responding in kind. Because it's a good horse.

But that isn't enough to sell the experience of the horse, given the transitions between the gaits was still a problem: as I mentioned already, the acceptable movement speeds of a given gait helped with the believability, but transitioning between gait animations requires a gradual transition, that isn't guaranteed to be fixed. Plus, critically, it has to synchronise with the feet. So the horse's animation transitions aren't fixed, instead, it uses continuous acceleration and deceleration cycles that observe the change in speed within a fixed window, and whether the animation should shift at that time. This is used not just for a horse speeding up, but also for slowing down and coming to a stop. The game has a custom deceleration cycle for a horse moving towards stop, that is designed largely to stop the horse feeling completely unresponsive and playing a fixed animation as they move to a stop, given the player could at the last second change their mind. But it also adds enough wriggle room to remind the player that they are not in complete control of the horse's behaviour.

Terrain

So with this incredibly complicated animation system in place, as a result of the introduction of lateral movement, there was another problem to add on top of this: responding to the terrain. Terrain has a number of influences on the horse. First of all, there's a need for it to respond to local geography, is it going to dodge a nearby obstacle that the player is still steering it towards, or completely ignore the player's inputs given you're barreling towards a tree or clifftop. It's worth remembering that the horse is still an AI character, it feeds off of the inputs of the player, but it will ignore you if it feels its self-preservation is under threat.

But on top of this, it's still an animation problem, because if a horse is riding up an incline, down a ravine or into water or snow, it needs to react and change its animations accordingly. Typically, to do this you rely on IK or inverse kinematics in technical animation. What IK does in a nutshell, is it allows for moving parts of an animated character to calculate the appropriate positions of limbs with respect to external forces, both in a way that is realistic by recognising the underlying skeleton and joints but also still respecting the original animation. So if you have a character that is walking along uneven ground, it uses the same animation it would use if it was on flat land, but the feet and legs are placed on the ground in a way that looks more realistic.

The problem is that sorts out the feet, but when you line it up with the body and head of a horse as it's climbing up a hill, it doesn't look right. So instead, Rockstars solution was to add specific animations that handle inclines and declines and then add another to the animation system that assessed the pitch of the horse. But rather than having them all lumped together as one big mass of animations, this time around they used a clustering technique to group all animations that are inclining, others that were declining and those on flat terrain. Then it based on the current pitch of the horse, it would grab the animations from one or more clusters and then blend them together appropriately.

This clustering process was then extended even further, with each of those three groups being extended to allow for horses that lead with their left or right foot. Meaning now instead of three separate 3D blend spaces, there was now six, handling three velocities, the pitch of the horse and which foot it was leading with. Meaning now they could blend animations for speed, turning, pitch and even a lead change - which is when a horse changes its leading foot.

Now of course in order for the horse to know what pitch it was going to need, there's a separate system that looks at the local geometry, the path that the horse is currently moving along and double-checks against the navigation mesh to validate the surface angles that it is about to traverse. In addition, there is also a mechanism to detect what are known as 'deep surfaces' such as water, snow and mud. And even things like the local wind strength is measured and stored at runtime.

So the horses have all of these animations, and the variety of different speeds, running gaits and the local geography, there's still a need for it to figure out what actions and animations it runs and in what sequence. The horses have a locomotion system that breaks up every possible individual action it can make into what are known as motions. This can mean an action on its entire body such as a canter or gallop, an action on a separate part of the body such as the head or tail (using what's known as a layered animation) and even the horse rider themself. Based on the current in-game conditions, the system calculates which motion actions it's going to use, then selects all of the animations accordingly. It's this massive data-driven framework, that handles everything from a horse stomping through the snow (and the horse reacting differently depending on the depth of the snow), to the rider reacting to turning into the wind and even the subtle details of the horses tail reacting to what the rest of its body is doing.

The number of unique motions is… actually kind of ridiculous, around 6300 in the game. Though as I mentioned earlier, the developers had sought to rework their locomotion system. So while there are over 6000 motions, the same system is being used for other animals and even the human characters as well.

Source: [Kleanthous, 2001]

But getting back to the obstacle avoidance problem, how does the horse try to avoid obstacles even when you're telling it otherwise. The game pays attention to how you're riding and the local geometry you're headed towards and calculates whether or not the proposed locomotion path that the horse is head along is going to collide with something. If a horse detects it's going to hit something, then the newly added lateral motion is used to have the horse strafe slightly to a safer path but is done so as smoothly and quickly as possible, so it then stops interfering and the horse becomes responsive once again to your inputs. The problem is this actually worked a little too well, to a point that the team were asked to tone it down, such that the player could still, if they want, have the horse barrel straight into a tree and they can actually hit it.

But, there is one main override here: having the horse stop on the edge of a cliff. Cliff stopping actually works on multiple layers, the locomotion system will detect the horse is heading for a cliff. In order to ensure this always happen, the level designers had to go in and tag all the cliff edges with metadata that denotes it's a cliff. Once the horse detects the cliff, it runs a speed penalty system that calculates the distance, the speed it needs to slow down from and the upcoming beat cycle of the horse's feet to figure out the appropriate animations it needed before using a halting animation.

Personality

Now while this piece has largely been about the animations of each horse, and how Rockstar sells their realism, it's important that the player feels like these AI characters are real animals. As a result, there's a lot of extra little tweaks made to the horse's behaviour. This either manifests solely in the animations or it influences the control, to reinforce that while you are in the saddle, the horse is driving you where it wants to go.

One small thing to talk about is how the games make it look like the rider is actually riding the horse realistically. Given the huge range of animations available for the horses, there are a whole bunch of separate human animations for horseriding, with a separate team dedicated to them. But now you gotta sync those two things up, and the trick to making this work is known as sync points. Each horse animation has metadata in it that points out the rhythm of the horse's movement, paying close attention to the feet falling and the high and low points of the body in the context of the gait. There are then separate rhythm tags synced to the rider as well, the game then synchronises these, so the rider is bobbing up and down on horseback in a way that looks realistic.

As we discovered already with the obstacles, one of the easiest ways to make the horse have some personality, is to respond more to the world around them. When the horse responds, it might ignore or push back against the player's inputs. This is of course a tough balancing act, given you want the horse to still respond to the player the bulk of the time but in situations where it makes sense for the horse to deviate, then it will do so.

The developers introduced the notion of 'motivation', whereby the horse would react negatively based on what's happening in the world. This could be because you're caught in a heated gunfight because they're being chased by a bear or you're being a wee shit and treating your horse badly. This motivation system is actually a type of utility AI - as detailed in a recent video of my AI 101 series - that scores specific things that influence its behaviour. There is an agitation motivation, which shows the horse responding to external elements. If these become strong enough, and therefore the horse is quite agitated, it adds conditions to the motion system so the horse changes animation, but it also changes their response to the player's controls. In fact, the players mechanic to calm their horse by patting it isn't just an animation designed to make you feel good. The horse will actually calm down as it causes those utility values to decrease. Provided you're not trying to calm it down six feet from a bear that is.

But agitation isn't the only motivation. The same system is used to denote when a horse is tired or hurt, meaning it again will influence the motion types it can animate from and also interfere with the controls. It adds that sense of realism and personality. Though my favourite AI motivation - and the dumbest fact in this article - is the poop motivation! That's right kids, the horses in Red Dead 2 don't just defecate based on some random timer. The horses actually have a motivation meter for poop that slowly increases, and once it hits that all-important moment, it decides to do its business!

Closing

If there's one thing to take away from this piece, it's that building realistic animals or more specifically realistic animals you can ride, in videogames requires a tremendous amount of detail and attention. Rockstar themselves admitted that they were sceptical of taking such a heavily automated approach for the horse's animation, but the final results - and the feedback from fans - suggests that this has paid dividends. The work detailed here is incredibly complicated and shows a real dedication towards just one small part of a much larger game. It's really quite something and gives us a taste of the level of detail put into Rockstars work.

Bibliography