'AI and Games' is a crowdfunded YouTube series that explores research and applications of artificial intelligence in video games.  You can support this work by visiting my Patreon page.

The Forza series is home to one of the most impressive AI implementations in modern gaming.  Drivatars are AI-controlled racers designed to drive like real people, learning from their behaviour and reproducing it in other peoples games.  Allowing players all over the world to face off against their friends and other players across the Xbox Network, without ever actually playing the game together.  It's one of the longest-running applications of machine learning in the videogames industry.  So let's crack open the bonnet and find out what makes this engine tick.

Whether you prefer the simulation of Turn 10's Forza Motorsport or the open-world anarchy of Forza Horizon from Playground Games, you'll come into contact with Drivatars: a portmanteau of the words Driver and Avatar.  Drivatars are the product of a highly complicated system, inspired by an AI technique we call Player Modelling.  The idea of player modelling is that by gathering enough data of how players play a given game, an AI system can begin to make intelligent decisions that reflect that knowledge.  In the case of Drivatar: by understanding how you drive cars in Forza, it can then drive an AI car in the game as if it was you in the driving seat.  If you play any entry of the Forza series offline, all of the other racers you come across are Drivatars.  They're all AI drivers that attempt to clone the behaviour of real people.
  
But what makes Drivatar so special is that it has been in the Forza series since the very beginning, meaning it's been in ongoing development since the first entry of the Forza Motorsport series, on the original Xbox in 2005.  Making it one of the longest-running machine learning applications in the video games industry.

What Does Drivatar Do?

What makes Drivatar so impressive, is that it does a whole lot more than reproduce actions it has seen the player make before. It's making intelligent assumptions about how you would race in situations you might never have tried before.  Your friend might jump into Forza Horizon and race against your Drivatar on a track you've never experienced and driving a car you've never driven.  Hence the Drivatar isn't copying exactly what you would do, but rather figuring out 'how would you drive this car in this situation?'.  This is where Machine Learning, rather than traditional symbolic AI, really begins to shine. 

If you consider a lot of the AI applications we see in this series, techniques such as Half-Life's Finite State Machines or the  Behaviour Tree of Alien: Isolation's Xenomorph, they are chock full of rules and logic that tell the AI what to do in a given situation.  If the player gets too close, have a Combine soldier melee attack or fall back into cover.  If the alien saw you go into a locker, chase the player down and rip the door off. In either case, it requires programmers and designers to figure out what these rules are going to be, and then put them into the system.  If there isn't a rule for a given situation, then the AI might look stupid or broken, given you expect it to do something in that situation, but the character doesn't have a response.

Now consider this in the context of a racing game.  You want to make a turn into the bend by a certain degree, as well as apply a certain amount of gas.  Okay, fine.  But that's going to be based on our current direction and velocity, so you need to factor that in there, oh we also need to think about the weather, given that will influence the traction of the tyres, oh we also need to factor the weight of the vehicle, the types of tyres we're using, the horsepower of the car and - you know what I think you get the idea.  If any one rule is missing, then the AI is not going to drive that car properly and you then need to factor this for every corner, of every track and every vehicle.  

It is possible to achieve this with an approach that models the physics of the car and handles how it would follow the race ribbon.  But this complexity increases the number of tracks, race conditions and vehicles are added.  If we consider the most recent release, 2018's Forza Horizon 4: that game has close to 100 racing events and around 700 vehicles.  Simply put, it's increasingly demanding to do this effectively before we even get to modelling the player's behaviour.

But, with machine learning, this is much more plausible.  Why is this?  Well, one of the things machine learning is very good at is identifying patterns in large complex datasets.  And it can then generalise them, so it knows how to respond when it sees a particular pattern in the data.  What does that mean for a racing AI?  Well, instead of having to write thousands of different logic rules for different conditions, a machine learning system will recognise that a lot of these situations are *very* similar to each other, and as a result, it can say on a per-frame basis, what steering and gas should be applied across the myriad of similar but unique cases. 

So let's get into it: we're going to explore how Drivatar works, but to do that, we need to talk about the one gameplay system both players and AI interact with: the physics simulation system.

Forza's Physics Simulation

When the Forza series was devised, it was focused on delivering an authentic and realistic simulation comparable to that of rival series Gran Turismo on the Playstation.  Hence it has a highly complicated physics layer built into the game engine.  This physics simulation was built by pouring over the likes of Bill and Doug Milliken's 'Race Car Vehicle Dynamics', the textbook for anyone seeking to understand the physics of driving.  This allowed the development team to capture the physics of race car driving in their own rigid body simulation engine.  In much the same way simulators used by the likes of MacLaren or Ferrari do when training F1 drivers. 

But they don't stop there.  The physics system is continually updated with each release of the series, and often the team at Turn 10 studios work with car manufacturers to get more accurate reproductions of how their cars handle in specific conditions.  Ignoring the marketing schpiel and relying on the expertise of motorsport engineers, sometimes even contacting the manufacturers of car components themselves.  In essence, the Forza physics simulation is one of the most complete and accurate race car simulators that exists in the world.

The physics system is hyper performant, to a point that while the games renderers have fluctuated between 60 and 30 frames per second depending on the target hardware, or the demands of Horizon's open-world environments, the physics simulator runs at 360 frames per second and has done since the very beginning. This system is so complex and reacts to the smallest of changes, that it actually has several layers built into it that act as buffers between a human or an AI as they attempt to drive a vehicle.  This is why Forza Motorsport and Forza Horizon feel completely different from one another.  There are layers of adaptation between the player and the physics system designed to separate the simulation-heavy approach of Motorsport from Horizon's more arcade-y vibe.

Plus there are layers on the player controller level, such the how triggers and sticks of an Xbox controller would map to the realities of driving a car.  Plus there are many assists that real drivers would use: from the likes of automatic gear changes to using the ABS for braking, or the traction control to ensure the car doesn't skid out.  It helps even the most novice players get a feel for how to race in a variety of conditions and in each game the player and customise these in the Assists menu.

Now before the Drivatar gets its hands on it, we need to talk about the AI controller layer.  The AI controller is designed to know exactly how to drive a car in a given situation based on the current physics simulation.  Whenever you race a track in Forza, you are presented with the racing line on-screen, this is the most efficient path you take for this immediate section of the track based on the physics simulation.  The AI controller layer adopts this information to achieve the maximum utility of the vehicle on that stretch of road.

So for example, if you're driving a car around a particular bend, there is an AI layer that already knows what speed and steering is required to drive that car so it follows the race line as effectively as possible.  This isn't a secret, and if you need evidence of this, you can turn on the Super Easy steering assist introduced in Forza Motorsport 7, that when combined with the ABS and Traction Control, pretty much drives the car for you.  All you have to do is keep your foot on the gas.

So with all these layers in place, the last step is Drivatar.  Drivatar will then tweak that AI controller layer to drive like a specific humans behaviour.  So let's start, by winding the clock back to 2005 and the original Forza Motorsport.

The Original Drivatar

Given the novelty of what Drivatar represents, it required a unique insight into making it happen.  Hence, while Forza Motorsport was developed at Turn 10 Studios, in Redmond, Washington in the United States, the original Drivatar was conceived and developed at Microsoft Research Cambridge, in the United Kingdom.  This isn't the only time that Microsoft Research has had a significant impact on Xbox game development, with the likes of the Kinect peripheral and the TruSkill matchmaking systems built on the AI research conducted by their divisions.

The original Drivatar is quite interesting, given that it exploits a feature unique to the Xbox at that time: a hard drive. Unlike Nintendo's GameCube and Sony's Playstation 2, the original Xbox came with an 8GB hard disk installed and ready to go.  An idea that became more commonplace with the next generation of consoles.  So in the early Forza Motorsport titles 1 through 4, your Drivatar was trained on the hard drive of your console, meaning it only existed on your Xbox.  Meanwhile, other drivers are based on existing pre-trained data from the developers at Turn 10 that was shipped on the disc.  But as we'll see in a moment, each Drivatar is fundamentally a really small amount of data.  And while they wouldn't be shared via Xbox Live, you could grab an Xbox Memory Unit and copy the Drivatar for use on other consoles.

The reason that Drivatars are so easy to transport is that they are what are known as artificial neural networks.  Neural networks are a popular form of data inferencing inspired by the human brain, where you have sets of neurons - which are each simple individual processing units - that are connected to other neurons on subsequent layers.  The idea being, that when data is entered at the input layer on the left-hand side, each neuron processes the information and passes it along to the next layer.  This process continues through the layers very quickly and the output comes out the other side.  The trick with a neural network is that - outside of the basic parameters of the neurons that are consistent across all versions of that network - the thing that separates one Drivatar from another is the weights on the connections.

Training a neural network to solve a given problem involves a process of modifying the weights of the connections.  This is a long-form process, but when it is done, the resulting Drivatar is essentially a list of numbers that correspond to the weights on the connections of the network.  So transferring those files is relatively easy, given the data being transmitted is more or less just the numbers for the connection weights.

Now the one problem is that neural networks typically only have one answer for one general scenario.  You feed the input, - in this case a particular part of the race line - it will have one output: what the car should do in that scenario.  But we want some variety, and that's where bayesian neural networks come in.  

Bayesian Neural Networks apply the idea of Bayes theorem to a neural network.  What this means is that any decision it can make is also based on the probability of its outcome.  And that probability is going to be influenced by the frequency with which specific behaviour appears in the recorded training data.  Bayesian neural networks are designed such that you don't just have one set of weights, you have a collection of different sets of weights attached to a probability distribution for that input.   When you want an answer, you run it several times, using the different weight sets.  In each case, you have an answer, but also a confidence value as to whether the network thinks that should be the answer to that problem.

In the context of Forza: if we've seen the car take this particular turn on the track across several laps.  At each step along that race line, the Drivatar can suggest one or more options for the angle the car should turn and the gas it should apply.  It then selects the option with the highest confidence.  That's what the car should be doing at that point because it's the most similar to what we've seen the player do before.

But this was just the beginning of the Drivatar system, given it was heavily reliant on whatever data it could gleam from that training set.  Meaning it had difficulty adapting to new tracks and generalising for scenarios it had not seen before.  Plus it was limited by only being able to base its knowledge on the cars you had driven.  The next step was to make it a continuous and ongoing form of learning: adapting and evolving after every race you enter in every game of Forza.

Drivatar v2.0

Starting with Forza Motorsport 5, a launch title for the Xbox One back in 2013, the Drivatar system received a significant upgrade.  The most notable change is that instead of being trained and iterated upon locally on your hard drive, Drivatars are now hosted on the cloud as part of the Xbox Network.  This means two big changes: first, in order for your Drivatar to learn, you need to maintain an online connection as data is uploaded to the cloud.  But more critically, it meant that your Drivatar could then be downloaded to other peoples Xbox's with ease, resulting in your racing AI appearing in other peoples matches.  

Given that each Drivatar is a trained neural network, they are very small file sizes, because it's just the network weights.  So the game can quickly download new Drivatars prior to a new race, meaning you face off against your friends or random players.

Now the exact inner workings of the modern Drivatar is not publically known.  However, creative director Dan Greenawalt has advocated that the modern version employs a form of deep learning, which would mean the original bayesian learning paradigm has been replaced with reinforcement learning, which is a lot more popular among machine learning researchers and developers.  That said, there is still a ton of information to cover on this new and improved version.

If you recall from earlier, the AI controller layer that sits atop the physics simulation is able to calculate the utility of the vehicle's physics.  With Drivatar 2.0, the system attempts to calculate your utilisation based on turn types in the track.  In a given turn and the current condition of the physics system, Drivatar will guess the speed and turn angle with which you take this segment of the track and what acceptable range you do this in that is reflective of your driving style.

To make this work, each race ribbon in Forza is broken up into individual turns and segments and are examined individually both when you race the track yourself, but also when the Drivatar takes the wheel.  Drivatar tracks 12 different turn types for any and all Forza race tracks:

So for any given track, it will break up the ribbon into each of these turn types alongside the straights that exist and examine the player performance on each.  For example, if we take Road Atlanta, which has appeared in every entry of Forza Motorsport, the full circuit is broken into segments thanks to 12 main turns.  But the Drivatar system breaks up the ribbon into 27 unique track segments.

When you race this track, it will pay attention to the position of the car, your speed and how consistent that behaviour is throughout.  This behaviour is uploaded for the race as a whole, the main segments of the track and each turn that has been identified.  This data is then used to update the Drivatars trained model in the cloud.  Each time data is uploaded, it's timestamped.  As such, as you're building more and more training data about your driving style, Drivatar recognises that it should treat older readings with less confidence, given you might be getting better at the game the more you play.

When the model is updated, it focuses on achieving the precision of your play style but also can it infer similar behaviour for different conditions.  Now when the drivatar races, it can operate in one of two ways.

First of all, if you have played that track before it will attempt to mimic the race lines you have established as best as possible.  Similarly, if you have driven that car before, it will mimic your utility as best as it possibly can.  This means that if your friend is racing on a track you've experienced and your Drivatar is in a car it has used on that track before, it will aim to match what you did as closely as possible.  In fact, the Drivatar remembers what cars you've used on a given track.  So if that car is permitted, it will not only pick that car to drive but also use the livery and tuning that you have selected.

However, if the race conditions are not the same as what you've experienced before, the Drivatar instead while calculating the utility based on existing knowledge it has of how you handle those type of race segments and their specific turn types on average.  So it's more of an approximation of how you handle those turns typically.  So if you've never driven the Nürburgring in the game, it will rely on your previous experience to guide it through, sort of like you would do in real life.

One quick item to point out is that the Rewind feature negates the training of the Drivatar.  If you botch a corner and use the Rewind feature, the Drivatar ignores everything that happens until after you pass that segment of the track successfully.  This is a deliberate design decision: given it prevents players from gaming the system and creating the perfect Drivatar by taking the same corner again and again.  So if you want to get your AI buddy taking those corners like you, you'll have to learn to do it successfully without the rewind feature.

Drivatar Modifications

Given that the Drivatar seeks to reproduce human behaviour, it doesn't just bring along your racing prowess, it also learns to exhibit some of your more aggressive traits, as well as your flaws as well.  Learning directly from your data, a Drivatar will sometimes learn to take corners badly - just like you, they might even spin out on certain turns - just like you would and they might be highly aggressive and bash other cars in proximity - because that's what you would do.  But the problem is, when you put that into someone else's game, that isn't always ideal.

There's a twisted irony in that when played in single-player, Forza games need to reinforce the players power-fantasy of driving fast cars like a boss.  Meanwhile, the Drivatars are designed to make it feel like you're playing online with your friends.  But in reality, your Drivatar doesn't always play like you would, because it's important that in certain situations, your AI driver behaves like a professional driver would and avoids behaviours that could impact the desired single-player experience.

For example, while you might play super aggressively offline and smash into other cars, your Drivatar will not exhibit the same level of aggression.  You might also shave edges off corners and try to pass someone by cutting over the grass.  Both of those might be behaviours you exhibit when you play, but if your Drivatar does that in a race, it might prove frustrating and detract from that player's experience. Hence, there is a layer of additional control that sits between the Drivatar and the car controller that allows designers to tweak how your Drivatar behaves.  This can mean it minimises aggression, stops your Drivatar doing doughnuts or even racing backwards - because yes, some of you had to try it didn't you?  This AI control layer goes one step further by having Drivatars avoid collisions with players you don't have on your Xbox friends list.  Leaving your friends to deal with the more annoying digital version of you racing alongside them.  That said, there is still a chance that a random will hit you, given the Drivatars only pay attention to one car at a time.  So it might still just bump into you by accident.  But of course, if this annoys you, you can just turn it off.  They added the ability to stop your friend Drivatars from hitting you in the assists of Forza Motorsport 6.

This still leaves open the issue of rubber banding: a common tactic in racing games to tie the performance of all AI racers to the player.  Making them less performant they're too far ahead or becoming godlike racers if the player gets a lead.  Turn 10 have stated in developer talks that the Drivatars do not rubber band, and technically that's true.  They don't rubber band the AI, they rubber band the cars.  

If a player is too far ahead, the game begins to manipulate the cars so they're less performant.  They slightly increase the weight so they're harder to get moving, while also scaling back the torque of the engine or the friction of the tyres.  Similarly, if you're too far ahead, it does it the other way around and makes the cars slightly better than they really are in real life.

Of course, this has limits, given the accuracy of the physics system means that excessive changes will make the cars undrivable.  So it's all done within sufficient range that it doesn't break the cars physics model entirely.

Closing

Racing games are already an incredibly complicated problem space for building AI.  But crafting racers that are inspired by player behaviour is even more complicated.  The Drivatar system isn't the only example of machine learning applied to racing games, but it's thus far only to really deliver that personal touch.

References