AI and Games is a crowdfunded YouTube series that explores research and applications of artificial intelligence in video games.  You can support this work by visiting my Patreon page.

In 2017's Halo Wars 2, players take command of the UNSC Spirit of Fire once more as they face off against a new enemy - The Banished: a ruthless band of mercenaries led by the Brute warlord Atriox. Stranded in deep space with no means of help and outdated combat technology, Captain James Cutter and his crew must survive a war of attrition against one of the most ruthless warriors in Covenant history.

Developed by 343 Industries and Creative Assembly and released for both PC and Xbox One, Halo Wars 2 was designed from the ground up to present Atriox and his subordinates as exciting new additions to the Halo universe. But for this to work in a real time strategy game, they need to be intelligent and have a sense of personality. Let's explore the AI Commander system that enables for Atriox to not only exhibit strategic intelligence, but be capable of defeating players on various difficulty levels without cheating.

Cheating in Games

I've previously discussed on the show that sometimes developers allow AI characters and other gameplay systems to cheat to better facilitate the experience. As shown in our report on the companion AI in Ghost Recon Wildlands are often invisible to the enemy, spot targets for the player without seeing them and cheat sync shots to ensure targets go down as expected. But while companions such as the Ghost Squad and Elizabeth Comstock from BioShock Infinite exploit knowledge of the world for their own purposes, no genre adopts this principle more, than real-time-strategy.

RTS games notoriously cut corners to improve the experience: from having build behaviours that don't require them to acquire the resources needed (Warcraft 3), to enemies that ignore the fog of war and always know where you are. One reason why some RTS games cheat is that it's an incredibly difficult problem to write competent AI for. You need to factor at least two levels of intelligent decisions: the micro scale where you're moving individual units, and the macro scale where you're coordinating the overall army. One need only look at our look at the the AI of Total War back in 2018 - where the evolution and scale of the AI systems involved at both micro and macro is simply staggering. This isn't an easy problem to solve, but Halo Wars 2 employed a fresh angle.

The designers intended for Atriox and his commanders to exhibit strategic play without resorting to cheating. They should make intelligent moves on both macro and micro scale, adopt specific play styles and adapt to various game modes and difficulty levels. But more critically, each AI commander should exhibit human like behaviours: helping teach players to play the gmae, but also inject personality into these characters and bring them to life within the Halo universe.

Now you might think with the recent success of Google DeepMind training human-like StarCraft players with AlphaStar that employing deep learning is the way forward. But this isn't a cost-effective and pragmatic approach for a video game. AlphaStar isn't designed to be an interesting opponent for varying skill levels, its built to win at the highest levels of competition. Plus, there's the element of cost. As cool as it all is, AlphaStars training process cost tens of millions of dollars to train and that's simply impractical for even the biggest AAA studios.

Designing the Halo Wars 2 AI

So while 343 and Creative Assembly didn't train a machine learning AI based on human players, they employed a similar principle. Their AI commanders show human-like traits in combat, because they're based on strategies by high-ranking human RTS players, with one specific player a critical part of the process.

Balint Marczin, more commonly known in RTS circles as Nakamura, is a competitive RTS player, streamer and caster and at one time ranked as one of the top 10 players in the world in Halo Wars for both Xbox and PC. But in addition, Marczin is a game designer at Creative Assembly, with his work to-date including both Halo Wars 2 and Total War Warhammer 2. His job was to research the macro and micro layer strategies human players exhibit within the game and work with the programming team to build a system that would enable for AI to replicate it in various scenarios.

This resulted in five macro strategies that define a particular style of play being encoded into the AI system:

Meanwhile on the micro layer there are 10 unique behaviours the AI can execute and it allows for a lot of minute control of the individual units. This includes common behaviours such as focussing fire on individual units, holding ground at a specific location or ordering special abilities such as cloaks, shields, grenade throws and healing. But there are also micro tactics commonly exhibited by human players such as the pullback and kite. In pullback, the AI commander will monitor a squad of units in combat and cycle out those that receive heavy damage from attack range, while kiting is a tactic that keeps enemy units just beyond attack range, allowing for either long-range attack or opportunities to quickly deliver damage before backing of once more.

Kiting behaviour (image used with permission from Creative Assembly)

So at any point in time, the AI system in game will read the current game state and feed it into both a macro strategy planner and a micro behaviour selector, meaning the system can make intelligent decisions about a long-term strategy to execute, but also run micro behaviours to help ensure that strategy is being maintained in the long-run. While the micro layer is focussed purely on movement commands, the macro layer is built into multiple layers that focus on construction, tech ladders, leader powers and the management of armies. But one of the real successes of this two-tier system, is that it allows for both macro and micro layers to be customisable based on the in-game scenario.

A high-level overview of decision making. (Fagan D., 2017)

In certain campaign missions or in multiplayer skirmish matches, the AI system has complete control of both layers, allowing it to make its own decisions and conduct strategy its own way. But there are times that the game designers wanted to have more control over the system. Campaign missions that need more scripted behaviour, or a reduced competency at macro behaviour in Blitz firefight, or completing disabling the strategic elements to become purely reactive or having zero intelligence whatsoever like in some campaign or tutorial missions. All of these game modes, all facilitated by the same AI system.

The macro strategy planning is broken up into numerous layers and relies on strategy tables to define commander traits. (Fagan D., 2017)

Customising AI for Gameplay

So with all these systems in play, how do you create unique personalities or customise the AI to operate at different difficulty levels? The trick lies in the decision making at macro level, which is broken up into three sub-systems: the alert manager, the strategic decision maker and the strategy manager. The alert manager is how the system reacts to local information whilst making strategic decisions, hence it can interject and influence decisions being made to respond to dynamic changes in the world, such as creating new units and adding defensive structures such as turrets. The strategic decision maker is reliant on the strategy manager: a system whose decisions are influenced by one or more strategy tables.

The full AI system in action (image used with permission from Creative Assembly

At the beginning of a match, the strategy manager will read in a strategy table. Each table describes the strategy types that are being used by this AI character and defines their style or personality for the game. While this defines which of the macro strategies and micro behaviours it includes, the strategy table loaded into memory is influenced by the game mode and we'll talk about that in a second. The strategy table also includes build orders, tech and leader power unlock orders and preferences and the type of sub-missions this AI will frequently use - ranging from attacking enemy structures and defensive tactics to scouting the map and foraging for resources.

By defining all of this in a table, the game becomes more heavily data driven. The programming team have built all of the strategies and behaviours for the AI to execute as well as a XML driven format for designers to feed strategy tables into the system. This gives designers the freedom to develop new strategies - and even unique executions of the same strategy - without having to worry about the programming side of things. This in-turn crafts the personality and behaviour of individual leaders. This isn't just for the Banished characters, but also UNSC leaders that appear in Skirmish modes such as Professor Anders, Jerome-092 and Captain Cutter himself. Plus returning to the system overview, the strategic decision maker extends the life of a given strategy table, allowing the system to dynamically add new behaviours to a chosen strategy in order to flesh it out and show growth over time.

Image used with permission from Creative Assembly

Now the last thing to address is difficulty: since a given leader will play more or less intelligently based on the difficulty level selected. To address that, there are two approaches: in campaign there are multiple strategy tables for a given leader - one for each difficulty level - which gives level designers a lot more control over each playthrough. While in skirmish, a strategy table is built for Legendary difficulty and then scaled down based on the selected difficulty. This limits the number of uber units, the max number of turrets deployed, restrictions to tech and leader power upgrades, and even limitations on the types of micro behaviours the leader can execute.

While this creates plenty of flexibility for programmers and designers, this is all an incredibly difficult balancing act to pull off. At launch, the game had 94 strategy tables built: 28 for campaign as well as 66 for skirmish. These require the design team to be stay on top of the meta of the game as more DLC and updates roll out. Halo Wars 2 had a fairly extensive set of DLC release post launch, including new characters as well as an entirely new campaign. This added another 15 campaign tables and 111 skirmish tables bringing it up to a total of 220. This has seen a lot of work on design side to manage these AI personalities after launch, with a lot of the skirmish AIs on the lower difficulties not just heavily tweaked but actually dumbed down after launch based on player feedback. This resulted in AI players who upgrades, leader power usage and re-building of destroyed structures much more limited as well as forcing them to fan out a little more. Given lower-level AI leaders would frequently turtle to a point that it was too difficult to beat them for more novice players.

Closing

Halo Wars 2 strove to be a polished and accessible entry to the franchise for both seasoned RTS players as well as those new to the genre, with a cast of characters that evoked their personalities when in combat against the player. What's really interesting about this is that the creators of the Total War franchise sought an entirely new approach to building AI for a strategy game, rather than adapting existing systems from their own work; enabling programmers and designers to work together to achieve the best outcome for a variety of situations and players.

References

Acknowledgements

Special thanks to Balint Marczin, Derek Fagan and Lucy Boxall of Creative Assembly for their feedback and the in-engine development video footage used in the YouTube video that also generated the screenshots shown in this article.