How do Ubisoft's AI-driven NPCs handle dynamic player interactions?

During GDC 2024, Ubisoft officially unveiled NEO NPCs, its generative AI-powered tech for in-game conversations and player interactions with non-playable characters. The NEO NPC project is a collaboration between developers at Ubisoft and Inworld AI, which is providing the large language model (LLM) for NEO NPCs, allowing developers to create and form characters with their own backgrounds, personalities, and attitudes. Nvidia is also involved, providing its Audio2Face tech to animate NEO NPCs in real time.

While the NEO NPC tech is still in an early stage, there was a playable demo available during GDC to showcase the tech in action. This included three separate scenarios in which the player speaks directly to a NEO NPC and converses with them, getting real-time responses and progressing objectives based around getting information out of the NPC. 

The first scenario included learning about the background of a character named Bloom, sharing your own background in a role-playing sense, and getting information about the resistance group you're involved in.
This scenario also featured a relationship bar I had to fill, akin to that of RPGs with a social sim element, which is reactive to an emotions system and the pertinent topics depending on in-game objectives. Through the open-source voice recognition that's integrated with NEO NPCs and Inworld's AI Engine, they're able to interpret what you say to them, respond in kind, and remember what you've said which can be referenced at certain points later on. I tried testing the system by telling them I had superhuman powers and that I used to be on the enemy’s side, and Bloom reacted with measured responses within the bounds set in the game world.

all images courtesy Ubisoft

The second scenario had me watching a video feed of a remote-controlled drone alongside Bloom, which showed a NEO NPC's ability to respond to things happening within the game's environment and a player's ability to talk to them specifically about it. There were constraints in terms of having natural interactions while the NPC is focused on a different task, however, if the NPC was focused on one thing and unable to interpret something said to him. 

The third and final phase of the demo had me planning an operation to infiltrate a building alongside a new NPC named Iron. While talking to her and asking questions provided more insight on the goals of the operation, it was more structured like a traditional dialogue tree: there were correct responses for the best outcome; for example, suggesting the team's hacker disable cameras as opposed to creating cover with smoke grenades was the optimal choice.

It's worth noting that the Ubisoft narrative team dictates the parameters for NEO NPCs to be compliant within the bounds of the lore and story of the experience, so there is a level of curation when it comes to the creation of NEO NPCs. Xavier Manzanares, who was previously the lead producer on Mario + Rabbids: Sparks of Hope, is now a producer on the NEO NPC project, and he spoke to the design philosophy in a presentation.

"Something we keep in mind is every new tech that's on our table cannot create games by itself. It has to go in line with design and innovation as well. It's really important that GenAI is exactly the same thing," Manzanares said. He continued, "GenAI is a tool, it is tech. It doesn't create games, it has to be connected to design and it has to be connected to a team that really wants to push something with that tech. The questions we always had within the team is, 'What's in for the creatives? What's in for the players?'"

Nurturing NEO NPCs

So, filling out NPC barks or having more reactive NPCs based on the GenAI model is only one way it's being used. It is clearly a tool for writing teams, which narrative director Virginie Mosser spoke about in a post-demo panel. "So my role is to nurture them [NEO NPCs] to build the backstory, the flow, the qualities, the colloquialisms, because it's important for the dialogue style. It takes a lot of content for them to be more coherent, to have a real personality and a real focus for emotion," Mosser stated when asked about how the tech has changed how she works. She also mentioned that she had her doubts going into the project 

Kylan Gibbs, CEO and co-founder of Inworld AI, spoke to the boundaries designers can set for NPCs, saying, "We go through a variety of steps, not only like detecting anything that might be unsafe in what the player says, making sure the model ever sees that, then detecting topics that might be unsafe then passing through a variety of safety filters." He continued, "Then there's the narrative controls, making sure that you're not going outside of the intended domain of the world." Gibbs emphasized that the tools also allow developers to make NEO NPCs' behaviors stay grounded in the information that the narrative team provides.

As of now, the NEO NPC project includes the Ubisoft Paris, Montpellier, and Montreal studios forming a team of about 30 people, according to Manzanares. This includes senior data scientist Mélanie Lopez Malet and scientific director David Louapre who are also guiding the project. Currently, there are no known applications of NEO NPCs in upcoming Ubisoft games.

Game Developer and Game Developers Conference are sibling organizations under Informa Tech.