How Prompto's AI-driven selfie system in Final Fantasy XV was built

When Square Enix finally shipped Final Fantasy XV late last year, many players were pleasantly surprised by the game’s focus on capturing the mundane details of the protagonists’ road trip to save the world.

The in-game snapshots and selfies captured by the protagonist's companion Prompto aren't always in focus, but the way in which they capture both actual and imagined in-game moments delighted players and inspired more than a few game developers.

Prasert “Sun” Prasertvithyakarn served as lead designer on Final Fantasy XV’s buddy system and AI; at GDC this week he took the stage to talk a bit about how the AI-driven snapshot system was designed and built.

“When we designed this game, what we felt was most important was the UX: the user experience. All systems and features are there to maximize the road trip experience for the player,” said Prasertvithyakarn. “And of course when we talk about road trips, we have to talk about photos.”

He says that taking trips with his friends inspired him to pitch the idea for the system to the game’s design team, and they came up with a selfie system for the game that had three goals:

“Humans have several reasons to take pictures, so we had to have several triggers,” said Prasertvithyakarn. So the game has auto-snapshot triggers like events (during important cinematics, interactive vents, or reaching new destinations) or whims. “You don’t always need a reason to take a picture, do you?” Prasertvithyakarn said. “Although we say ‘whim’, there is a pattern behind this; we take whim triggers only when there are no other triggers around.”

A graphical breakdown of Prompto's selfie system

This, he explains, ensures that Prompto is able to take snapshots that aren't covered by the other triggers, ensuring that the AI generates an even spread of images. He notes that no trigger works perfectly, but together, they mesh to generate a good variety of snapshots.

The snapshot AI also classifies photos into 20 themes. Exciting battlefield shots, for example, or selfies, or “wonderful” -- shots that try to capture the world’s beauty via tricks like wide angles.

Once a snapshot is triggered and the AI picks a theme, it can apply “detail enhancements” -- making a character turn to face the camera and strike a pose, for example. The AI then checks the subject’s bone and collision info to try and frame the snapshot well. Next, it can apply some post-processing to replicate Instagram-esque filters.

After a photo is captured, the game sorts it into an album of images captured that day.

“We don’t want users to spend too much time looking at this, so we try to reduce it to 8-12 pictures per day,” said Prasertvithyakarn. “But we actually take 20 or more photos a day.”

So the game’s team designed a redundancy score and uniqueness score that’s dynamically applied to each photo, based on what’s been captured. The game then autofilters some shots out every day, and displays the rest as the album from which the player can pick and choose what to save.

“I believe what we get from this is one of the most powerful storytelling tools in Final Fantasy XV,” said Prasertvithyakarn. He acknowledges RPGs are about stories, but they’re also about storytelling.

“How can we possibly persist on doing the storytelling through cinematics and a linear script, when players spend 99 percent of their time running around inside the world and fighting the monsters with their friends?” said Prasertvithyakarn.

Instead, he says his goal with Final Fantasy XV was to evoke an emotional response in players by giving them room to tell their own stories, and reinforcing those stories with systems like Prompto’s snapshots.

“When Aerith died, I cried a lot,” said Prasertvithyakarn. “And ever since then I wanted to be a designer who makes games that make people cry.”

In terms of lessons learned, Prasertvithyakarn shared a few -- starting with the importance of setting reasonable scope.

“You need to sacrifice something; you can’t do everything, especially when you’re trying to make a new feature,” said Prasertvithyakarn. “If you don’t have a very concrete goal you will go astray and spend time doing things you don’t really want. So try to hold on to your goals, and make them clear.”

So as an example, Prasertvithyakarn said the team was tempted to allow players to tell Prompto what to shoot. But it would be costly (they’d have to create a new interface to guide Prompto's photographic eye, and all the available buttons were already assigned), so the team compromised and made it so Prompto can only be told what to photograph during battles, where there was already an interface in place for telling allies what to do.

He also recommends fellow devs with an interest in these sorts of systems spend time researching AI, and know the limits of what AI can do.

“Be a researcher, but be a producer,” advised Prasertvithyakarn. “I am an AI researcher too, and we research it; making AI is fun.”

Prasertvithyakarn closed out his talk with a word of encouragement for fellow devs: if you think you have a cool idea, don't give up -- even if it seems like a silly, bad idea during development.

“When you are doing a new thing, it’s really hard to share the final visions with other people. So don’t give up! Time will tell,” said Prasertvithyakarn. “Because now that everything is finished, people come up to me and say 'oh, Prompto's snapshots are great.' But many many times during development, people would come up to me and say this feature isn’t working; it’s not fun at all.”

“If you guys are challengers and you are trying to make new things that nobody knows before, be prepared to not be understood; be prepared for critics. But if you believe it’s the right thing then just walk tall, and never, ever give up,” said Prasertvithyakarn. “That’s the key to making something new, I guess.”