Witch Beam co-founder Sanatana Mishra explains how to interpret feedback from public demos

Sanatana Mishra, co-founder of Unpacking and Assault Android Cactus developer Witch Beam, believes the professional obligation of developers is "empathy."

It's quite the opening salvo for a GCAP 2023 talk entitled "Interpreting Player Feedback from Public Demo Events," but Mishra explains that soliciting feedback from players is "really about developing your empathy" with a massive and diverse group of people who've likely never engaged with your title before.

Mishra explains that public demos serve as both a great marketing tool and a giant play-testing session, but suggests it's imperative to prepare accordingly for "deluge of feedback" you're about to receive. For starters, he says its important to set expectations by establishing the anticipated audience for specific events before identifying the subgroup you want to attract with your game and booth design. After that, he suggests assigning a weighted value to every piece of advice you receive, and claims that process starts by "understanding the difference between who you're building the game for and who's actually sitting down to play."

"As an added bonus, if you aren't attracting the audience you intended or imagined, that's a really valuable piece of feedback in itself," says Mishra. "There are also 'known, knowns,' which is the idea that you might already be aware of major issues in the game—an issue you aren't just hypothesizing about but are actually planning to solve in a specific way. This can be important to think about because you can plan to steer people away or quickly disregard feedback related to it. Just be careful not to handwave people away, or players might assume the rest of their feedback isn't worth giving and stop talking to you."

When preparing, Mishra says it might be useful to consider 'top highlights," which are key parts of your experience that you expect to be "absolute highlights" for players. Those can be used to measure how your own mental map of your project compares against how players actually see it. "It's surprising how often these things will differ," he adds.

Taking those things into consideration, Mishra often filters feedback in to two categories: feedback related to an existing hypothesis that allows devs to make a case for or against a specific design choice, or unexpected data that necessitates deeper analysis. Mishra says the latter is less immediately useful because it can't be applied straight away, although it still has value because it "represents an area of your game that you just didn't really understand—and that's a really serious thing."

Priming players to generate feedback

Although he concedes it might be controversial, Mishra suggests that priming players based on the level of onboarding already implemented in your game can be "very beneficial." He says that telling players explicitly when they should be interacting with you and when they should be figuring things out on their own will help guide feedback away from areas that aren't really being demoed.

"There are three simple categories your game might fall into here," he says. "The first is when it's the core experience only. Now, this game lacks onboarding, so you'll want to act as a human interface for the inputs, mechanics, and systems you aren't looking for feedback on. It's okay to tell players you're going to explain things for them and really get involved. In this case, you're trying to prime players to give you feedback on how they feel about the core gameplay itself, because onboarding will come later with a huge assumption that it will somehow solve all of those other issues–and hopefully it does, but probably not."

Witch Beam is currently working on musical adventure Tempopo

The next category applies to games that already feature a tutorial. Those games include onboarding but lack the tissue that connects it properly to the core game experience, such as incomplete level designs or other missing elements. "You should make a point here to tell players to talk to you if they have any questions while they're playing the experience," he continues.

Finally, you might have a "perfect" demo that is considered a complete user experience. "It might be years away from release, but you're confident it explains itself well and guides players through the experience." Mishra says this is pretty obviously the best place to be, and in this scenario you would push players away from interacting with you, instead advising them to let you know if they run into problems while reiterating the game should tell them everything they need to know to navigate the experience.

Watch your players and make connections

After setting the stage, Mishra advises developers to really watch their players because it's the "only way to fill in the hundreds of invisible gaps that will exist as you try to understand their feedback." While keeping an eye on players, he suggests looking out for specifics like strong emotional beats (which could be a joyful or frustrating reaction) and divergent behaviors that could highlight a "missing link in your design."

Elaborating on how the latter might materialize, Mishra recalled how he was once demoing a game to the public and had multiple six-year-olds figure it out in seconds. Then, an experienced player from one of the big platform holders sat down and completely failed to understand the controls and mechanics after at least five minutes. Mishra believes the more experienced player had built-in assumptions that necessitated a stronger hand during the tutorial, while younger players were more willing to simply learn and accept behaviours.

Crucially, however, Mishra reiterated that the experienced player wasn't to blame. "It was still the game's fault," he says. "You can't just assume someone is bad at the game just because a six-year-old got it. That's not how it works."

When it comes to actually interacting with players after the demo, Mishra explains the first thing you'll want to do is make a connection. You might choose to ask 'how was it?,' and while there's nothing necessarily wrong with that query, Mishra says it's important to understand how the questions you ask will subconsciously influence players to offer a certain response.

Witch Beam demoing Assault Android Cactus at GDC 2014

"I think it's important to find emotional truths, but broad or emotion-based questions will result in less specific or more experiential responses," he says. "People are rarely wrong about how they feel, but they're regularly wrong about why they feel that way. Questions like 'did you have a favorite part?' or 'Did you feel in control of the game?' often provoke more thought from players and generally end up either identifying an underlying problem, which you can solve in tonnes and tonnes of different ways, or telling you something important about how your game is being perceived."

The point here is that you should choose your words carefully and perhaps limit the amount of problem—oriented or pointed questions—like 'how would you change [X]?' Or 'Did you like [Y]?' Because they can generate guided responses that can limit your thinking about a problem.

"It's easy to subconsciously create justifications for your own choices while pretending you're being guided by empathy for the player," says Mishra, who also acknowledges that it's okay to break some of these rules. "Some of the most interesting conversations I've ever had have gone off the rails from my intentions, big time. Just remember that what you say will directly color the feedback you receive, and explaining your intentions for the game will become the singular lens through which players view the entire experience."

He also notes that every question creates a "false confession." Those who'll try your game are people who've chosen to place themselves at events like PAX and then visit your booth, which is several steps removed from someone who simply felt passionate enough about your project to post a review on Steam or reach out with their own thoughts.

Become something of a scientist yourself

In something of a full circle moment, Mishra explained that interpreting feedback is a chance to use your "professional empathy" by placing yourself in the mind of players and attempting to understand their point of view. To achieve this, Mishra likes to build a profile of each player to assign weight to their feedback and compare "mutually exclusive player desires."

To create profiles, he suggests leaning on factors like their level of experience, their interest in this space, how well they understood your game specifically, and the level of passion they displayed while chatting with you. At this stage, however, it's important to really consider which elements are vital to you and your project.

"This is a very personal thing about your game and your intended audience," he says. "There's also inherent feedback based on the demographics that choose to play your game, which is important but not necessarily what I'm talking about here. I'm generally talking about directed game design feedback.

Mishra says different feedback requires different interpretations. For instance, he believes you should try and work backwards from overly pointed or solution-oriented feedback by asking questions like 'why did they want something to be easier?' or 'why did they suggest adding this particular feature?' For Mishra, taking the proposed solution at face value is actually less valuable than seeking to understand why it was suggested in the first place, even if you eventually choose to implement precisely what was pitched.

He also suggests that basic praise, while somewhat expected, can also be a sign of a deeper issue. Even if someone claims to have enjoyed your demo, it's vital to assess whether they actually noticed and engaged with the litany of unique design beats that (hopefully) make your experience unique.

"Are they paying attention and noticing some of the finer details? Do they make comments that sound like someone who's engrossed and looking at things as a holistic experience? Or, are they taking a step back and talking about your game like it's a pile of clay? You don't really want to be clay in this scenario," he continues. "It's so cool at first when everyone has ideas for your game and they're telling you what they want it to be, or what it could be, but that also means it isn't really coming across as being anything."

Ultimately, Mishra says it's important to view feedback through a scientific lens. He explains how you can turn an observation into a hypothesis through research, which can then lead to an experiment like demoing your game, which then provides data that allows you to make conclusions before you start the whole process over. This, he says, is the most basic thing a developer should do when assessing feedback.

"Everything someone says or does should be mapped back to the game's well-defined intentions. It works for art, sound, systems, and everything in between. This must be such a boring part of the talk, but sometimes the solution to a complex problem isn't some magical bullet or a life hack. It's just slow, deliberate, excruciatingly difficult work."

Game Developer was invited to GCAP and Melbourne International Games Week by Creative Victoria, which covered flights and accommodation.