Coming into Focus: Understanding Video Game Market Research

“Don’t come back ’til you fixed Itchy & Scratchy!”

Before I started working with VGMarket -- a market research firm focused on the video game industry -- almost three years ago, nearly everything I knew about market research I had learned via The Simpsons, specifically the episode “The Itchy and Scratchy and Poochie Show.” Confused (and hilariously angry) with a plunge in ratings for the once-popular cartoon, Krusty the Klown and his network overlords set about making changes. For this, they recruit Bart, Lisa, and a handful of their friends for an audience research session. A short time later, the new character of Poochie the Dog was born.

The episode served as an amusing takedown of focus testing for entertainment. The kids (except for Lisa, naturally) provided enthusiastic but completely contradictory feedback, and executives with no creative inclinations whatsoever dictated to the writers and artists based on trends they thought were hot (“I feel we should rastafy him by...10 percent or so.”)

Krusty introduced Poochie’s premiere episode by equating it to the moon landing, but the reaction by the fans was...not warm. Milhouse was upset they never made it to the fireworks factory, Comic Book Guy uttered the famous “Worst. Episode. Ever.” line, and Poochie was soon killed off sent back to his home planet.

Now that I’ve been on the other side of the two-way mirror, I have a much better understanding of the market research process and how The Simpsons got it right and wrong in regard to game development.

Say the phrase “focus group” to someone and the reaction is often negative. At a GDC talk in 2012, Cliff Bleszinski (then of Epic Games) said “screw focus groups, they suck and need to go away.” At GDC Europe 2002, noted designer Mark Cerny presented “The Method,” which quickly dismissed the idea of focus groups being helpful in game development. Elsewhere, focus groups have been called “the f-word.” 

There’s an important difference between play testing and focus testing.

Play testing happens near the middle or end of the game development process. The core design is locked in, and the research goals usually include identifying the strongest features and the areas that need the most improvement. That is to say, if a respondent says he or she doesn’t like the art style, it’s likely too late to do anything about it. It’s a fairly straightforward approach: have respondents play the portion of the game that’s being tested, and then answer survey questions about their experience.

Focus testing typically happens earlier in the development process, usually before any playable code exists. The creator wants to figure out appealing art styles, ideal features, top competitors, any early marketing angles, etc. This is where people can get the idea of games being designed “by committee” or circumventing the true artistic vision or what-have-you, and what I suspect the Bleszinksis/Cernys/et al. are referring to. Indeed, the very next slide in Cerny’s presentation touts the value of qualitative and quantitative play testing, the same type of approach taken internally at studios like Valve, Bungie, and Microsoft.

Critics of that type of focus testing aren’t necessarily wrong. Market research that begins with poorly defined objectives (“We want our own Call of Duty!”), includes inappropriate respondents, or is based on corrupted data can lead to disasters like Poochie the Dog. Likewise, if the results are misinterpreted or not analyzed thoroughly enough at the executive level, decisions could be made that betray the creative drive behind the project. And if a studio is absolutely committed to and believes in a certain idea, whether it be a feature, art style, character design, whatever...there’s nothing wrong with sticking to your guns.

Good market research is based on figuring out what you want ahead of time, conducting research using a sound process and hard math and statistics, and a thorough review period. If used properly, the resulting data shouldn’t supplant or destroy the artistic vision that’s driving the project—it supports and refines it. Research can also help to shape marketing campaigns (effective advertisements, trailers, and box art, for example).But you have to go into it knowing what you want to know, and market research is the tool that will help you get there. 

Like any other artistic tool, the result depends on the intent and ability of the artist that wields it.