Focus Groups, Testing, And Metrics: Developers Speak

[Developers and testing firms discuss the merits of different kinds of testing and analytics, and Gamasutra investigates whether focus groups really have become, as has been suggested by Vertical Slice's Graham McAllister, "the f-word."]

When it comes to research methods, few invoke so much distain and ridicule as the focus group. Compared to many other forms of research, focus groups are quick, cheap, and widely used. They're also adaptable, being as useful for gathering opinions on high-level concepts as the particulars of a product, even though they stop short of the systematic quantitative data gathering of usability testing.

Yet they remain very much the research method everyone loves to hate. And the game industry is no exception. As Graham McAllister, director of game usability testing firm Vertical Slice told the Develop conference in Brighton back in July, "Focus groups are the f-word."

He's not alone -- there are plenty of celebrated examples of game industry successes that snubbed the "wisdom" of focus groups. Nintendo ignored the feedback from its focus groups, and pushed ahead with the launch of the NES in the U.S. anyway.

The Sims overcame intense criticism from its focus groups and became the biggest PC game of all time. Even now, as the industry becomes ever bigger and more professional, there's a strong distrust of the focus group.

"My instinct about it is that it feels like it gets more and more like a process of designing by committee, and diluting the spirit of the developer," says Neil Barnden, the executive director at Magic: The Gathering - Duels of the Planeswalkers 2012 developer Stainless Games.

"My main raison d'être at Stainless is to push forward the new Carmageddon, and my gut instinct is that the last thing I would want to do is to use focus groups to test reactions to that. We certainly wouldn't expect to improve our vision for the game by having it tested on a random selection of people."

But now that games appeal to a true cross-section of society and not just the console-owning males of the past, some think the need for focus group testing of games is growing. "In the game business, we're very used to writing games for ourselves," says David Braben, the founder of Kinectimals studio Frontier Developments. "But as soon as your audience includes people who aren't you, you absolutely have to see how they react to it."

"Everyone who reads Gamasutra is already massively knowledgeable about games. That's fine, and a lot of games target our group, but if you want to cover people who don't play games every single waking hour but only once or twice a week then you have to be aware that they might have a different mindset."

Frontier started using focus groups in early in the 2000s and hasn't looked back since.

"There's lots of rubbish spoken about focus groups, particularly in relation to political parties, where people see them as a way of abdicating responsibility for something. There's a lot of sneering," he says. "But you have to make sure people don't get confused or hung up on the user interface or the instructions and that the game is not using buzz words."

While focus groups have helped shape many of Frontier's releases, one example stands out for Braben. "When we did our first Wallace & Gromit game, we did an informal focus test ourselves, because we wanted to make sure that it worked well for kids," he says. "It was astonishing because half didn't understand how to overcome a puzzle."

Frontier responded by adding a message that appeared after a set period of time to advise players to look up a tree to solve the puzzle before putting to the test with another focus group made up of different children. "What was interesting was that then something like two-thirds were then fine with the puzzle, but a third of them didn't do as the message suggested," says Braben.

"We asked them afterwards why didn't they do what the message said, and they said 'because it told me to do it'. The problem was they said they were happy doing other stuff, but then wrote that the game was dull because there wasn't much to do -- because they just went around collecting stuff and not proceeding with the puzzle."

The results of the second focus group led to a further tweak to the game. "We changed the message to say 'don't, whatever you do, go up the tree -- it's dangerous'. Then every single kid on the following focus group did it and said the game was great because there was tons to do," says Braben.

Braben says picking up issues like this is why focus testing matters for games. "I don't think we would have picked up on that bit of reverse psychology without having seen it in the wild, so to speak," he says. "One of the tragedies is when you've written a game that you are really proud of and then you roll it out and hear from a friend or relation that they got stuck right near the beginning. It's often the tiny things that put someone off."

One of Frontier's more unexpected findings came during the development of Kinectimals when the focus group sessions highlighted that American children throw differently from British kids thanks -- presumably -- to the popularity of baseball in the U.S. and cricket in the UK.

Braben feels that the hostility to focus groups is largely an image problem and it would be better if game developers just forgot the f-word. "Drop the word 'focus', just call it 'testing', because then you begin to look like an idiot for not doing it," he says.

"Focus is a fancy word put in front of the word testing to validate it. If you don't test or validate ideas with the people you think will like your game then you are missing a trick because catching problems early is really important."

The usefulness of focus groups is often about how they are used, too. Professional research agencies offer advantages, not least their experience of managing focus groups and dealing with the danger of individuals within the group dominating the discussion.

But, says Braben, it is often "better to do more tests than just one or two at greater expense" because -- as his Kinectimals example shows -- people are not homogenous. "Because the States is such a big country, you find there are different opinions in different parts of it, and similarly in Europe -- a focus test in London might not be valid in Germany, where there are different sensibilities," he says.

Kinectimals

Another tip is to test different implementations of a game with different people, rather than the same group. "We often implement something in two or three different ways, and put different groups of people in front of each approach and see how they react," says Braben. "That is very useful, because you can see the different techniques side-by-side, and whether people got it instantly or floundered for 20 minutes."

But now that online connectivity is allowing game companies to harvest vast quantities of data about how players actually behave within games is the focus group with its reliance on participant memory and honesty a dated idea? Given that online games firm Bigpoint employs a dedicated team of data analysts and boasts about the enormity of its data mining operations, it would be a fair assumption that the German giant would have no need of focus groups.

But the truth is rather different. "The basic principle at Bigpoint is listen to your community, but believe your numbers," says Philip Reisberger, the chief games officer at the free-to-play specialist. "We are gathering a lot of data, but we like focus groups a lot because they provide you with instant feedback and you see the reaction of the people. A number on a spreadsheet is just a number. If it goes up 5 or 3 percent, that's something, but there's intelligence behind that change."

In fact, Reisberger feels that focus groups might actually be more important for an online game provider like Bigpoint than for publishers and developers of boxed retail games. "We're a service business, not a product business," he says. "So understanding how the service is consumed and viewed by the user you only see when you talk to them."

Even game research agencies that focus on quantitative data agree that metrics are just one part of the testing puzzle. "Metrics are complimentary," says Justin Johnson, chief technical officer at game analytics firm Playmetrix. "The more awareness and understanding you can get about how your target demographic is playing your game, the better shape you are going to be in."

Metrics, he suggests, can shed light not just on what's happening but also on when to trust and to distrust the opinions gathered from focus groups. "You can still give a lot of sway to what people say, but sometimes you get a bit of contrast with what's actually happening inside the game such as what they're doing and what they're not doing, which you might not get from a survey or focus group," he says.

Reisberger says it is important that the results of focus groups are not taken as gospel. "If I was to listen to my community, I would have game mechanics that the community loves, but we don't only seek to do games that people love, but games that should monetize," he says.

"You might listen to the community and they will say, 'We don't like clicking this 50 times.' If you just listen to them you would have it as a one-time click.

"But, since you want to monetize it, you could think outside the box, and say let's monetize this thing so that we sell you, or you could earn, a virtual assistant to do that for you. If you follow the focus group blindly, you could design a game the community loves, but does not monetize."

Another problem with focus groups is human behavior, says Braben. "If you have a group, one or two individuals will tend to lead the group, which is very frustrating," he says. "You can have some very loudmouth, opinionated people, particularly in informal focus groups, who will tell others how to do things. That actually invalidates the point of the exercise, as you're trying to find out if each individual would have got it on their own."

Vertical Slice's McAllister adds that a common problem is that focus groups are misused. "There is value in focus groups, they're just misused in some ways," he says. "Focus groups are very good at gathering opinion, but some use focus groups in place of usability, user experience, or play testing, which is about people's behaviour."

Although Vertical Slice does run focus groups on games that are in the earliest stages of development, the firm sees a marriage of biometrics and interviews as a better solution for testing games at later stages in the production cycle. In Vertical Slice's case, this involves monitoring people's eye movements, facial expressions and sweating as they play and conducting a post-play interview to shed light on why their body responded in certain ways at different points in the game.

"When doing the interview know that something's happened that we want to validate or identity," says McAllister. "With galvanic skin response, for example, we might see a spike but we won't be sure why. That information has to come from the player."

The kind of usability testing done by companies like Vertical Slice overcomes many of the limitations of focus groups, but what of Barnden's gut feeling such research is a step towards game design by committee? Doesn't the feedback from focus groups, usability tests and online data simply pave the way for neutering game designers, and ensuring that games become more and more middle of the road?

McAllister says it's not the job of researchers to tell game developers how to do their job. "We do not try to tell you how to do game design," he says. "We're experts in understanding people; they're experts in making games. The relationship is 'How can developers use this data to make the game better?'"

It's also important to remember who is in charge, adds Braben. "You're still in control. You can ignore the information, but you do so at your own peril," he says. "If you think of it as just running ideas past different people and seeing how they respond then we all do it -- even when writing a game for ourselves, because you show it to your mates. Ultimately, the biggest focus group is when you release previews, but that's a high-risk focus group, because there's no going back then."