Author: by Maurice Tan

[There is often some confusion among developers with regard to the terminology used by user researchers. Just what is focus testing, what's user testing, and how do the goals and methods of the two differ? The IGDA Special Interest Group for Games User Research (GUR-SIG) would like to clarify the matter once and for all.]

In this day and age, it is nearly impossible to think about developing a game without some form of systematic testing being performed at different stages of production. Testing has proven itself to be instrumental in ridding the game of bugs (QA), figuring out how to best sell the game (market research), and getting the product closer to the designer's vision in terms of the player experience (user research).

The field of user research may be relatively young within both the games industry and the world of academia, but decades of multidisciplinary research in fields like ergonomics, usability, psychology, communications, and anthropology form the pillars that support researchers to answer that all-important question: Are there any unforeseen impediments to an optimal player experience, and how can the experience be improved?

While there is little doubt about the merits of testing in general, there can be some confusion about who does what kinds of testing. User research has its own lingo that can sometimes confound those who aren't familiar with it. Especially with its diverse academic baggage, user research can produce jargon that means little to outsiders, or leads to misunderstandings (even amongst insiders).

For example, some terms like "playtesting" are used both in QA and user research, but in the two contexts can refer to very different techniques. General familiarity with decades of market research jargon such as "focus groups" has created further opportunities for confusion. As such, it's no wonder that the term "focus testing" is sometimes mistaken as a synonym for "usability testing". This article seeks to clear up some of these misunderstandings so we can hopefully all understand each other a little better.

How to Understand Your User Researcher

Let's start with the term "games user research": This is the name for the overall discipline that Games User Researchers occupy within the game industry. The discipline also covers job titles such as usability specialists, playtest managers and player experience researchers. "Games user research" is the umbrella term.

As such, games user research is a game industry discipline on par with other disciplines like animation, marketing, programming, QA, and other vital parts of the development and publishing process; it distinguishes itself from these other disciplines primarily via a) its function, b) its object of inquiry, and c) its method.

The primary function of games user research is, simply put, to test and evaluate the viability of the design with the ultimate goal of helping to make more enjoyable games.

However, this alone is not enough to distinguish games user research, as the testing and evaluation of design is also a function of QA and market research. The factors that do distinguish games user research are that it specializes in a specific kind of evaluation, namely "user test"-based evaluations, and that it has a different object of inquiry than QA or market research, and therefore uses different methods.

Janus Rau Sorensen, user research manager at Crystal Dynamics & IO Interactive, offers building a car as an analogy to explain this difference: "If we were making cars instead of games (in which case this site would be Carasutra), QA would be looking for blown gaskets, flat tires, loose connections in the electrical circuitry or a malfunctioning A/C. They would try to answer the question 'To what extent does the machine work?' and the main object of inquiry is 'the machine'.

"Market research would present the concept of the car to a group of potential consumers, or explore consumer habits and preferences, and try to answer the question 'To what extent can we sell this machine to these people?' the main object of inquiry being 'the relationship between the consumer and the sales pitch.'"

"A car user researcher, on the other hand, would run a 'user test' by putting a potential buyer in the same room as the car and see if this person can make it into the car, can find the A/C button, turn on the engine, drive from A to B without crashing the car, and perhaps investigate whether or not this person feels the car is fun to drive.

"The question the user researcher would be trying to answer is 'To what extent is the interaction between the car and the user usable, satisfying and enjoyable?' Therefore, the object of inquiry for user research is 'the relationship and interaction between the consumer and the product content,' and the principal activity in which data and information about this interaction is acquired is called 'user testing'. This is also the case for games user research."

In short, games user research is:

The Methods, Tools and Techniques of Games User Research

According to Sorensen, a user test is, broadly speaking, a user (e.g., an actual representative from the potential target audience) interacting with product content while this interaction is tracked, analyzed, synthesized, and communicated in a "systematic way". In this context, "systematic way" refers to the method, tools and techniques used.

A "method" is a disciplined and structured approach to solve a problem or a challenge within a given context, for instance "usability testing to detect errors in how a game character moves in the world". A method has a certain "perspective", or in other words a theoretical foundation for understanding a phenomenon.

Finally, a method has practical guidelines for its application: "techniques", "tools", and "principles of organization". The techniques and tools are the practical instruments (surveys, observation techniques, interview techniques, data mining, eye tracking, etc) and how to apply them, and the principles of organization concern the division of labor, the project management model, the allocation of resources, and so on.

Note: different tools and techniques can be used in more than one method, and the difference between what constitutes a method or tool can be up for academic debate.

Even though all games user researchers do user testing (which is the recommended term for this activity, since neither developers nor user researchers are clear about what "playtesting" means exactly) there can be a huge difference in how the activity of user testing is carried out from studio to studio and publisher to publisher. This is caused by differences in the method, tools, and techniques used.

In turn, which methods, tools, and techniques are used depends on the time and budget available, as well as the background of the researcher, and the goals of the current user testing.

One group of researchers may prefer observations, video recordings, and questionnaires, using a small group of users (eight to 10) to get data on the key gameplay and UI issues very fast. Others may prefer group sessions for party games or for the multiplayer aspects of a game. It all depends on what problem the user researcher is trying to solve, but it is all games user research.

But What About Those Focus Groups and Focus Testing?

As a game developer you might think, "Ah, but I don't really care what you call it and how you do it. Just get me the results!" and that would be a great approach to take. The problems instead arise when developers ask, "Can we do some focus group testing on this part of our single-player game?" or "I've read some stuff about biometrics; can you do that for our open world RPG? By the way, the player can have up to 20 story and side-quests at any given time." Unfortunately such requests more often than not create opportunities for misunderstanding and wasted time rather than productive results.

The terms "focus group" or "focus tests" in particular seem to be often mentioned when user research is brought up, and it's not hard to understand why. They are terms with a long history and are methods of product evaluation that have been used in market research for decades. Therefore, since games are still products, why wouldn't you use such a focus group?

The problem with this is that in user research, a focus group is just one among a wealth of methods available, and not one that is often used, if ever, by a lot of games user research teams. To the outsider, the meaning of the term itself is as muddled as it can be.

Sure, it involves a group, but what is the focus? Is the focus on measuring qualitative attitudes towards the IP, on how much the users like the story or gameplay, if the game is experienced as fun or frustrating, or on something else? For all these aspects, there are other methods and tools that can offer better and more precise insights into the minds of users.

The article Beyond Focus Groups: Getting More Useful Feedback from Consumers by Bill Fulton, an important figure in games user research who started the user research group at Microsoft Game Studios (MGS, now Microsoft Studios), and Michael Medlock, Senior User Researcher at MGS, offers an excellent perspective on the use of focus groups.

Essentially, what focus groups are good for is generating ideas at a very early stage, and gauging a general response to a concept, but they require a highly trained moderator to yield good and usable subjective data that isn't overly tainted by group dynamics.

However, if what you are after is both subjective and objective data on player behavior using a playable build of your game, a usability test with, usually, a small number of participants, is more appropriate.

This allows the user researcher to observe and evaluate the player's actions and compare them to the player's statements; a trained observer can combine the two to pinpoint the reason for a player's unexpected behavior.

Focus groups aren't necessarily useless or evil -- they're simply one method used for a specific goal. This goal differs from the generally evaluative goals that user researchers try to attain.

On the subject of the confusion between user testing and focus groups that exists today, Fulton points out that "the term 'focus testing' is often used in the same way as 'user research' -- not to denote a specific method, but as an umbrella term for 'bringing in outsiders'."

As such, Fulton suggests that "most people are not limiting themselves to what we'd call 'focus groups' when they say 'focus testing' (although they do sometimes mean 'focus group')," which creates opportunities for confusion.

The experience of working with developers who don't know what to call user testing is shared by Rich Ridlen, senior games user researcher at EA: "Many of the dev teams that I work with at EA use the term 'focus group' or 'focus test' to mean any method wherein consumers are brought in to test," Ridlen notes.

"They understand, have observed, and see value in different methodologies (or tools), like one-on-ones, small group, high N survey-based tests, etc. They have even correctly ascribed methods to tests from working with us. However, it's all 'focus groups' and it doesn't matter how many times I correct them or that I never use that word unless that's specifically what I'm describing."

Okay, Just Tell Me How to Get the Solution to My Problem

Here is what we suggest: If you want something to be user tested, perhaps it is best to simply tell your user researcher that you want them to perform user testing on a certain aspect of the game.

When you ask for something specific like "do a focus group", then that team's response might be that there's no point in doing so, even though perhaps you just wanted something user tested that has nothing to do with running focus groups.

Your user research team will likely propose another method for testing what you want, and perhaps that counter proposal will make you go, "Yes, well, that's what I asked for, isn't it?" and eyes will be rolled on both sides, with ten minutes having been wasted on just communicating the goals.

So, whatever your role is (development, management, etc.), save yourself some trouble and remember that user testing is what user researchers do, and it is their job to explain to you how they will do it and for what reason given the problem area that you lay out for them. As such, it is best for you to describe what your concerns are, and the games user researcher can then design a test that can examine the validity of those concerns (just like the report from the user researcher only describes the problem areas and does not prescribe specific solutions).

As a final point, it should be noted that user researchers are ultimately here to help you realize the designer's vision, and as such the data they collect is not intended to dominate or limit you. Rather user research aims to simply provide additional information to help you in building a great game.

As an aside, if you are a user researcher (or just interested in the discipline) and haven't joined the GUR-SIG LinkedIn group yet, you can do so here. Joining allows you the opportunity to participate in discussions such as the one that led to this article, connect with others in the area, and share your insights into methodologies and novel approaches.