User Research: Make a questionnaire to talk to players better

I wonder if you have encountered the following situation when conducting questionnaire research:

Designed the questionnaire, put it out on market to gather data, watched questionnaire gathering numbers rise, and thought it was about to reap the fruits of victory. But when you got the questionnaire results, looking at the percentages and the bar graph, the research, you just didn't know how to turn the descriptive results into explanatory conclusions.

Here are some most common examples: ① the research results of user satisfaction is 3.7 points, But what exactly does that mean? That’s what designers often ask; ②40% of players like program A, whilst 39% of players like program B, which one should we choose?

The above problem occurs both in beginner and veteran researchers. So why do such problems occur and how to solve them when you encounter them? For the first question, the most fundamental reason is that before using the questionnaire as a research tool, the researcher did not think about how they intended to use it. For the second question, I actually don’t encourage to make up and add somewhat explanations after gathering the results, but to avoid such problems through proper preparation at the beginning of the questionnaire design.

Therefore, in this article, the main idea I want to share is what needs to be done before questionnaire design: how to clarify the questionnaire research problems, and what needs to be done from research problems to questionnaire design.

01 Clarify the questionnaire research problems

It’s always said that before questionnaire design, the first thing you need to do is to clarify the problems of the questionnaire research. But what I want to share with you here is not that you should define research problems first, rather - how to determine real research problems.

Here is an example:

One day, a designer comes to you to talk about a research project. Define if the following statement is a real research problem or not:

As a game user researcher, I think you might quickly identify that this is not a real research problem, this is just the result he wants to get. So you keep digging for the research problem that he is seeking for an answer.

Some may stop here and start to create a survey form. But the above reply seems to be a real research problem, it’s possibly not. If the reasons of wondering which PvP mode is more popular are asked, it would be more likely for you to dig out the real purpose of initiating this research project.

What’s the difference between the first research question and the second one?

“Wanted to know which PvP mode is more popular”，this is only a question；

“Planning to add a PvP mode in the evening time, but it seems there are already too many things they have to do in the evening. So we are considering replacing a mode that is not popular.”

It shows us that the above answer will help designers to solve a problem.

The real research question would be like this:

What are the problems to be solved using the research results?

Why is there so much emphasis on understanding what the research question is behind the research results that designers want. Designers can interpret results by themselves, yet in fact, it is not that simple, if we do not understand properly the real question behind the scene, the questionnaire design can give them the data results, but the results may not solve the real problems that they are facing.

For example, for the data result of "which existing PvP modes players prefer", the design of the research will vary greatly depending on the purpose behind it. In this case, designers use the result only to rank the percentage of preference of existing PvP modes that players prefer from high to low.

However, if the ultimate purpose is to plan the upcoming PvP modes through this research result, then the existing PvP modes preference ranking cannot answer this question well. The proper research method would be abstracting the core mechanism of different types of PvP mode and do preference research about it. Therefore, only if we clearly understand what kind of question we need to solve/answer, user researcher can give appropriate research results as reference to assist in decision making.

02 from research questions to questionnaire design

Once we understand the real research questions, can we start designing the questionnaire? Not yet, there is still a very important step to be taken. When we design the questionnaire,  we are actually doing a research project design. And because the questionnaire design process is very simple, and there would often be some templates that can be referred to, many researchers tend to ignore the research design & methodology consideration in this step.

What problems will occur if lack of project consideration (the middle step)? Here is an example:

Satisfaction survey is a frequent research method when we do questionnaire research.

Suppose you just receive a research question like this:

"We want to know player satisfaction of the recent update to see if this update has improved their satisfaction.”

And you just rushed into the process of designing a proper questionnaire. But if you get a 3.7 point of satisfaction, what does it mean? Is it a real improvement?

For the purpose of the research "has this update improved player satisfaction", at the beginning of the design, we need to consider what kind of results we want to use to prove it, and to confirm whether the results can prove it or not. In different cases, our research plan will need to be adjusted.

Situation 1: It would be ideal if there is a recent satisfaction rate or the satisfaction result for the last update.

Question design:

Rate your experience so far:

Very dissatisfied,       Somewhat dissatisfied,       Slightly dissatisfied,       Har to say,       Slightly satisfied,       Somewhat satisfied,       Very satisfied

Situation 2: If there is no satisfaction rate before, how do we answer this research question?

The above scenario would have no frame of reference to explain the score of 3.7, and then we would need to make some adjustments.

Question design:

Does the recent update improve your gaming experience?

Somewhat improved my experience

No obvious influence

My experience got worse

From the above example, we can see that even a common satisfaction rate research, in different situations, we also need to use different approaches/ideas to achieve the purpose of the research. And under different scenarios, our questionnaire design will also have to make changes. Therefore, after getting the research questions, it might be better not to rush to design the questionnaire. First, think about what kind of results you want to use to solve/answer the questions of designers.

The above example is actually relatively simple, and we often face more complex problems in research such as: to diagnose what problems users encounter through questionnaire research. If we don’t first sort out the assumptions about the question, and then design the research projects accordingly, then the results are likely to be incorrect, resulting in a waste of time and cost. The worst thing is to mislead game design.