Background

In the last few years I have been engaged with the incremental/idle games community and I made a few experiments/prototypes myself. If you don't know about these games, I recommend that you read about them, but to make sense of this post the most important feature is that these games are partially automated, with progression while the player is away and very long play sessions.

A couple of months ago I decided to revisit an old prototype and add some more content to it. Since I launched the game several months ago, some feedback was lying around, and it encourage me to get serious and properly update it.

Since I started putting so much time and effort, I decided that I should make it right, so I wanted to use playtest to validate or not my assumptions. First I tried with regular playtest, but it didn't work well. Due to the nature of incremental/idle games, I had to deliver a sped up version of my game to the tester, and still it felt really awkward to sit there for quite a while waiting for things to happen. Then I thought that it could be a good idea to use online surveys to gather feedback. I looked up on the internet trying to find resources to learn how to do it properly.

There is a lot of resources online explaining how to do playtesting, but little regarding surveys. At best, they say that you should give testers a survey to fill at the end. At worst, they discourage using them, telling to do regular playtest instead or using other means (e.g. online communities) to get feedback.

Since none of this worked for me, I decided to give it a try and see how that works.

Results

I made a survey of about 40 questions using Google Forms in about 1 day and posted it into /r/incremental_games. I left it open for about 3 weeks and then analysed the data.

I made a whole thread commenting the results here if you want to dig deeper. The most relevant statistics are:

In the following points I will give concrete tips on how to build your own surveys from my experience, and I will close with some conclusions.

When to use online surveys

I used online surveys almost out of necessity, but after the experience I am convince that a wide number of games could benefit from it.

Online surveys would fall under the blind playtest paradigm, one that you see more often used in board games. I think of online surveys as a middle ground between traditional playtest (in person) and game analytics.

If you are doing traditional playtest, you should already have a survey that you can easily adapt to put online. Even if you don't have one, it takes less than 1 day to put everything together and running.

My prototype is a webgame so it is trivial to just put the link in the game and invite people to fill the survey. However it should be pretty easy to put the link inside a mobile app, or bundle it in the game docs or link if it is downloadable.

Bottom line, as long as you have a digital prototype that is distributed through the internet, is reasonable to do a playtest survey.

What and how to ask

The questions in the survey should be specific and focused. It works best as a tool to confirm or not assumptions that you have about your game, instead of for instance get new ideas. There is a couple of exceptions, such as demographics, open feedback and general questions.

But far more important for surveys is how to ask. It is a very well known phenomenon the way that the same question is asked can change the way that people answer. The most typical example is asking a yes/no question in a positive or negative framing e.g. The graphics are pretty vs The graphics are ugly.

There isn’t much that can be done there, except to be aware of it, be careful with wording and try to minimise this kind of questions. Some survey platforms allow for A/B testing, where a question has two versions that are shown to players. With these feature you could word the same question differently and average the results.

It is also very important to ask short, concrete and clear questions. This isn’t condescending to testers, but it makes the survey more accessible and gives better answers. For instance, my question “Are the graphics elegant?” performed poorly, probably because many testers didn’t really understand what element means.

Another point is how many questions to ask. My survey included 40 questions, which is a lot, and still got many responses. Generally speaking the longer the survey, the more players will be turned away. However you don’t want to make it too short, to gather as much data as possible and make it worth your while.

More important than the number of questions is how long it takes to fill in, and this depends on the number, but also the type and complexity of the questions. No much information about the best values, but I would aim for the 5-10 min range (mostly the lower end).

Lastly it is important to timebox your survey, that is, to close it after a certain time. For one, if your game is being constantly updated, you want all your answers to be based on the same version of the game. Second, you want to get to a point where you can gather the answer, analyse them and try to get information out of it. Lastly, if survey feedbacks are exceptional events, you are more likely to get players attention and encourage them to fill it in.

How to structure the survey

The order of the questions is also relevant, with a couple of very important points.

To begin with, you should start your survey with a short disclaimer describing what this is it about. You should make clear that the purpose of the survey is not to give suggestions or report bugs, and link to the relevant material such as dev email or game communities. This may seem unnecessary, but I was surprised at how people reported bugs or used the text boxes to write very lengthy suggestions.

Generally speaking you should group questions into topics e.g. graphics, gameplay, multiplayer, etc. The reason is that it is easier for players to fill questions about the same topic if they are together since they are “already thinking about it”, that if they are all over the place. Inside each topic I group questions by type, e.g. ratings, choice, etc. No special reason, other than that it looks good.

The first section of my survey was a few demographics questions. These are not terribly informative by themselves, but may be useful to answer questions about your target audience or to draw correlations. One particularly relevant question here is asking if the tester has played the game before. This can help you to separate new impressions if you game has been out for a while.

The second section is ratings, and is one of the few specific advice that I managed to find online. The thesis here is that you want to get the tester “gut feeling” about the game, by asking him to rate the overall experience and different aspects of the game on a scale. The reason why this goes at the beginning is because just like with wording, it is a very well known phenomenon that asking questions about a topic changes the way that a person feels about it. At this point players should answer the first thing that comes to mind, without reflecting too much into it.

Here I also included a free feedback text box, and that requires deeper explanation. Generally you should include a free feedback text box; most of people will skip it, but there are testers who want to share their thoughts, and you don’t want to waste that chance. For reference, about one third of people filled this box in my survey.

Usually free feedback goes at the end so that people can write anything that crossed their mind during the survey. However, I wanted to keep that “gut feeling” intact, but I didn’t want people to have to go back to the beginning if they felt that they had something to say. My solution was to put two free feedback boxes, one at the beginning and one at the end. Moreover I included a free feedback box at the end of each topic.

On retrospect that was useless and wasteful. The first general feedback was the one that most people filled and gave the most relevant data. The others weren’t as effective, and even some people called me out on the repeated questions. Also, I didn’t feel like the placement of the boxes had much impact.

After this experience, I would recommend to place one single free feedback box at the end of the survey.

The rest of the survey is filled with the different topics, and at the end there is one general free feedback section with open ended questions and text boxes. As per the previous paragraphs, this was mostly useless. The only open ended questions that produced relevant answers were “What did you enjoy the most/least” and “What single feature would you add”.

Types of questions

In general questions can be of the following types:

Some questions can be asked using different types, but it will feel more natural or produce better data in one format or another. If you have problems deciding which one to use, think about what type would produce the most useful answers.

Next some tips about each type.

Ratings

I can hardly think these are useful for anything else than rating something. I only used them for the “gut feeling” high level questions. This information is midly interesting itself, but it likely won’t answer your questions e.g. if people rate the mechanics poorly, it doesn’t answer why.

I recommend to use them sparsely.

Choice

Players choose one of several available options. Many questions can be answered this way. It is also suitable for clear and concise questions that the player can answer quick and easily. Examples “Did you complete the game? Yes/No” “Did you get engaged by the game? Since the beginning/After a while/Not at all”.

Checkboxes

A choice where you can select mutiple answers. I didn’t use any but they are useful in specific circumstances. The caveat here is to avoid asking vague questions that can lead to poor answers. An example of a bad question would be “Which stages did you enjoy the most?”. Well, how many should I check?. A slightly better version would be “Which stages did you enjoy?”, but this still requires some thought process and is difficult to set a limit. Examples of good questions would be “What areas of the game did you visit” or “What character classes did you play”. These questions have multiple answers, and the player can ask them clearly.

Likert scale

A likert scale is a scale to measure how someone feels about something. There are many variations, but I used a 5 points scale from “Completely disagree” to “Completely agree” with neutral in the middle. In the survey, I wrote it as a matrix question.

Likert scales became the core of my survey for two reasons: It produces short, clear and concise questions that are easy to answer, and produces simple answers that are easy to analyse.

There are two big caveats here.

The first one is the scale. I used a 5 points scale because is what I saw online. However I didn’t feel the difference between completely agree and agree was significant. In fact, I processed the results to reduce it to a 3 points, which is easier to write, answer and analyse.

The second big one is that Likert scales are NOT rating scales. I’ve seen some people reduce liker results to numbers and calculate means, etc. The key difference between Likert scales and rating scales is that Likert has a neutral position, that means that the player doesn’t particularly care. If a person rated something 5 out of 10 we would consider it a very bad score. However the neutral option gives players an escape route to clearly signal that they just don’t mind a certain aspect of the game.

Rankings

At some point in my survey I ask players “I would like the game to include more…” as a Likert scale. That wasn’t good. As you can expect, most of the people wanted more of everything, which is not very useful.

On retrospect, it would have been better to ask players to rank items by preference. The problem with rankings is that it is usually hard to precisely rank a whole table. An alternative would be to pick the top x out of a list.

The only problem here is that survey tools offer limited and clunky (if at all) support for these kind of questions.

Text boxes

These can be at the same time the best and worst feedback you can get.

As mentioned before you should at least include one free feedback text box at the end of your survey. Most of the people won’t answer, but many will and listening to what your players have to say is a must. Here you will find all kind of feedback, from useless to really insightful, from really uplifting to nasty. It can be daunting and time consuming to go through it for large test groups, but it will be definitely worth it.

For the rest, text boxes are hardly filled, and they produce poor answers with the few exceptions mentioned before. Also they are time consuming to process since they can’t be analyse like the rest of answers. One common mistake was to ask questions that can be answered with “yes/no”.

However there is a big exception. In my survey I asked “Did you finish the game”. Then the next was a text box that said “If not, why?”. Almost everyone who answered no, filled in this text box. For me this shows that if you can ask a direct and clear open question to the players, text box can be really useful.

Caveats

One of the things that I was afraid of is that people filling in the survey were a vocal minority that would give me a distorted vision. However the answers followed a very wide spectrum from people who really liked it to people who really hated it, and a 25% participation rate seems reasonable.

Another concern was that people would lie. Sure, right after I posted the survey, two people had already answered the form. Also I got answers claiming to have completed the game in an impossibly short time. However overall these kind of answers were a very small percentage, and I don’t have any reason to suspect that people would go through the trouble of filling in the survey just to give useless feedback.

One thing that I didn’t anticipate but was to be expected is people filling the form prematurely and repeating. Although I wrote a disclaimer asking people to please fill the survey only after they finished playing, I got a lot of answers saying that they were still playing. This opens the question of whether you should let people change their answers, or submit a new form. For my test I decided that submitted forms as “work in progress” were valuable data, and people could just submit the survey again if they wanted to revisit it.

Conclusion

I enjoyed the experience, I consider it useful and will repeat it if necessary. I would recommend other people to also do it, and in fact I would love to hear from other people's experiences or even help where possible. My takeaway messages about online surveys are:

That was a long post. I hope you find it useful.