How do you best set up a playtest, and then use that playtester feedback in the development process? Facebook and mobile developer Arkadium offers up the best practices it uses when both running and interpreting tests.
January 23, 2013
Author: by Vin St. John
One of the most important elements in the lifecycle of game development is playtesting. When you see someone from outside the studio actually sit down and play your game, you're able to better understand whether your game is accessible, usable, and if its mechanics are actually appealing. These are not questions that you want to answer after your game is released to the public. Therefore, data collected from playtesting can be an invaluable tool for mitigating risk and giving your game its best chance at success.
However, there are many ways that playtesting can go wrong, so it's important to establish a process that you're confident in, and to improve it over time. At Arkadium, our playtesting process is a collaboration between game design and marketing, and typically involves inviting players to our office. We believe that the best practices we've identified over time may be helpful to your team as well, so we've boiled them down to these five tips for better playtesting.
When designing your playtest, you should be keenly familiar with the type of player your game is for. While your game may appeal to a larger audience, it's a mistake to "design for all." We find it helpful to identify personas for our target audience -- for example, "Jane, 25-34 years old, owns an iPhone" -- and try to recruit players who closely match that description.
Some questions we always consider when recruiting are:
How old is our intended audience?
Is the intended audience predominantly male, female, or split?
What device or platform is our game targeting? (Don't recruit players if they've never played a game on that platform before).
Does your game assume any prior knowledge from the player?
The last point is often the most important. If you're developing a sequel to a popular first-person shooter (FPS), you might assume that most of your players have played the first game, or perhaps a similar FPS. In this case, it would make sense to recruit playtesters who are already familiar with the basics of FPS games, and solicit feedback on the elements that are unique to your game. If you're developing for a casual audience, however, you may actually want the opposite. We often find that it's imperative to test our game with players who have never played a game's predecessors, especially when testing the game's tutorial or a game that's meant for more casual players.
Every time we have a playtest, we always strive to have some new playtesters who have never tried out our game before. This guarantees we have a fresh set of eyes providing feedback every time, even if we also invite returning players to playtest something new.
We also have a policy against recruiting friends of our employees for playtesting. While their feedback can be very valuable in a pinch, we find that when they know someone personally who has worked on the game, people have a tendency to want to love it more than they would otherwise, which can drastically skew the data.
Of course, it can be difficult to find and recruit playtesters who closely match your target persona. There are likely online communities -- some related to your game, for instance -- that are full of players who would love a chance to visit your office and provide you with feedback. However, while your biggest fans will be eager to help, they tend to be more informed and more engaged with your games than the average player.
If you only collect feedback from your biggest fans, you're much less likely to hear about the problems that new players face on Level 1, and much more likely to hear about how great Level 10 is. We supplement our pool of playtesters by recruiting on sites like Craigslist, FindFocusGroups.com, and Meetup.com.
Before your testers arrive, set aside some time for everyone on your team involved in the playtest to do a run through. Testing your playtest will ensure things run as smoothly as possible, by ironing out the kinks ahead of time and getting the whole team on the same page.
Test your player's experience from the moment they enter your door. Your playtesters should be greeted by someone when they enter or find a sign-in sheet. Every detail matters to create a comfortable experience.
You may have a state-of-the-art usability lab and great refreshments, but it won't matter if the building's front door is locked! Is all of your technology working? Check your speakers, mouse, keyboard and internet. You don't want to have to call IT during a playtest because you forgot to install a plugin that your game requires to run.
Brief all playtesting helpers on your process. At Arkadium, the whole team can get involved in playtesting. Everybody involved in the process is briefed that day on the goals of this playtest, what we hope to learn and the process we're using.
The concept of playtesting is still new to some game developers; for the average human being, it is completely foreign. Your playtesters may be unsure of what to expect, and that uncertainty can have a negative effect on your test results. Make your playtesters comfortable so they can focus on playing your game and you'll get much better feedback.
Be sure to tell your moderators to segregate themselves from the game during the test. We assure players, "We're testing the game, not you," and that we are hoping to find areas where they struggle or get confused, because we want to improve the game. We don't want our players to be afraid of giving honest (negative) feedback, so we always stress that we are observers, not designers, and are here to listen to any and all criticisms.
Our observers sit back and let the playtester play the game with very little interference. However, we find it helpful to encourage the playtester to think out loud during their experience, to help us understand their point of view.
Recording the test for the rest of the game's team to watch later is also helpful. Because most people don't play video games in front of a note taker and a video camera, we find it's important to put a player's mind at ease about these elements and explain to them why they are helpful. Players usually enter our office not knowing what to expect, but leave understanding what a playtest is and why we conduct them.
You'll almost always want to ask your players some questions after they're done playing. Try to predict these questions ahead of time and put them in survey form so that you collect the same set of data for every playtester. If you have any spontaneous follow-up questions, you can ask them when they're done taking the survey.
A carefully worded, highly focused survey can make sharing your playtest results with the team much easier. Focus on questions that directly address the goals of your playtest. Leading questions like "Was the tutorial confusing?" are much less helpful than questions that test a player's knowledge, like "Please describe what the green button does in this game."
If your playtesters are newcomers to your game or genre, they are probably unfamiliar with many of the terms and conventions that your team may take for granted -- plan accordingly by using plain, descriptive terms in your questions whenever possible.
Our surveys always include some basic player profile questions like "What games have you been playing in the last month?" to give us context. In addition, every multiple-choice question includes an optional space for written explanation.
After a playtest, you're going to have survey results and notes from many different playtesters and observers. Compile that data quickly and share it with your team in aggregate, without offering analysis or drawing conclusions.
Analyze the results as a group, and start on the aggregate data -- your game designer may have seen one player who thought the game was too easy, but is that what everyone else saw? If all the other players said the game was difficult then you know that "too easy" is not a trend. If you like you can still return to that playtester's feedback afterward and address it as a special case. Analyzing your data in aggregate first will guarantee that your entire team benefits from the full playtest.
Once you've got your data compiled, don't just forget about it! Identify the issues that your playtest has brought to light and prioritize your next steps. The data you collect during your playtest -- which can include anything from the player's emotional state, to the number of failed attempts to click a button -- should always help you draw conclusions and come away with action items for your team. If your data is inconclusive, consider revisiting the structure of your playtest and survey for next time.
These are the best practices we've developed at Arkadiun's New York headquarters, and they have helped us implement recurring playtesting in a consistent and reliable way. Of course, we're always improving our process, and what works well for us might not suit the needs of your studio. If you find any of these tips helpful, or have a different way of doing things, let us know in the comments section.
Read more about:
FeaturesYou May Also Like
