What Should a Playtest Smell Like?

Playtesting. The science of analysing players’ behaviour and emotion during play. Observing real players’ actions during playthroughs of your pre-release or prototype game helps you find flaws in the experience and inform changes, thereby bringing the game closer to your design intent.

Running great playtesting sessions is notoriously difficult, requiring significant preparation and expertise, but they’re richly rewarding when run well. There is a reason all the top grossing apps, the critically well-reviewed games, and the commercial mega-blockbusters achieve such lofty heights: They’re being playtested properly.

To get unbiased and honest feedback from invited members of the general public you’ll need to carefully consider their every sight, sound and smell during your playtests.

So what does a great playtest …

        … smell like?

       … taste like?

       … sound like?

       … feel like?

       … look like?

       … and is there a sixth sense?

What does a great playtest smell like?

A playtest shouldn’t SMELL LIKE FEAR

When running your own playtests there is a huge temptation to sit next to a player, look over their shoulder or to talk to the player while they’re playing your game, but consider the impact this is having on players’ confidence and attention. If you’re sourcing players from the general public (and you should be), having a complete stranger sit and observe your every move is more than uncomfortable - especially for children. Playtests shouldn’t smell like fear - either on the part of the player (“Why are they watching me? Am I playing the game right? Do I look stupid?”) or on the part of the moderator too (“Should we interrupt? What are they thinking? Should we explain this mechanic, or point out where the assets are missing?”). The best way to avoid situations is to not be in the room with the player. By using screen mirroring and a video camera, or one-way glass, you remove the pressure on both the observers and the player, allowing for a more relaxed atmosphere. Playtesters will be nervous, and potentially shy, so do all you can to mitigate uncertainty or fearfulness.

Playtests shouldn’t come up SMELLING OF ROSES

Every game has flaws. If your playtests aren’t finding things that players acknowledge they don’t understand, or have difficulty interacting with, or their ratings aren’t changing over time, then your problems are likely with the playtesting process itself. You’re perhaps asking leading or biased questions and undermining the feedback, you’re not inviting playtesters with enough diversity of skill or genre experience, or perhaps the playtest environment itself is affecting players’ willingness to be honest, meaning that your game is unduly ‘coming up smelling of roses’.

What does a great playtest taste like?

Playtesters shouldn’t be REWARDED WITH FOOD

If you’re in a country that makes paying your playtesters in cash legally difficult, or if budget is pretty tight, it can be tempting to reward playtesters with food and drink. Providing payment with banquet of tasty snacks rather than cash is legally and logistically simpler. One of the biggest challenges in running great playtests is in sourcing players of particular demographics or experience; in order to encourage the general public (non-fans) to attend your offices, players should be offered a fair reward for their time. Saving a small amount on player incentives risks biasing your playtester pool, and skewing your playtest data.

Playtest food shouldn’t be SICKLY, SMELLY OR SUGARY

An over-consumption of free pizza, soda or sugary sweets can easily turn well-mannered children into sticky-fingered monsters focused on gummy sweets and not the game. If you do need to feed players, it is generally better to stick to simple, non-pungent foodstuffs, not forgetting to explore the dietary requirements of your playtesters. Furthermore, the most important thing you should provide along with food and drink is the location of the toilets, ensuring children have the confidence to ask to go. Having a sense for when a child is distracted by their need to visit the little boy’s or girl’s room is just another of the unsung skills of a playtest moderator.

Playtests shouldn’t leave a BAD TASTE IN THE MOUTH

Asking players to attend your premises requires significant trust in your professionalism and legitimacy. Players might be considering taking a leave-day from work to attend, or travel a long way. What you see as a one-day iterative playtest to explore a small part of the game, players see as a once-in-a-lifetime opportunity to be part of something they love. As you might imagine, this discrepancy in enthusiasm can cause very significant issues. It will be important to manage your playtesters’ expectations at every stage of the process, starting from their very first interaction with you, lest they leave with a bad taste in the mouth.

What does a great playtest sound like?

Playtesters shouldn’t OVERHEAR DEVELOPER DISCUSSIONS

Professional playtest labs are carefully built to be completely soundproof. Nothing will throw players off their stride more than overhearing talking, laughing or yelping. Give playtesters over-ear headphones to help block out any background noise, and ensure that the observing development team know to keep the noise down too. Streaming the playtest live to a different floor or a back-room can help lessen the impact of noisy observers.

Playtesters shouldn’t GET AN EARFUL OF QUESTIONS

As a playtest observer you’re going to be brimming with questions to ask the player as they play. Part of the reason you’re not sitting next to the player is to deny you the temptation to continually ask questions. In doing so you’re interrupting the flow that your team has so carefully designed, and invalidating players’ feedback on pacing and understandability. Unless you’re a seasoned playtest coordinator you should write down and stick to a watertight interview script, lest you blurt out a biased, mal-timed or leading question.

Playtests shouldn’t ever be held on a NOISY EXPO FLOOR

Showing games at expos is a seemingly great opportunity to gather player feedback, but they often lack the controlled environment needed to contextualise players’ behaviour. Surrounded by booming music and noisily eager fans, players will have difficulty in concentrating on audio prompts, and conversation will be difficult. More than just the noise, players may also have opportunities to watch other players play before playing themselves, or skip tutorials in favor of ‘getting to the game’. These aren’t normal behaviours, so it is unwise to change your game based upon them. Perhaps most importantly, by delaying playtesting until a stable expo-ready build, you’ve likely waited far too late into development for players’ feedback to be meaningfully implemented: too little, too loud, too late.

What should a playtest feel like?

Playtesters shouldn’t FEEL SICK

The challenges of interaction design in virtual reality makes VR playtesting doubly important. Simulator sickness is a uniquely troublesome factor, not least because of the ‘suggestibility’ of the condition: warning playtesters about potentially feeling ill can make them more likely to feel ill. Ensure you’re doing all that is necessary to keep players comfortable: don’t use wheely chairs with seated VR; avoid sickly food or drink and keep playtesters hydrated. Be prepared for some number of players to feel unwell and halt the playtest; for this reason, VR is the exception to the ‘don’t be in the room’ rule.

Playtests should feel IN CONTROL

Playtests are stressful. Builds break down, players don’t show, and sessions overrun. Not communicating these pressures and stresses to the playtester is important. Having a dry-run session the evening before a playtest can be great to soothe nerves and work out any kinks in the schedule. For all-day playtests, consider pinning up the day’s time schedule on a board or wall to help players feel reassured. Ultimately though, acting naturally around your playtesters while being secretly terrified about build stability or accidentally asking leading questions is a skill that takes a lot of practice. As you might have gathered by now, there is a great deal of preparation and expertise that goes into running great playtests - and here we’ve only really considered the playtesters’ experience, not the science of observation or analysis of player’s feedback.

Playtesters should FEEL WELCOME

Playtesting is all about introducing diverse voices into game development. By recruiting playtesters of all genders, races, experience-levels and capabilities, a game’s audience and appeal is broadened accordingly. There are very many easily-implemented ways in which games can be made more accessible to persons with disabilities, for example, and you should consider including playtesters with impairments. Inviting playtesters that are outside your pre-conceived audience for the game can help expose flaws that genre-fans and experienced players can overlook or won’t encounter, all toward more inclusive and more successful games.

What should a great playtest look like?

Playtesting shouldn’t look like a MARKETING EXERCISE

Consider the environment you’re inviting playtesters to; if your walls are covered in award cabinets or concept art, you’re putting up barriers between yourselves and the players’ honest feedback. The objective of playtesting sessions should never include trying to make a sale; in fact, the opposite should be true. Making the playtest environment as dull and uninteresting as possible helps to reduce biasing playtesters through excitement, or making players themselves feel as if they have a duty to be polite in return for your niceties.  It can be valuable to try and distance yourself from the product being playtested in order to encourage honesty: “I wasn’t involved in the making of this game, so it won’t hurt my feelings if you find things you don’t like or feel you don’t understand”.

Playtesting sessions shouldn’t look LIKE A SCAM

In asking prospective players to sign up to a playtest, it is easy to overlook the risks you’re asking players to take in sharing their personal details with you. Asking for players’ names, ages, addresses or other details in exchange for the chance to play unreleased videogames can look too good to be true. There are also both legal and moral implications in collecting personal information - doubly so if you’re wanting to invite children to playtest. Ensure wherever possible that all players’ touch-points are suitably branded, sent from official email addresses, and feature a comprehensive privacy policy.

On the ‘sixth sense’ of a playtest moderator

Crafting an environment within which players can feel comfortable and willing to give honest feedback - soundproofing, sick-bags and all - is just part of the answer for studios to truly benefit from player-centric development. Reseachers' sixth sense of when to ask players for detail and when to keep quiet; when to press development teams for playtest builds; when to apply one playtest format over another, and so many other situations, is critically important. Groups of Researchers have been perfecting this science of player research to aid their development teams in iterating towards a great experience for their players; starting from concept-stages, through design and production.

Those studios adopting a more player-centric development process have seen, felt and maybe even smelled the difference that well-run playtesting can make.

Have these tips helped you come to your senses?

What might your next playtesting session smell like?

About the Author: Seb Long is a Games User Researcher at Player Research examining game usability and the player experience through playtesting and game UX analysis. Player Research are the multi-award-winning playtesting studio based in Brighton UK, with a portfolio of more than 200 games, 1 billion+ players, and clients including EA Games, Microsoft, SEGA, Zynga, and a whole host of creative indie teams.