Working with Local Developers to Gain Experience in Games User Research

Games User Research (GamesUR) involves testing the player experience of digital games with real players in order to gain actionable insights and improve design. Where Quality Assurance concerns the detection of buggy code and other software issues that crop up during development, GamesUR focuses on player’s thoughts and attitudes about a game. Data collected by games user researchers alerts game designers to any potential issues that might hinder an optimal experience, and also sheds light on design elements that are delivering a great experience. In other words, games user researchers help ensure a game is fun, engaging, and balanced. Landing an entry-level job in this discipline is notoriously difficult, so having experience of running playtests and working with game developers can set you apart from other candidates. If you’re seeking experience as an aspiring games user researcher, you may have asked yourself: “how can I find real game teams to work with? And how do I convince them to collaborate?”

In this article, Jess Tompkins and Hannah Murphy will share their experiences working with student game developers and local independent game designers. Both Jess and Hannah worked with indie teams and local studios to gain valuable games user research experience, and subsequently landed highly-coveted research roles inside AAA studios. This article outlines several best practices and principles for getting started in GamesUR, finding indie and/or student teams to work with, building a rapport with these groups, conducting the research, report writing, as well as following up with teams post-report stage.

Finding a Developer

If you’re still in university, working with students is a good option to build up your GamesUR repertoire. You’ll want to ask around and check your university’s list of student organizations to see if you could attend any student-run game development clubs or meetups. Be sure to attend club meetings to get to know the students and what they’re working on. If your university doesn’t have a central student game development organization, find out which computer labs students are working in and where they hold meetings. If doing the latter, it might be helpful to reach out via email first so that you’re not showing up during a time that would inconvenience the team.

If working with students is not an option, you can research  your local independent (indie) game community. Even if you don’t have friends-of-friends who know any local developers, you can seek out opportunities at game development Meetups, or with resources like gamedevmap. Another option is your local International Game Developers Association (IGDA) chapter, where you can attend events and start networking. Hannah found a fantastic development team through through a local non-profit, Glitch, that supports game makers as a community.

An alternative to working with local developer is reaching out to indie developers online and explain why you’re interested in their game. Just ask if they would be open to letting you conduct user research on their game and provide them with your feedback. There’s even a subreddit specifically for game developers wanting feedback on their game, which is a great place to start.

Earn Trust

When you meet new teams, let them know that you want to work with them to assess their game’s usability, what that will reveal about their game, and what it won’t. Be ready to explain why you want to coordinate and conduct playtesting, design user surveys or other feedback instruments, and how this data can help their game. Be honest that you may still be learning these skills yourself, but in doing so, your aim is to help the designers get to the root of issues impacting player experience. Reassure the team that the observations and data you collect will benefit them, not just you.

Any team working with games user research for the first time will have questions, whether they’re students or not. Be prepared to address misunderstandings and manage expectations: you’re not a bug tester, or a last-minute addition, but an impartial assessor of player experience helping them improve design.

Building and maintaining a working relationship with developers can be challenging. They may be somewhat hesitant to embrace an outsider’s perspective on their design, simply for the reason that criticism is not always easy to acknowledge and act upon. Watching talks about case studies in games user research can help you prepare for potential skepticism or apprehensions.

Specify the Work

In working with the team, your job is to inform them of potential ease-of-use and usability issues related to their game design. Sit down with them and ask questions about the intended player experience: what do they want players to feel, and are those feelings are contingent upon a certain level/fight/or interaction? Such conversations inform what to look for when conducting playtests and how to frame your questions for playtesters. Take notes to reference while writing survey/interview questions, or while rewatching videos of your playtest sessions.

To ensure everyone is on the same page regarding the work being done, confirm the development team agrees with your methods, the number of players needed for a playtest, the length of the playtest or depth of your usability assessment, the areas you will probe upon or analyze (e.g., usability, appreciation), turnaround time for the report, as well as who is supplying the build and when. You will also want to cover what is outside the scope of your work (e.g., bug reporting). If you’re being paid for your work, be sure to agree this in writing with the team.

Finally, assure the team your job is not to dictate the design process. Avoid presenting your own opinions as facts or serving as an ‘armchair’ designer (e.g., offering your opinions despite lacking practical design experience). This will quickly erode trust and professionalism. To avoid this, confirm that you have a strong understanding of the intended experience and reiterate the developer’s desired experience in the report. Should you be unsure about the implications of an issue, say so in your report or have a discussion face-to-face about it.

Conducting the Work

A playtest is a valuable method for collecting data about usability and player experience for games that are beyond the graybox (i.e., early) stage of development. To conduct a playtest you will need to recruit people to play the game and provide feedback. Feedback might be collected in one of several methods, or you may combine methods together to provide an overall richer data set. Observation and note taking are essential tools of games user researchers, and you can collect a good amount of data just by paying attention to how players interact with the game. Additionally, researchers often use surveys to collect feedback during and after a playtest. Qualtrics is a robust survey tool but may not be available freely to everyone (some universities provide graduate students with a free Qualtrics account). Free and easy-to-use tools such as Google Forms and Survey Monkey will also get the job done. Remember, you’re not doing market research on the game, so make sure your survey questions address usability and gameplay. Areas to probe on include comprehension of game mechanics, visual feedback, and UI as well as attitudes towards difficulty and balance.

After the playtest, you might also hold a group discussion or one-on-one interview to ask about additional feedback. Be objective while asking questions, and avoid leading participants to come to a particular answer by asking open questions, such as “how did you feel when X happened?” instead of “did you like when X happened?” Start by asking about overall first impressions of the game, then work your way to questions that ask about the finer details, such as (but not limited to) difficulty, enemy/item spawns, weapon balance, level navigation, understanding of UI, and narrative comprehension. The Interaction Design Foundation provides a guide to conducting user interviews, which can be adapted to GamesUR. To give the interviewee your full attention, record the interview on your phone so that it can be transcribed/listened to at a later time for analysis.

The equipment required to conduct a playtest can vary depending on the goals of your test. Aside from having the gaming equipment necessary, you may find it useful to use a laptop for notes. Asking players to bring a laptop if they have one is useful if you plan to give them a questionnaire after the test. In cases where players don’t have their own computer, you may offer your own for them to use. If possible, have a recording app to record sound and gameplay installed. Windows 10 has gameplay capturing software, and there are many free apps to download on your phone to record audio. If you don’t have a laptop to use, you can print out surveys to give to players and take written notes.

Although playtesting is a valuable skill to practice, it isn’t always the best research method to use. If a game is in the greybox stages, or too buggy for a play session, perhaps your time is better spent on a heuristic analysis/usability review - this is especially true if the most recent build can be completed in under 20 minutes or so. For expert reviews, the game is evaluated solely by you, the user researcher. You play the game using a ‘cognitive walkthrough’, documenting major and minor usability issues. There are many web pages that get into the finer details of conducting a usability review. The books “Game Usability: Advancing the Player Experience” and “Games User Research” also provide advice from experts on conducting both usability reviews and playtesting.

Finding Players & Space for Playtesting

If you’re running a playtest you’ll need players too. You can post recruitment flyers around your university/community, or find a class whose professor would be willing to let their students participate in a GamesUR study. Another option is to find participants through social media or ask your friends to extend the offer to their friends. It’s important to use players that are not your own friends or family, since you want the sample to be as unbiased as possible, thereby not tainting your data by participants knowing you and your objectives.

Depending on the nature of the study, it can be difficult to find a room to test a game, especially if it’s a multiplayer game. For single player games, you can try asking the participants if they feel comfortable with you coming to their house to watch them play. This would be an ideal environment since it is a natural one. You can also opt for a coffee shop and find a quiet corner to test in. Alternatively, you can try using a room on campus. Universities sometimes let students reserve a room for a specific time. Another option is to use a public library. Often, libraries have rooms available to reserve.

Report Writing

Game developers need their feedback in a timely manner. Be mindful of their time and their upcoming deadlines. Your role is to review the game or playtest session, analyze the data, and get the report written and delivered as soon as possible. In industry, you will be working at a similar (if not, faster!) pace. Typically, within 72 to 96 hours is standard.

When it comes to writing a report, a good format to follow is the games user research report template as provided by user researcher Steve Bromley. This template can be adapted to report the results of observational playtests, any survey data collected post-playtesting, as well as for heuristic analysis and usability reviews. It can be adapted for Word document or PowerPoint presentation format.

Also, don’t forget to include things about the game that are done exceptionally well. Your assessment will likely point out several minor and major issues with gameplay, which can be demoralizing. By acknowledging the design’s strengths, you can avoid seeming overly negative and help boost the team’s morale. This also lets them know that a particular mechanic or system is successful!

Report Delivery

When presenting your findings, aim to have a meeting with the entire development team. Since GamesUR is not always well understood among indie developers, establish a connection with everyone on the team and answer any questions that may arise during the delivery meeting. Make sure you’re reporting information in a way that isn’t full of jargon. The information should be easy to digest and understood by anyone. Time is valuable and you want to make sure that the development team knows the key takeaways from your test.

Post-Report

It’s important to be upfront about your compensation expectations, and agree them before you begin. It can save awkward negotiations that may arise later on in the research process. Students and indie devs are usually working with limited resources. Since you’re probably just starting out in GamesUR, it’s a great way to gain experience and feedback to further refine your GamesUR skills. Ask the developer about being in the game’s credits, or asking the dev if they would be willing to provide a testimonial on your LinkedIn page in exchange for your work on their game.

In addition to asking these things, consider sending the dev a survey about their experience working with you. For instance, you can ask the developer to rate the quality of your research presentation, the research report, and level of professionalism. Additionally, you may want to ask what the dev found to be helpful and ask what you could do to improve for future work.

Conclusion

In this post, we’ve outlined several best practices for developing experience in GamesUR with local resources. Whether you work with students of independent developers, collaboration will provide valuable experience for the professional world of games user research.

This blog post was equally written by Jess Tompkins & Hannah Murphy. They both currently work in the games industry in user research. Opinions expressed in this article belong to the authors, and do not reflect those of their respective employers.