PlaytestCloud makes it easy to run diary studies without many of the typical difficulties that arise from conducting in-house research. We handle most of the usual annoyances such as finding the players, designing the testing environment, and figuring out how to keep track of what the players thought about your game – since players are required to think out loud when playing.

We designed our longitudinal studies product with these common struggles in mind. We want to make it easier for mobile game creators to gather the valuable insights that come from a well-conducted diary study, without the pain. All we need is your game, and some initial input from you, including who you want to test and for how long – the rest you can leave to us.

In Case You Missed It: Ian Bickley at Exient Talks Bake Escape and Playtesting Mobile Games

Last year at the Games User Research Summit in London we got a first-hand introduction on how to conduct a diary study in-house from Ubisoft Blue Byte user researcher, Jochen Peketz. Even though his talk doesn't apply to how our longitudinal product works, we felt there were many great insights for those who are conducting diary studies in-house. You can check out the whole talk titled "Diary Studies To Research Long Term Behaviour" on YouTube. It's a great watch for newcomers and experienced researchers alike.

Below we've put together a quick "Do's" and "Don'ts" list for conducting in-house diary studies, taken from Jochen's talk.

Do's

The planning stages

Do find players who meet your target group. You need to have players who like playing games similar to the game you want to test. You need to be selective with who you choose to let participate in your study.

Do pay your participants. A key finding by Jochen Peketz and his team is that players tend to play more than is incentivized – up to 50% more. Therefore, it is recommended that for a diary study you should pay players.

You can also try asking your paid participants if they would like to receive less actual money for more in-game currency. An example would be getting $10 less in cash for $20 worth of in-game currency. If a participant agrees to this, it can be a useful tactic for getting an impression of real player habits, because they'll feel as though they are spending their own money. You won't get business intelligence data from this, but you'll get a feeling for what works in reality.

Do set up communication channels. Skype is a great communication tool. Not only can you talk to individuals over Skype, but you can even set up group chats. Consider creating a group chat for the participants of your diary study to make mass communication easier.

Do give yourself a time buffer throughout the study. Usually, a diary study runs for two weeks or a little bit longer, and you need to have short breaks in time where the study doesn't run. If your diary study is a month long, you cannot expect players to participate every single day – maybe they have a trip or family event they need to attend.

Do set up a reporting system. You can use email, online survey, excel sheets – it doesn't matter. What is important is that you set up some way to collect data from the players.

Start with the statistics. It's important to get this done first, so you know what you're going to report and how you're going to report it. User stories are helpful here. Jochen Peketz and his team use user stories to help them understand the connection between the player experience and the data.

Do get all the data you can. And if you have access to a business intelligence person or team, have them track your participants each day, as self-reporting tends to be more error-prone.

Do give yourself time. You will have large amounts of data to sift through, and self-reported data can be tricky.

The diary study itself

Do brief the participants on what the study is about. Then send out a follow-up email that also includes the instructions – because even though you've briefed them, people tend to forget. Written instructions also give people a good reference point for later, so they can check back and see what they need to do: "Oh, I need to play five days out of seven and update the spreadsheet every morning."

Jochen Peketz suggests asking players about their three top emotions during their playtime. You do this so that players can offer a feeling about something they encountered within the game. It's also best if you pre-define which emotions, so that it's easier for you to use the data they've sent to you. It also stops players from offering any random thing as an emotion; i.e., tired as an emotion.

Do stay open-minded and empathetic. Imagine a player who says they broke up with their girlfriend the day before and therefore they cannot play today. You need to deal with situations like this with an open mind. It is different here than with a one-time playtest or if you conduct your playtesting in a group in-house. The design of a diary means that these situations will arise.

You also need to pamper your participants. Both technical and personal issues with your participants will occur, you need to be ready and prepared to handle anything that's thrown your way.

Do track the player's efforts. You should check to see if they played, how long they played and also to make sure they're not cheating or copycatting the previous day's results.

Do write down all the data you're getting as it comes in. Don't wait until after the study, because then the debrief will be impossible. You need to have all the data ready so that you know what questions to ask your players.

Do conduct a debrief. The debrief is where you can ask questions that weren't explicitly asked or answered during the study. You can ask open questions. Do the debrief over Skype – this is what Jochen Peketz recommends.

Don'ts

Don't underestimate diary studies. They take a lot of time. The reporting takes a lot of time. Also, don't forget that weekends mean that Mondays are essentially three days, and require extra time for the data collected over the weekend.

Don't underestimate the pampering needs. Your participants need to be pampered, and this can take a lot more time than you want to give it.

Don't forget to manage your human resources well. Make sure that someone manages all parts of your test. Otherwise, your reports will start running late.

Don't forget things! Don't forget your business intelligence and data tracking resources. You don't want to miss any information or have to track down data yourself. Utilize your business intelligence.

Don't forget to have more participants than you need. People will drop out of your diary study. So, taking a twelve person diary study as an example, it's a good idea to have three or four more people involved than you need. If you need twelve, get sixteen to be safe. Otherwise, your study risks failure.

Don't just give players in-game money to spend. They will spend it in ways that they never would in the wild. It won't reflect how they would usually play. You'll get wrong insights if you do this.

Don't track how players spend in-game currency either. It will lead to wrong assumptions, as they know that their time is limited and that any progress they make will be erased once the study is over.

Don't ask them if they could imagine doing something within the game. People can imagine everything, and if you ask people, "Could you imagine?" then you will get wrong insights. Jochen Peketz says that his team has a saying around their office, "Could you imagine jumping out of the fourth-floor window?" The answer is yes, but it doesn't mean any of them ever would.

A real-life example of this question, "Could you imagine?" question being counter-productive is when Coca-Cola asked study participants if they could imagine drinking the "new Coke" recipe and almost all of them said yes.

The rest, as they say, is history.

Are you interested in running a diary study?

This article only scratches the surface of how to conduct a diary study. If you're looking for more information, you can find a treasure trove of resources on the Games User Research SIG website.

However, if you're ready to go and have a mobile game you're interested in conducting a diary study for, then look no further. PlaytestCloud makes it easy to run your diary study! We offer flexible automated diary studies and can assist with every step of the setup. If you want to learn more, we're always available to talk.