How to get the most out of your awesome early access community

The article was written by Filip and originaly posted on our blog thingtrunk.com, with his permission I'm glad to share our experiences with community-supported game development during Book of Demons early access. 

According to Steam Spy, starting from 2013, around a thousand games have entered Early Access on Steam. There are a lot of reasons why companies decide to launch a playable yet not finished product to the public. But two main of those are:

We, at Thing Trunk, have released Book of Demons on Steam Early Access because of both those reasons. The financial one is not very interesting so I will focus on the second reason, as different companies approach player input during Early Access in a different way. Let me tell you how we do it.

The game

For those who have never heard of Book of Demons, it’s a dark fantasy hack and slash game set in a pop-up book world of Paperverse. It’s a game that draws both from the old and the new. On one hand, it’s a sentimental journey to the roots of the PC gaming from the ‘90s (you could call it a tribute to Diablo 1), and on the other, it’s our attempt to breathe new life into the H&S genre. The game features multiple twists and changes to the usual action RPG mechanics, such as the card system that unifies spells, skills, and items, or the significantly changed movement and fighting rules.

Testing games in the past

Back in our casual games development times, our pipeline of testing a game on a larger audience was always the same. We created a playable demo of the game, which consisted of the first hour of gameplay of our product and we sent it to people on our newsletter list. This was our open beta. When a player finished the demo (either after finishing it or before that) she/he was given a survey to fill. After a week or so we read all the surveys and, besides having a lot of quantified data, we siphoned the ones that were most valuable. This, of course, doesn’t mean the positive ones – we were looking for perceptive players, who fit our target audience and had interesting observations and remarks.

From those surveys, we used to choose around 40-50 players and offered them access to the whole game. This was our closed beta. Out of those 40-50 players, around 10-20 were active and gave us constant feedback about the game. From our experience, after carefully addressing all the problems found on this stage, we could get a pretty polished game that was ready for publishing.

This approach worked quite well for match-3 and HOPA games, but you can’t really compare it to a full-scale hack and slash game in terms of complexity, balancing or flow. This, together with the fact that Book of Demons is not a casual game but a midcore title (which means that we don’t have such an easy access to a large number of our core audience emails), lead us to a conclusion that we needed to change our approach and create better tools and communication methods. Book of Demons landing on Steam early access was a perfect opportunity to up our game.

TTFT or Thing Trunk Feedback Toolset

When we have launched Book of Demons on Early Access, we needed 3 main things from the community:

For all of those purposes, we have added additional systems and features to the game. We wanted our players to have the easiest possible way to contact us whenever they encounter something interesting, weird, fun or terrible in the game. No more searching for a contact form on a website or writing emails to an address you don’t even know.

As usual, we could only implement a subset of the ideas we had. In the end, we went with:

In the following chapters, I will summarize each of those things and try to note what we learned from it and how it influenced game development.

Survey – sum up your experience

In Book of Demons, a survey opens when the player exits the game for the first time. This is something we knew has worked in the casual game market so it seemed natural to add. It’s a crucial moment when the player has seen the game, played it for some time and quits. And the reason he or she left is something you as the game developer want to know. In the survey, we asked about technical difficulties, some features specific to Book of Demons (card system, cauldron system, Flexiscope), game difficulty, player demographics and the most important question: how did the player like the game and how fun was it to play it compared to other games.

You can see the survey here: https://goo.gl/forms/4yCFeaAjBaVGD5jl1

But how do you track the answers changing over time? After all, a game in early access keeps evolving rapidly. In the first weeks after launch, we did patches every day, sometimes even more often. The way we approached that, we were able to divide all of the answers into 3 groups – each separated by a major and comprehensive update to the game. This way the surveys not only help us know what players think about our Book of Demons but also if our patches really improve the game.

Out of almost 9000 players we got more than 5500 survey responses.

Stability of our game (defined in the survey as not having any technical difficulties during gameplay) improved from 85% (early access launch) to 90% (current version). This is a pretty good score for a game in early access (at least in our opinion). We hope to make it even higher as we go.

After adding new and improving already present tutorials, the opinion on game help and tips improved from 92% to 95%.

We were working hard game balance. This was (and still is) tricky as we want to cater to both seasoned hack’n’slash players and casual players. Our philosophy in this area in that the story mode should be easy enough for occasional players to have fun but difficult enough so that hardcore players won’t get bored. After finishing this mode, four difficulty settings should suit everyone. This is still work-in-progress but from the survey, we can see that we are going in the right direction. In the current build, more than a half of players (52.7%) think that the difficulty is “just right”. It looks like we can make the game a little harder (23.3% of players says it’s a bit too easy) but what we need to focus on is almost 20% players stating that the difficulty was mixed (sometimes too easy and sometimes too difficult). Those spikes in difficulty are a real pain for us to smooth out, with random dungeons and bosses or different card sets.

Although the overall score (1-5) hasn’t changed that much (4.1, then 4.32 and 4.33 after big updates) but we have eradicated all of the lowest scores (0.4% then 0.3% and currently 0% of score 1 in our surveys). We are aiming much higher with the score but it’s good to know that currently, no participant considers our game as completely terrible 

Feedback widget – tell us what you think

Survey was something we were used to adding to our previous games, but it only gives you a one-time feedback at the end of the first game session. And we wanted to give players a chance to send us feedback at any time. So, we added something we call a Feedback Widget.

At any time in the game, a player can click a small button at the left edge of the screen that opens a window with a combo box with topics, text field, 6 emoticons to check his/her attitude about the issue and an optional place to write email address.

This proved to be a great idea! So far (three and a half months), we got about 2300 comments. Hidden fields give us game version (so we can filter the comments based on this) and topics/emoticons help us categorize each issue faster.

We got a lot of feedback this way. Players wrote us notes whenever they encountered something worth mentioning. Most of the notes are actual feedback for the game (bugs, feature requests) but there were a lot of (most of the time) positive words about the game. Here are some of my favorites:

So this solves not only the problem of getting feedback from players, but it can actually boost the morale of the whole team! Of course, sometimes people get a little angry:

Thanks to our players and the feedback widget, we managed to quickly patch a huge bug which allowed players to skip the final battle with the main boss, the Archdemon.

I agree with some of the comments on our Steam Community Discussion that this kind of feedback widgets should become a standard in early access (and maybe not only) games. It gives the developers a way to plug into players’ emotions/frustrations in exactly the moment when they are fresh. Sometimes, we get two or more feedbacks one after another, when a player changes his/her mind or explains the previous messages. We can observe the whole process which helps a lot when we are trying to find a solution for reported problem. I strongly encourage all developers who really want to gather quality feedback from players to implement (hopefully improved) version of this feature.

We read all the feedback daily and try to respond to everyone who leaves an email address. All cheers go to Adam for this one. In the future, it probably won’t be possible to answer everyone, when the game gets (hopefully) more popular, but for now we’re happy we’re keeping a low profile and can talk to everyone directly.

Crash reporting tool – does the stability improve?

One feature we have added during the first 2 weeks that wasn’t there from the start is a crash reporting tool. All the info about game crashes we had either from surveys, feedback widget or from the Steam community. But we know that those reports were just a fraction of players. Most of the gamers don’t write about bugs and problems. That’s why each time a game crashes and then is launched again it sends this information (if the player agrees to do so). Thanks to a crash reporting tool, we can see how the number of crashes decreases with every new version (or we can quickly react when it unexpectedly increases). Our current crash rate is 0.4% which includes all not game related incidents like power failures, OS crashes or intended ending of the process – anything that doesn’t include the proper closing of the game.

Beta feedback – all the data possible

“Beta feedback” is the name of our broadest data gathering system we have in place. This feature sends detailed feedback about gameplay events and style of each player (if he/she agreed to). These are information ranging from how far a character has reached in the game, how many deaths there were (and which monster killed him/her), which cards were used the most to a number of times a player buys cauldron contents. All this information helps us balance the game. Most of the important events during gameplay log some additional information that is, later on, send to us.

So far we use this mainly for gathering quantity data. This data together with feedback messages and Steam discussions led to us making some major changes in cards that were rarely used, to nerfing some of the most difficult enemies and polishing some of the basic mechanics.

In some (rare) cases, we analyzed someone’s gameplay step-by-step just by looking at the logged events. This, for example, together with analyzing “death density” (number of deaths on a particular level) helped us smooth out many difficulty spikes in the story mode. We have changed special mechanics present only on quest levels and removed infinite monster spawn loops (monsters spawning monsters that spawn monsters that can spawn monsters).

Sometimes when going through a large set of data you find something that intrigues you. When analyzing data from players exiting the game, we found out that a massive amount (between 6% and 10% depending on the game version) of players quit Book of Demons before ending the tutorial. This got us worried. Very worried. Why do they do that? Was it because of some technical issues? Didn’t they like the game? Was it too difficult or too easy? After a lot of speculations, we thought we should go straight to those “deserters” and ask them what’s going on.

Deserters! – why do you leave our game?!?

If a player quits the game before finishing a tutorial we display a special, super short survey with just one simple question: Why did you leave the game so early? It has a couple of prepared answers the user can select, and a text field for additional comments or options we didn’t think about.

Thankfully, it turns out that 82.4% of players leave the game not because they don’t like it but because they don’t have time at the moment and will resume playing later on. Sometimes the reasons are really important:

Steam Community

As this is our first game available on Steam we were not sure what to expect in terms of community. We knew that it will be important to keep a very close relationship with our Steam community so from day one Konstanty is always sharing time between implementing new features, fixing bugs and talking to people on Steam forum. And I have to tell you that the openness, patience and willingness to discuss of people there (opposed to demanding attitude that we were afraid of) exceeded our wildest dreams. A lot of players are not only reporting problems or issues with the game but help us shape it to be more engaging, enjoyable and fun to play. Players who suggest new features, understand when we say when they are not possible for us to implement (like rotating the camera in Book of Demons which is a 2d game) and discuss with each other the positive and negative consequences of proposed changes.

Frequently one of us wakes up, finds a new question about some game mechanics posted on Steam which is already answered by one of our veteran players. Seeing many players so active is highly motivating and we’re very proud to have such an awesome community!

Conclusions

I can’t say how different Book of Demons would be if we kept on working on it after launch without all the new tools that help players communicate with us. We worked isolated in our office with sporadic tests from our friends and/or family. Probably we wouldn’t have to compromise at all, we would be right on schedule as no external inputs would affect the way we work. But I honestly think that the game would suffer a lot. It would improve only in a biased direction and we would definitely overlook major problems we were able to sort out with player feedback.

I think that every minute we spent before the launch on designing and implementing all of the systems I talked about above, the time we are spending now on Steam Community forums and on reading all the surveys, feedbacks, and emails, was (and is) time well invested. In the end, we want our games to be fun not only for us, developers but for in the first place for players.