How to Judge Game Design With Achievement Analytics

I’ve been taking a closer look at the analytics that come with videogames -- more specifically, the achievement rates on games bought on steam. Looking at them has given me insight into the hallmark of what truly makes a good game, and it has nothing to do with the number of reviews, accolades, and surprisingly, number of sales. For today, we’re talking about player retention when it comes to single player games, and what we can learn about how people continue to play (or quit) a game.

The Critical Contradiction

Before we begin, it’s important to mention that we’re not talking multiplayer or live service games in this post. Games that have continued content added in retain players differently than singleplayer.

Developers and gamers alike are often quick to point out what they think is the metric for success with a game. Maybe a title had a huge opening week and sold millions of copies, maybe it won GOTY, or every Youtuber and streamer can’t stop talking about it. The problem with these metrics is that they don’t tell the story when it comes to something game developers need to know -- did their game design work? Game development is a constantly iterative process; if you’re not growing as a designer, then you will fall behind. As we have said countless times, one success does not magically guarantee all future ones -- such as the failed “Bioware magic” piece on Anthem. And before anyone says it, the number of copies sold does not correlate to game design success as we will discuss in this piece.

There’s a reason why when we talk about game success it’s often hard to put into words why one game did well as opposed to another. That one sentence could easily frame multiple posts and books, and something I continue to try and answer. What we often see surrounding the best games is this sense of a full package: great gameplay, great aesthetic, and everything just kind of fits into place. When a game is good, it keeps people invested in playing it, akin to reading a good book or binge-watching an entire series. What I’ve learned and have seen is that just because people are talking about a game, or that it has a lot of reviews, doesn’t make that a great game. To study the quality of a game and how well it did, we need to talk about how the fans reacted to it.

Burning Fans

It’s time for a salient point that every developer needs to hear: Most people are not going to finish your game no matter how much praise it gets. This isn’t a slam against designers, but just the nature of the market and playing games. No matter what the game there will be drop-off when it comes to fan engagement. This can happen fast, it can happen slowly, but looking at the achievement list of games tells the tale.

People can stop playing a game for several reasons. Some people will load up a game, try it for a few minutes, see it’s not for them, and uninstall it. Maybe there’s a really hard section or massive difficulty spike that drives people away. For some, the game could be over when they’ve had their fill no matter how much more of the game is left to play. There is no one right way to make a game, as there is to play them, but drop-off is often the sign that something didn’t work right with consumers.

Again, even the best games released each year will have drop-off to them. What I look for when scanning achievements is the percentage of people who finished the game, and when we say “finished”, we mean to the credits, not 100% clear. The other metric is the drop-off spikes — when does a major part of the player base stop playing and how much was it? I’ve seen games that have been considered critical successes have 80% of their player base stop playing within an hour. There are games where less than 5% of the people who played it saw the game through to the very end.

What we can learn from these stats is how successful the designer was at keeping people invested via their gameplay and story. Major drop-off points can represent difficulty spikes, pain points in the design, or just where the game faltered in terms of engagement. With that said, there are a few other aspects I want to discuss and what they mean for designers.

Calculating Consumers

Before I reveal this next statistic, here’s a question for everyone reading this right now: Without looking below, in terms of finishing a game, what percentage rate of consumers would you rate as “good?”

From examining some of the most popular and not so popular games, I’ve settled on the following. If a game has 50% or higher clear rate, that is fantastic. Great games tend to average 40–49%. Most games sit in the 30–39% range which is good, and anything less than that would be poor.

Roguelikes kind of skew these results to some extent because clearing one run is often just getting started with playing the game. However, there are plenty of roguelikes with low clear rates. Whereas Slay the Spire was around 75% of beating the game once, and then the drop-off occurring when it came to its ascension mode.

Keep in mind that when we rate games in this capacity, the game’s reception is not factored in. Hollow Knight may be one of the best indie games of the last decade, but it has a poor clear rate sitting at 18% on steam. The question then: What can we say about Hollow Knight given its overall praise, but low clear rate? Again, the steam achievements give us a good idea. Hollow Knight had some quality of life issues when it came to playing it and learning it. From the get-go, the game lost 26% of its fans before they even got the first charm. Conversely, Dark Souls 3 lost 18% of its fans at the first boss fight, and Hollow Knight lost a total of 32% of its fans for its first boss.

That tells us that there was an issue with playing Hollow Knight that a good portion of people were not willing to overlook and continue to play. The next burnout spike is a 10% drop going from the first boss to the second, and then another 17% drop to the third. Here’s the thing -- those drops occurred for most players within about two hours of playing the game. Losing that much of the fanbase in that span of time is not good.

As we said, most people are not going to finish a game, but we also want to see if the burnout rate is a gradual drop instead of major points. Dark Souls 3 has one spike dropping from 82% to 63% at the game’s second boss. However, from that point forward, the drop off becomes a consistent amount-- lowering 2–4% for the bosses going forward. Even the game’s difficulty spike with Pontiff Sulyvahn had a clear rate of 45%

For every game, no matter what, the drop-off rate will stabilize for the fans who are “in it” over the long haul. This percent usually sits around 30% or lower for a lot of games. These are the fans that will try to do everything in your game before giving up. At the 10% or less metric, we have the super fans who will go for 100% clear. Studying this is where the challenge of knowing who to listen to about your game comes in. If the hardcore fans say that your game is perfect and not change a thing, but they only make up less than 10% of the people who bought it, you may need to rethink some of their suggestions.

Always Room to Grow

To wrap things up for this piece, there’s one final point I want to talk about when it comes to game development. There’s a tendency for people to become complacent with success in all walks of life. For games, we have seen developers fall into the echo chamber effect that their games are “perfect” or “the best game ever made” if they blow up. For games that become critical and commercial darlings like God of War, Death Stranding, Horizon Zero Dawn, Prey 2016, or The Last of Us 2, critical discourse is often swept under the rug in favor of just outright praise. It’s easy to lump critical breakdowns of games with people going on mad tirades about it. Or a developer will say that the people who didn’t finish the game “just aren’t fans.”

Despite being an amazing game, very few of Hollow Knight’s fans saw it through to the end

Unfortunately, not hearing about the issues is how game developers hurt themselves going forward.

There is always room to improve and grow when it comes to game design, no matter how great your previous game was. An amazing game keeps people invested while cutting out or mitigating as many pain points as possible. Right now, the game that has had the highest clear rate I’ve seen is Ori and the Will of the Wisps sitting at 52%.

Just listening to the praise and ignoring everything else will keep your hardcore fans happy, but not grow that base. I have not enjoyed any game from Bioware going back to Jade Empire, as I saw this echo chamber of not wanting to look at or improve playability concerns from someone who isn’t already a Bioware CRPG fan. Each game I played, the general gameplay just did not work for me. We often praise Blizzard in its heyday for listening to fans and critiquing their titles and designs. As we have said, there is a difference between someone not liking a genre and not liking your take on it.

There are always ways to improve the design, and keeping your head in the sand while other people are innovating and growing is a surefire way to fall behind. This is part of what lead to the indie explosion last decade -- indie devs making innovative and amazing titles that major studios did not want to risk money on. Listening to feedback and seeing where you can improve can go a long way towards making a better game. Remember, just because someone isn’t complaining, doesn’t mean that your game is perfect.

Finally, for you reading this. The next time you finish a game on Steam, check out the achievements page and see how many people cleared it. For developers, pay attention to the drop-off spikes and try to remember if those points were noticeable for you.

If you enjoyed my post, consider joining the Game-Wisdom discord channel open to everyone.