Featured Blog | This community-written post highlights the best of what the game industry has to offer. Read more like it on the Game Developer Blogs or learn how to Submit Your Own Blog Post
Looking through Steam game data for little-known but highly-rated games, seeing what can be found and how common it really is to have a great game that doesn't sell.
Every now and then someone makes the argument that all really good games eventually do well, otherwise "if there are really good games that don't sell, how come I never come across one?"
That argument is always refuted by the statement that of course you don't come across games that are obscure, by definition. Which is itself then easy to refute by saying yeah, but surely you'd still come across one occasionally if you went looking.
Let's go looking.
---
First, let's take all the games on Steam.[1]
Then let's sort them by Evan Miller's recommended weighted method from here. That is, a lower bound of Wilson score confidence interval for a Bernoulli parameter. Basically, look at the score, but also consider how confident you can be in that score - i.e. how many reviews it has in total.
Now we have quite a nice list of the best games on Steam:
It's very similar to Steam's official top rated games page, with the notable exception of #1 game Wuppo and a couple of others. This is because Steam's official page sorts the results into "buckets", and to get into the top "Overwhelmingly Positive" bucket you need a score of at least 95% and at least 500 reviews. It's essentially a more primitive way of weighting the results so that games with few reviews don't top the charts. But it means that even though Wuppo has a score of 100%, since it only has 200 reviews total it has to slip in later past all the Overwhelmingly Positive games.
---
Next we want to fish out any "hidden gems": Games that people really like but that aren't very popular. Here's a basic formula I made up:
Take the weighted score above.
Leave games with less than 150 reviews[2] per year alone. Rate of user reviews is a decent gauge of popularity.
If the game gets more than 150 reviews on Steam per year, start penalising its score. I slowly rolled the score off linearly, penalising it 0.0002 per additional review.
Lastly, I added a release date limit of 1 January 2008. That one is a bit of a hack that shouldn't really be needed, but the problem is I'm basing "reviews per year" on the stated release date, but some games came to Steam recently yet list their original, earlier release date. This meant that without the date cutoff, re-released games from the 90s ended up looking like the most hiddenest gems ever.
Some older games that have been recently re-released on Steam, and have used their Steam release date as their release date, have ended up on the lists. I decided to leave them in since I would have had to go through manually to find them all. Technically their data is correct, for Steam.
Putting all that together, here's a list of top Hidden Gems on Steam:
Wuppo is again a good example of this system. It's looking like it'll get around 329 reviews per year since it has 200 reviews now, but it's been out less than a year. This is above the 150 reviews/year "no effect" cutoff, but the penalty also comes in slowly; so because it's such a good game and it's not that popular it still gets in at #15. Castle Must Be Mine and The House in Fata Morgana also make it in from the previous list, with the latter shooting up to the top spot now that all the big popular games are gone.
There's a lot of obscure stuff, a few cult games that you might have heard of but didn't buy, obviously (the Blackwell games, Escape Goat), and… Bejeweled 3? Apparently Bejeweled just doesn't sell on Steam. Although it actually has over 900 reviews, it's just that that's only 144 per year - it's pretty old too. You could choose to cap the total number of reviews in a Hidden Gems formula to avoid games like this.
I think it's interesting that the top Hidden Gems here tend to be games in less popular genres, or VR games requiring uncommon hardware. There are three theories I can think of for that. One theory is that great games in a popular genre always tend to sell better than great games in obscure genres, keeping them off the Hidden Gems list. The second is the exact opposite: That great games in a popular genre tend to sell worse due to the flooded market (apart from a few big successes), where having less reviews would push them down in the weighted lists below the niche games. My final theory is that games in popular genres may have more critical reviewers, since the players have a lot more games to choose from. In less popular genres, or in VR, people may be happy to be getting a decent game at all!
There's a lot more investigation that could be done here. In a more complete Hidden Gem finding system, you'd probably want to be able to view a list per genre, per tag, or based off what else you've played. That way you could view the complete list if you wanted to explore whole new genres you might like, but you could also easily find little-known games in your favourite genre.
---
There are also various ways you could decide what a Hidden Gem is exactly. In my system above, since it's based on the weighted score, a game with 95% positive over 150 reviews will still be considered "better" than a game with 95% positive over 100 reviews. Only after 150/yr does it start trending down.
Maybe you'd rather treat all games that fit in the "Hidden Gem" range equally. Here they are with the same rules as before (plus one extra - see below), using an unweighted score:
I also added a 20 total (not per year) reviews minimum threshold to this one, since weighting doesn't take care of it in this version and games with a very low amount of reviews just can't have reliable scores. This variant doesn't work so well in practice because all the 100% games get lumped together first, right down to the 20 threshold, before the 99% games start. Without the 20 review threshold you'd have a ton more.
---
Looping back to the initial statement of this article, how many games are out there on Steam that have great reviews but little exposure?
The answer is A LOT.
For instance one game I knew of already was NeonXSZ. It's a 6DOF shooter made by one person, with an 89% positive score on Steam and 63 reviews. It's always made me a bit sad that it got so little exposure while being so good, but it's telling that in the weighted hidden gems calculation above, it's #1147.
There are over one thousand near-unknown games on Steam, in all sorts of genres, that are subjectively better than NeonXSZ.
It might be nice if Steam had a way to easily find lesser-known but well-reviewed games like this, so more people could try them and see if they deserved a boost. This is the category of broken dreams. Lars Doucet does some of this sort of theorising and a bit of my knowledge of this stuff comes from his work.
One thing I will note, is that while there are games in popular genres that fall into the Hidden Gems category, so it's not a guaranteed ticket out of obscurity, it definitely seems more likely for games in less popular genres to get in there. If I had the data to split games by genre without manually going through them all, it would be interesting to see how much less likely a highly-rated FPS is to languish in obscurity versus say a visual novel. In fact it could be that popular genres aren't any less likely to end up in obscurity, but instead they have harsher critics who prevent them from being highly rated enough to make the top of the list! There are plenty of hidden non-gems.
Popular genres have a bigger audience but they're also more flooded with games to choose from. Wuppo is near the top of the Hidden Gems list and it's a platformer, one of the most popular genres of all. But there are so many platformers.
---
Finally, a couple of bonus lists.
There is actually some really interesting stuff in there. Dream Quest for instance is a deck building game with a glowing review from the creator of Magic: The Gathering. Although actually the Steam-listed release date for that one is set earlier than its actual release date on Steam, skewing it up in the charts.
After these there are many more games with great scores but even fewer reviews. Eventually the review counts become too low to get any real meaning from the scores.
These are of course the most popular really bad games. In the depths below these there is an enormous supply of badly rated games with few reviews.
And a bonus chart:
---
[1] As of 2017-05-09. I couldn't find any information from Valve about what they think re scraping data from Steam. I wrote a little tool to grab some basic details of each game on Steam and I only ran it once, and I'm not sharing the full data set here. I hope that's simple enough to be okay.
[2] Review counts used here are the ones Steam counts in its overall rating, which at the time of writing is only Steam reviews where the game was purchased directly on Steam. Reviews don't count for users who just activated a key on Steam - this is a measure by Valve to prevent shady developers gaming the system.
Read more about:
Featured BlogsYou May Also Like
.png?width=300&auto=webp&quality=80&disable=upscale)
