Analyzing the numbers and message behind a good GOTY poll

GOTY lists are a perfect blend of click-baity "Top X" list with actual informative content that help a lot of people decide what they will play next, while also rewarding devs for their noteworthy efforts.

These usually come in two varieties - Editor's / Jury's Choice and popular vote. But, unfortunately, most popular vote polls are very poorly done, for a few reasons:

1 - They present limited choices, so often you can't even vote for your actual GOTY.

2- To tie-in with holiday sales, they run on late November / early December, when there are still games coming out (Just Cause 3, Rainbow Six: Siege, etc) and people haven't played them all.

3 - They are simplistic popularity polls, that don't tell you much besides which game most people liked - sometimes without even giving you a % results or a full ranking.

4 - Usually only the Top 3 games are shown, the rest of the data is entirely discarded. 

Instead of just complaining, I've decided to do better.

So for two years now I've been handling the RPG Codex's GOTY polls, with a specific methodology: we list almost every RPG released, ask voters to rank them 1 (shit) to 5 (great), and then calculate the ranking using a Bayesian average - so that excellent games that few people played can compete against the AAAs.

And we publish the full results & poll data. Because everyone knows and likely played the Top 3 games, so deciding that one is 2% better than the other is pointless fanboyism / marketing - the real, useful question underneath it is "what else should I play? what have I missed?"

So here's the results of our 2015 GOTY voting, which listed 107 RPGs and was voted by 742 users:
 https://docs.google.com/spreadsheets/d/1nzVGja37jNVhItqkpFShcGPfPJ8q4jI5WXAOTtndCBg/edit?usp=sharing

It's an extensive, fully sortable google docs sheet, divided into three categories - Best RPG, Best "Tacticool" game and Best Remaster/Port/Translation - plus an overall listing.

I'd like to provide an example of how interesting this data can be.

Take Hard West - a tactical XCOM-like game set in a fantasy version of the Old West. From a quick look, it stis at #10 in the "Tacticool" category, so one would say it didn't do very well... and that's were most polls would stop. GG, better luck next time.

However, here we can see it was the second most voted game, only 6 votes bellow Invisible Inc., meaning that a lot of people were interested. And 49% of voters thought it was a good game, rating it 4/5 or 5/5! So what happened? Let's look at the graph:

A lot of people thought the game was good, but very few thought it was great, that's what happened.

For comparison, here's some very different graphs from other games in the same "Tacticool" category - Invisible Inc. was #1, Satellite Reign was #12, Blackguards 2 was # 17 and HOMM VII was #19:

Invisible Inc. won in its category, but it still had more people thinking it was good, not great - a sharp contrast with Underrail, Age of Decadence and Witcher 3, where 5/5 was the most popular vote.
Satellite Reign, on the other hand, was clearly considered mediocre-to-good by most.

Moving on to the lower portion of the poll, Blackguards 2 was mediocre-to-bad, with almost no one enjoying it. Quite sad, since Blackguards did well in our 2014 GOTY. Clearly people were disappointed. HOMM VII, however, was undoubtedly HATED. It's not a patch or good expansion that will save it...

That was just a quick look at the data, focused on a single category. There are many other variables one could cross here - number of votes, raw average, standard deviation, positive vs negative votes, etc...that's why the results sheet is fully sortable and included the full raw data from the voting.

While most people will dismiss this as a niche poll with little representation of the mainstream audience (fair enough, we did vote Fallout 4 the second worst RPG), I think it still offers a valuable insight into what +700 fans think of over 100 RPGs release (or re-released) in 2015.

I hope this proves useful, that in the future more websites improve their polling and that it motivates developers like the team behind Hard West. They got very close, all it needs is an extra push to convert the "good" votes into "great".

Meanwhile, players discovered some nice hidden gems, like Voidspire Tactics, Conquest of Elysium 4 and Swords and Sorcery - Underworld - Definitive Edition. Everyone wins.