Do game critics and consumers agree?

Introduction

In cinema, music and literature, critics usually have different taste than the average consumer. In video games, for a long time critics and consumers have seemed to agree. Ask a critic and a regular player about their favorite NES games, and there will probably be a large overlap.

But as video games are becoming more diverse we should expect this to change. Here, I will argue that the change is already underway.

Description of the data

To do the analysis I will compare the average critics score of games on Metacritic.com and the score the game received by the users on the same website. The analysis is based on all PC games rated on Metacritic.com the years 2013-2019. The critics score is on a scale from 1 to 100, and based on the grades given by websites and magazines. The user scores is on a scale from 1 to 10 (with digits, so that 5.3 is possible, for example) and based on what users of the site think. In order to make comparisons easier, I have multiplied the user scores by 10, so that they are also on a scale from 1 to 100. Perhaps the two scales should be used differently, so that 5.3 on a scale from 1 to 10 means something different than 53 on a scale from 1 to 100. I will not pay attention to this, however, since my analysis has to do with changes over time.

From 2013 to 2019

Below are two graphs based on all critic grades and user scores for PC games released in 2013 and 2019 respectively. Every dot represents a game, when the color of the dot is stronger, it means that dots are overlapping, i.e. several games had the same scores. The farther a dot is to the right, the higher user score the game had. The higher up a dot is, the better the critics score was. The blue, diagonal line is a regression line with intercept 0 and slope 1. If the public and the critics were using scores the same way, the grades would be centered around this blue line.

As we can see, critics and the public largely seem to agree. But in 2013, most observations are close to the blue line, whereas in 2019, the observations are more spread out, and tilted upwards. What this means is that there is more difference in judgment, and critics are more positive than users.

If we look at the years between 2014 and 2019, we see that there has been a slow but steady development to this state. (I omitted 2013 simply to get a more elegant grid).

This trend is shown more clearly in the next figure. There I have plotted the mean score every year for critics and users. The trend is that the critics scores are going up, while user scores are going down.

Another way of describing the difference between critics and consumers, is to look at the mean difference in grades. (This is done by, for every game, subtract the users score from the critics score, take the absolute value, and divide by the number of games that released that year.) Unsurprisingly, the difference has become larger. (I would actually have expected it to have risen more.)

Let's look at the difference in more detail and focus on the years 2013 and 2019. Below are graphs showing the distribution of grades for critics and consumers. Games are divided into groups depending on what range of grades they were in, so "40" means "40-49", "50" means ""50-59", and so forth. The bars overlap so that the difference is illustrated by the color of the top of the bar. As we can see, critics give more games 60 or higher, and the difference increased in both absolute and relative terms.

Why is this happening?

What is the reason we are seeing the critics and consumers drifting away from each other? Here are some possible answers.

1. There is actually no drifting apart. It's just that Metacritic's user scores are unreliable and they don't say anything about the average consumer. While there may lie something in this, I still believe that the Metacritic consumer scores say something about someone – perhaps a subdivision of consumers.

2. As the trend of "games as a service" is growing, there is a greater difference between what games are like when critics play them, compared to how the games are when the users play them.

3. As the economic situation for websites is getting tougher, they feel that they have to give higher grades in order not to come at odds with publishers. Consumers obviously don't have this pressure, which is why they are less enthusiastic about new games.

4. Games are becoming more diverse. There's a group of games that are original, ambitious and lend themselves to theorizing. Critics like those games more than the average consumer does. This has been the case with other forms of culture, so why shouldn't it happen now with games?

Most likely, all of these factors (and additional ones) contribute together.