How do you review a F2P game?

Game reviews have typically served a dual-purpose. On the one hand, they perform the same function as a movie or music review - is this product any good?. On the other, because games have traditionally been more expensive than other media, gamers tend to place a much greater weight on ‘value’ (often expressed as the number of hours a game lasts, though I’ll come back to this), so reviews also seek to answer the question of “is this game worth the amount being asked?”.

For most of the time game reviews have been conducted, this worked pretty well. A high-quality game with good replayability gets a high score, while a lower-quality game with no replayability gets a lower score, and a high-quality game that isn’t very long might get a “rent this instead, or wait for a discount” verdict.

But in the mobile landscape of 2014, most games now are F2P, and F2P throws all of this out the window.

How, I wonder, is a reviewer supposed to say “here is what the experience of playing this game is like, and here is how much value it provides”, when those things change based on how much, if anything, the player spends? Perhaps a game that’s boring and irritating upon installing it actually becomes enjoyable after spending a couple of dollars. Or maybe a game is good fun without spending anything and the in-app-purchases are all poor value. When we talk about value, are we talking about an inherent value of the game, or the value of the in-app-purcashes? Or are we now less interested in value-for-money than we are in value-for-time?

Problem #1: What is the experience of playing this game?

Game reviews have always skirted a fine line of wanting to seem somewhat authoritative on something that’s inherently a subjective experience. But while the experience varied from player to player, at least the two players would be experiencing the same product. With F2P, even that isn’t guaranteed. To give a basic example, a player who buys a coin doubler in an endless runner isn’t experiencing the same product as someone who doesn’t, they’re experiencing a game where it only takes half as long to unlock new upgrades. Given that this is the sort of thing that would usually be hand-tuned by a designer, I’d contend that these players aren’t experiencing the same game differently, they’re essentially experiencing a different version of the game.

But what is a reviewer to do? They can’t possibly have a bunch of reviewers all buy different IAP’s, and different amounts of IAP’s, and have each write up their experience, if only because the number of different combinations would quickly become overwhelming.

Many of the publications I’ve read have taken a fairly logical approach to this conundrum - it may not be feasible to review every possible experience, but it is possible to review the experience that the average player will have. How? Because F2P relies on having a huge number of players, the large majority of whom will never spend any money. Hence if you review a F2P game from the perspective of “the user who doesn’t want to pay anything”, then you’ll probably cover at least 90% of your readers. It seems like a fairly reasonable solution, but it leads to the next problem.

Problem #2: The best payment is no payment.

To be fair, the fact that consumers prefer not to pay whenever possible is how we ended up in this F2P world to begin with, and it’s a pretty solid principle of capitalism in general. But we also have to acknowledge that professional game development requires professionals, which means that people need to get paid. The inevitable consequence of reviewing F2P games as if the user wants to avoid ever spending money, is that only F2P games that monetise very poorly (or not at all) can get the best reviews, thus making it a system that is quite hostile to the very industry it’s based around. Those paying attention to the mobile scene for a while now will remember the case of Punch Quest in 2012, which gained some very high reviews, a lot of downloads, and very limited revenue. If the “paying anything is undesirable” reviewers were actually able to shape the mobile landscape, it would likely turn into a place where every game similarly struggles turn a profit, because the only recommended games would be those where purchases always feel completely unnecessary. As luck would have it, most of the hundreds of millions of smartphone users are not reading these reviews (or the reviews on iTunes/Google Play it would seem), but that’s not really ideal either - the best situation would be one where the voice of the critic matters. This article is already a bit long without explaining why, but I may dive further into this in a later post.

Problem #3: No standards for monetisation.

Here’s an interesting case - 3 different reviews of last year’s Plants vs Zombies 2 (1, 2, 3). According to the first one, it has an “amazingly non-offensive freemium model”, while the second says that you’re “asked to cough up or play levels over and over until you're fed up with them”, while the third suggests that you “don’t play Plants vs Zombies 2, because it exists primarily to wheedle you for money”. Even though all three reviewers seem to have adopted the position of the non-payer, we’ve ended up with three reviews of the same game, scoring it 100%, 70% and 40%, based largely on how offensive they found the same monetisation model, or more to the point, their experience getting around the monetisation model. All three of them thus played the game similarly, but because these monetisation models are so new and vary so much from game to game, everyone has a different understanding of what it means to be ‘offensively monetised’, so a review may only be providing useful information if you happen to have the same taste in games and the same standards of monetisation as the reviewer.

Problem #4: The hobby game.

As I’ve covered before, there’s an increasing distinction in gaming between games that are designed to be consumed and games that are designed to be an ongoing hobby. F2P heavily favours the hobby model, and it’s evident that the games sticking around at the top of the charts are the ones that convince players to keep coming back every day for as long as possible. If this is what people are looking for, this presents another challenge for reviewers - What they really want to know isn’t “how much fun is this game at launch?”, it’s “how much fun will this game be to play every day?”

Kudos here to PocketGamer, who have at least tried to formalise some kind of process here. For many games of this nature, instead of publishing their review immediately, they write up impressions on day 1, day 3 and then give a score on day 7. Which unfortunately still depends on when day 1 is...

Problem #5: The constant update cycle.

Even if the above problems were solved, there’s yet another problem - some of these games stay in the public eye for months, even years, and can change drastically during that time. Updates on Google Play can even be done on a daily basis, and any game with an online requirement can even be changing aspects on the fly. It’s not uncommon to read a review of an older game and download it to find a number of key differences, hence making the review only accurate at the time it was written. Unless the reviewer happened to be in an A/B-Test, in which case their experience may not even be what all other players on the same day experienced. An optimistic view would be to see the review as a baseline, in the hopes that the game has only improved since then, but if we’re sticking with our mantra of “paying money is bad”, then this might not be the case, because most developers will be looking to improve their monetisation over time, which is industry slang for “trying to convince people to spend more money”.

With hundreds of new games being released every week, most review sites already have to hand-pick which ones they’re even going to review in the first place. The ability to come back and update those reviews down the line would require an enormous increase in manpower - it’s largely unfeasible for all but the most popular games.

An aside on ‘value’.

Most of what I’ve mentioned so far covers the idea of the ‘game experience’, the first part of what I introduced as the ‘dual-purpose’ of game reviews. The other part is value, and value is a strange concept now. We used to talk about value in terms of the $50-60 being asked from the gamer before they could play it, and often references were made to game length, as if the reason that games were several times the price of a movie is that the experience lasted several times longer. I’m not sure that was ever true, but it might have been seen as a good justification for the price regardless.

Only now, it’s more common to see free, endless games, where you can optionally pay to play less. This has so completely upended our impression of value for money that it can be hard for anyone who grew up with retail games to properly comprehend it. But along with the rest of the entertainment world (video, audio, print), we’re now in the position of actually having more free content to consume than we could ever make time for. So what we need, going forward, is an acknowledgement that merely passing time and always giving the player something else to aim for is not enough to recommend a game.

This might sound obvious, yet it’s difficult to criticise any F2P game these days without someone jumping in to say that they spent hundreds of hours in it without paying a cent, the kind of logic that would suggest that free-to-air soap operas were the pinnacle of TV content.

All of which is a fairly roundabout way of saying that the number of people who will get hooked on a game and feel a need to keep coming back every day isn’t necessarily an indicator of quality, and critics should regularly ask themselves what is motivating them to come back, and whether or not a game is really valuing the players time.

Problem #6 - IAP value for money.

It’s largely pointless to talk about ‘value for money’ for the majority of people playing F2P games, because they have no intention of spending any money in them. But for those that do, it also presents a challenge for reviewers. Because of the focus on non-payers, most reviewers see any kind of hard paywall as inherently negative. For the non-paying players (the large majority), everything you can buy in a game should either have a way to earn it through playing, and/or be completely unessential. Reviews of the top F2P games on PC (games like DOTA2 and Path of Exile) often praise the fact that the purchases are not in any way required, and you can have a complete and fun experience without spending a cent.

But this means that games are rewarded for making purchases luxury or vanity items, which are, by definition, poor value for money. Meanwhile a game that attempted to monetise valuable gameplay content, like in the old shareware model, is scored negatively by reviewers (and the non-paying players for that matter) precisely because the content is actually valuable, and undesirable to be missing out on. I might be mistaken here, but I don’t know of any huge successes selling content on the app store, and the top grossing charts suggest that consumables are the only business model that makes for a long-term impact.

It’s a strange system that rewards a game for only having purchases that generally aren’t worth buying, but here we are.

Where to from here?

I’ve been struggling to figure out how to end this extended rant, because these problems are not easily solved, but it would be rude of me not to at least try some suggestions. One thing that’s clear is that the standard game review model is much better suited to the old consumable entertainment model that it sprang from.

Here’s a few things that might be worth thinking about: