Analyzing and Interpreting Feedback From Your Community

Monument Valley is currently being discussed as a cautionary example of the dangers of charging for paid content updates on mobile. The game received a “barrage” of one-star reviews when it updated last week with an option to buy paid content. Bad reviews are bad… right?

I don’t know whether putting paid content in your mobile game is a good idea – that’s a question for someone else, or perhaps a higher power. I’m much more interested in how players reacted to the introduction of paid content in Monument Valley, how developers reacted to those players’ reactions, and how developers can learn to interpret feedback from their games’ communities more effectively.

Any update to a game is going to upset some of that game's players. This is especially true when a real or imagined currency is involved. It's natural for developers to focus on the negative reviews: game designers and programmers usually enjoy solving problems, and one-star reviews are like being handed a list of problems waiting to be solved.

That’s not how people work though. No game will ever make everyone happy, and no game will ever achieve 100% positive feedback and unlock the “Perfect Game” achievement. Since it’s impossible to please everyone, there will be unhappy players sharing negative opinions about every game, regardless of its quality. If the mere presence of negative feedback is not enough to go on, how do you take a wider look at player feedback and decide whether a change made to a game has been received well or not?

It’s important to first look at historical trends. According to AppAnnie, Monument Valley's most-frequently rated iOS version before the introduction of paid content was 1.0.3. This version has almost 13,000 ratings, with an average rating of 4.4 stars.The version with the most ratings after the introduction of paid content received a greater percentage of one-star reviews than other versions. But the perceived “barrage” was more like a brief yelp of discontent, which ustwogames was able to drown out by leveraging their dedicated fan community.

The version that was current when this tweet was posted, 2.0, now has almost 5,000 ratings, with an average rating of 4.5. This is higher than the 1.0.3 version’s average. If most players were really unhappy with the game, there would have been enough one-star ratings to have had a negative effect on the review average, regardless of ustwogames getting sympathy on Twitter.

Timing is also very important, and is tied closely to players’ motivations for leaving reviews. One-star reviews began appearing very quickly when players realized that the new levels would cost additional money. The initial reaction was perceived as negative simply because of the timing. Players who were unhappy with the change reacted by getting upset and leaving a bad review to let the developer know they were upset. This takes only a minute or two at most, and after the initial backlash these reviews died down in frequency.

For players who were happy to pay for more levels, their reaction to the update was to buy the new content and play the new levels. Leaving a positive review does not come immediately or naturally, which is why ustwogames was smart to provide their fans with a call to action on Twitter.

It’s also important to identify who is doing the complaining. In the case of Monument Valley, the players who made fan art, dressed up as Ida for Halloween, created papercraft totems, and bought $50 prints from iam8bit were not complaining on Twitter and posting one-star reviews. That would have been a very clear indication of the existing community’s rejection of the new paid content. 

There is probably a large overlap between players who are complaining about the game’s new paid content and players who complained that the original game was too short and not worth the money. These are players who may not have purchased or played the game if they had full knowledge of what they would eventually dislike about it. They became “accidental players” in a way: they may enjoy some aspects of the game and may be invested in it due to sunk costs of time or money, but they were never part of the game’s actual target audience. It can be very confusing when players like this post feedback about a game they probably didn’t enjoy in the first place.

If a game is very good, like Monument Valley, this group is very small. Changing your game to win over a small group of accidental players can be tempting, but it may be at the expense of the games’ more-dedicated fans, the developers’ revenue, or both.

Highly-engaged fans appreciate Monument Valley for its level design, art, and story. One half-baked suggestion I’ve heard is to sell the additional levels as a new game, Monument Valley 2. But the developers have made it clear that these levels were designed to fit into the original game, and were never planned to stand on their own. Fans who appreciated the work that went into the story and pacing of the original would likely be disappointed by a new game that was just a glorified level pack. It’s much more important to keep your existing fans happy than to attempt to appeal to people who already don’t like your game.

Now let’s talk money: Monument Valley is already a huge success, selling well over a million copies despite having a price of $3.99. Would adding free levels to the game have expanded the audience of new players significantly? It’s doubtful that the few people still on the fence about purchasing the game will be convinced by 8 more levels. The game can still be beaten in an hour or two, so players who judged its value by measuring the length of time spent playing it are still going to be unsatisfied, free levels or not.

Based on this situation and many others I’ve witnessed, here are a few things to remember when looking at player reviews and ratings:

Thanks for reading, and feel free to post a comment and share your thoughts or stories about community feedback!