Opinion: Player Metrics Vs. The Vocal Minority

["The internet is a series of echo chambers where bias massively swamps any real signal about player behavior," says Spry Fox's Dan Cook, who questions the role of the squeaky wheel in the age of metrics. (Reprinted with permission.)] I continue to be fascinated by the disconnect between vocal players / reviewers and player metrics. The standard theory Over on Play This Thing, a user made a comment about how vocal players are like vocal customers that complain about service at a restaurant. The silent majority may never complain, but are very likely to share the same issues as the complainer. The theory goes that even though only one or two people are pointing something out, you should pay extra attention to these words of wisdom since they represent large customer service issues that threaten your entire business if left unchecked. This model is referenced by both vocal players and by self professed critics in order to boost their importance and authority. My experience Yet repeatedly I keep running into situations where this doesn't hold true. A vocal sub-segment of the community claims that something is horribly broken. Yet when you run the metrics of player behavior or run surveys of the silent majority, the complaint is either not an important factor or is an attempt to manipulate the game in favor of a very specific minority voice. Some examples (these have a lot more detail, but the gist will suffice) Example 1: There was a virtual riot (complete with organized virtual protests) in Realm of the Mad God when we changed the range of projectiles. Suddenly, players who could no longer kill enemies far off screen could, because projectiles went half as far. We did this because a group of players had decided to primarily play using the mini-map and onscreen tactics were going extinct. There were many complaints about how the game was completely broken. Yet fun scores and retention increased. Within a week or two of the change, people had moved on to talking about something else. Example 2: In Triple Town, experienced developers whom we show the game think that the turn system is a mistake. Traditional core players complain about the microtransaction system. Kindle players dislike the fact that this isn't a one time price for all you can eat. So one of the very first things we did was test how big an impact purchasing turns had on the player experience. The simple answer is that it doesn't seem to alter short term retention or fun scores. We are still collecting long term retention data (the game just launched), but it is immensely obvious that the big huge overwhelming issues facing the game have nothing to do with the primary complaint of the most vocal players. Our biggest issue is the first 200 turn experience around the tutorial. This is something that not a single player or review has mentioned. Example 3: In Realm of the Mad God, a small change was made to how a key for unlocking a special dungeon worked. This resulted in a massive fervor on the forums. A group of players was using these rare items as both a means of storing wealth and as a method of controlling who received rare drops. Since this group is the most dominant group on the forums, they created a large discussion about how the game was floundering. Again, the metrics showed either no change or a positive change. On deeper investigation, the elite group essentially was using their economic power to dissuade the majority of the players from accessing top end content. There was indeed something very meaningful going on (players really want to self organize into smaller groups and control who is in the group and who is out of the group), but it was occurring in a dysfunctional fashion. General themes There are a few reason why the disconnect occurs: Communities are internally consistent games independent of your game. The first is that the community is playing a game that isn't your game. Within a forum there are groups jockeying for power and influence. They'll say whatever it takes to advance their position independent of the actual situation in the game. Sometimes this is an organized rhetorical attempt to gain an advantage. Other times it is the result of limited perspective. To the players engaged in the game of running a mini-mafia inside of Realm, they were playing the game as their friends and the rules of the system dictated. The fact that "newbs" suffered was unimportant since they weren't meaningful players of the game at hand. Loss of old skills and knowledge or the requirement that players learn new skills always provokes an angry response. Change that isn't merely the addition of new content is almost always seen in a negative light. Learning has a real associated cost in time and effort. Anytime a game takes away a perceived skill, players feel that their time spent within the game is invalidated. They have immense difficulty imagining a better future as a result of the change and instead focus almost entirely on the intense sense of loss. The game is seen as an opportunity to promote an ideology. Being anti-"social games" has little to do with social games and almost everything to do with having an identity as a specific class of core gamer. How you buy and play games signals that you are part of an elite group and how you are not your Mom. In this light, any game that fits within a rough stereotype is seen as an opportunity to repeat a variety of message points. For core gamers, you get repetitive digs that X game is "a Skinner box," "immoral," "not a real game," "taking advantage of players," etc. Whether these points fit the particular game at hand is rather unimportant. Consider Hero Generations by Scott Brodie, a beautiful example of a painfully honest indie game trying to make the world a better place. Yet as soon as people see it is on Facebook, out come the various slurs that acts as a means of "dehumanizing the Other" and help a negative forum post cement the original poster's position in the tribe of inward-looking bigots. So you get hilarious comments like this one which one moment praises the game and then immediately starts repeating negative memes about Farmville:

Needed: measurements of actual player behavior. The internet is a series of echo chambers where bias massively swamps any real signal about player behavior. The more specialized the community and the more that its interests are focused on activities other than the game at hand, the less meat there is in its mental models and proposed solutions. This situation puts game critics at a huge disadvantage. Their natural authority is inherently low and their comments are deeply and systemically disconnected from any sort of objective reality. I continue to listen to passionate players, but I more and more preference measurable and observable player behavior over rhetorical opinion. There are indeed moments of truth that lead to improvements in the game to be skimmed from the morass of disingenuous deceitful, warped, navel gazing and self-serving game commentary. For example, the complaints about destroying Realm of the Mad God's accidental mafia helped crystalize thoughts about guilds. However, even in this case, the stronger signal was the behavior of the players, not the ranting itself. Why are teams embracing design empiricism based off measurable player behavior? Because the alternative forms of critique offer so little that is meaningful or truthful.