The Biases of Review Copies

This is a modified version of a post that first appeared on Meeple Like Us.

You can read more of my writing over at the Meeple Like Us blog, or the Textual Intercourse blog over at Epitaph Online.  You can some information about my research interests over at my personal homepage, or on my profile at Robert Gordon University.

---

Let’s talk about review copies since it’s one of those topics that consistently bobs up to the surface of the reviewer community like a dead rat in a sewer overflow.  Note here that I’m not going to talk about the emergent trend of ‘payment for reviews’ – that is a topic, perhaps, for another day.  In the meantime, The Thoughtful Gamer discussed this on his podcast and I am in broad agreement with all of the points he makes.

Whether you review electronics, games, music or books there is a constant undercurrent of criticism that goes along with accepting free review material from publishers.   It’s common practise but that doesn’t necessarily make it good practice.   There is a suspicion that reviewers are inclined to look more favourably on something they were sent for review purposes.    When the availability of review material depends, to a certain extent, on the indulgence of the manufacturer it seems inevitable that coverage will skew positive as a result.    How else can you keep the material coming in?

To an extent, this theory is even borne out by external analyses of online review practices.   A recent study by Review Meta, which is certainly not methodologically perfect, found out that for ‘incentivised reviews’ where the product was obtained for free there is an average skew of 0.38 stars towards the positive within the standard five point scale.   That may not sound like much but in a highly competitive marketplace, which Amazon certainly is, it’s enough to push a product from the midpoint of the rating curve to the top five percent.   That is a big deal, and there is a hugely aggressive business philosophy of manufacturers, at least those with little market presence of their own, targeting highly influential amazon reviewers with offers of free items and often actual compensation in exchange for favourable reviews.

I know that, because I am one of those ‘highly influential Amazon reviewers’, or at least I used to be.   I’m a member of the Vine programme which is an invite only system run by Amazon where select reviewers are given access to free items in exchange for fair and honest reviews.  I was also for a long time in the ‘Top 500’ of reviewers on the UK Amazon storefront, and I had an email address registered in my Amazon profile.  Those three things together meant than I would receive, on average, six to eight offers a day for free items to review.  Often these were things like USB cables or dubious herbal laxatives.  Sometimes they were things like dash-board cameras, wireless drones, digital projectors, and exotically large and intimidating multi-coloured sex toys.   That latter offer, as I’m sure you can imagine, was an absolute joy to behold when I opened the email at work.  All of these offers were unsolicited, and they were mailed out in bulk to dozens, if not hundreds, of reviewers at a time.  I know this too because often the whole mailing list was visible in a lazy CC every one of us could examine at leisure.

My response to these was largely to ignore them.   Of the thousands of offers I got, I think I only ever responded to twenty or so emails.  With those emails, I sent my list of terms and conditions which included things like ‘I don’t guarantee five star reviews’ and ‘I won’t accept compensation for reviews’.  Of the twenty or so responses I sent, perhaps five manufacturers were interested once they knew good reviews weren’t guaranteed.    Some of those asked the price I charged to guarantee the five stars.  There was no reason for the rest to take the risk on me – there are many, many Amazon reviewers out there that have no personal stake in the review system.  To be fair, Amazon have done a lot recently to help fix this incestuous reviewer manufacturer relationship but it remains the case that there is huge rating inflation happening behind the scenes and often in exchange for financial reward.  Even now I get hunted down by the makers of curiously eccentric random tat.  I get emails sent to my work address, and Facebook messages sent to the Meeple Like Us page.  All of them want reviews, and they are occasionally prepared to pay for them.  Sometimes they accost me on Linkedin (lol) or on Google+ (double lol).  They have found my personal email address more than once.  They’re in my house right now.  I can hear them in the walls.  I can’t get out.  I can’t get out.

Saw this one a couple of weeks ago in my personal Facebook inbox.

Don’t let anyone tell you that ‘provided free for review’ is something of which you can’t be reasonably suspicious.  I saw plenty of one star products being given five star ratings with various degrees of honesty in disclosure.   The Amazon vine programme itself is much better policed in this respect.  I have personally though been sent 4K video camcorders, high end wireless routers, expensive prestige pens and top-range kitchen equipment over the years.  I would like to think I have reviewed and rated everything fairly and honestly but who knows what impact a £2000 camcorder had on the star rating I might have otherwise given.   Who can guess what impact receiving that one has on the £800 camcorder I got sent the following year.   When a £400 wireless speaker system breaks, I shrug and put it in the cupboard.  When two of those, from the same manufacturer, break after a year and a half I am at worst slightly aggrieved.  All I can is say ‘I try very hard to be objective’ but I have no evidence to back it up and no reason anyone should believe me.

Within formal, media reviewer circles there’s another important consideration too – early access to review copies, ahead of retail release, is a hugely important part of attracting and maintaining an enthusiast audience.   When something like Twilight Imperium 4 is announced, it does Shut Up and Sit Down an immense amount of good to be able to say ‘We know about it, and look we’ve been working on a documentary!’.   That level of access to privileged material gives a marketplace advantage that outlets like myself and 95% of the others out there simply cannot match.   When reviewers are able to offer saturation coverage once a ‘review embargo’ is lifted those of us still waiting for the game to turn up at our FLGS might as well not bother.  By the time we’ve had a chance to play it, it’s already old news and the new hotness is out from its own review embargo.   For me, it doesn’t matter – Meeple Like Us is not primarily a review site and we’re not really focused on what’s new in gaming.  For others, the lack of a relationship with publishers means no early advance review copies, and no early advance review copies means they’re being put in a much weaker position in comparison to those that have them.  Losing the relationship with a publisher that permits this significant advantage is to risk, in some cases, losing financial stability and overall impact.

All of that said, I don’t think this is a major problem in board game reviewing but it is aggravating to see it entirely dismissed as irrelevant by those most obviously benefiting.  That either suggests a lack of some degree of self-awareness, or an uncritical acceptance of one’s own ability to withstand the subconscious biases that manage to plague everyone else.   Neither of these possibilities suggests a full acceptance of the real impact this potentially has, and neither inspires confidence that there is some formal control for bias built into the evaluative process.  Really though, looking at this in terms of ‘they received a review copy therefore their review is biased’ is the wrong lens through which to assess this.

Publishers are not giving away review copies out of the goodness of their hearts.  They’re doing it because there is a sound business case to be made for it – it is genuinely spectacular value from a marketing perspective.  For what usually amounts to $20-$40 dollars for a copy of a game they can have all but guaranteed coverage in outlets aimed at the very specialised audiences they hope to engage.   There are very few rates of return quite so significant as that of a review copy, especially when it goes to a big name outlet with a large and devoted audience.   For many publishers, a game not being covered is worse than it being covered poorly.  That’s especially true for board gaming where the inherent subjectivity of reviews is often considered a genuine detriment.  ‘Just tell me the rules, I don’t want your damn opinion’ is a common and disheartening refrain within this community.  If The Dice Tower or Shut Up and Sit Down don’t review a game, especially a high profile one, the community will ask ‘Wow, what was wrong with it?’.  That is inextricably linked to the trust their audiences put in their coverage.

The question you need to ask yourself here is then ‘Is the value of that trust greater or less than the MSRP of the board game they got for free?’.  If the answer you give is ‘less’ then really you need to be checking out other outlets.   Integrity doesn’t sell for cheap when your entire business model depends on the perceived authenticity of your viewpoint.  This isn’t the case for Amazon where your review is just a data-point in an aggregated overall rating – can you name the #1 reviewer on Amazon without looking it up?  Board game reviews where there is a known name attached to the output on the other hand – those reviews are artisanal.   We all live and function and flourish or otherwise within an intensely reputation based economy.   To worry about the bias introduced here is to ignore the far more powerful and demonstrable biases in reviews that come from self-selection, self-censorship (myself included) and simple expressed preference.  The problem isn’t really that people are reviewing the games they received for free in a favourable way because reputation is far more valuable than the cost of the product.  Having said that, the dismissive tone many reviewers take when the possibility is raised does come across as highly and suspiciously defensive.

It’s not then that individual review copies skew the end score – not in my view anyway – because the individual score a game gets doesn’t matter for the most part.   What matters more is the value the publisher puts on coverage in the aggregate and more than anything else that’s going to depend on the mutual benefit of the relationship.  Much of that benefit is gained from the perceived authenticity of the reviewer.  Publishers want their games well reviewed, but they want them well reviewed by someone without a reputation as a shill.  Nobody changes their buying behaviour on the basis of what a shill says.

There is though I think a more insidious and more problematic issue that comes from review copies, and it’s the issue of how the value of a game is referenced and assessed – or rather, how it isn’t.  I’ve had a number of long twitter conversations with people over the past few days.  I’ve also spent some time reading general discussions regarding review philosophies from Reddit and BGG.  It’s clear that for many prominent reviewers – maybe even a majority – price simply isn’t considered a relevant factor in reviews.   For those receiving a regular influx of review copies from publishers, it’s understandable – after all, price is literally not a factor.  It’s easy to say ‘This is a luxury hobby and consumers should make their own decisions’ and that’s very true.  However, it’s not helpful if you consider your role, as a reviewer, to appraise people of the necessary information that should guide their purchases.

I’m not talking here about something as straightforward as RRP – as many people in the twitter discussions noted this is both highly variable in a geographic and temporal sense.   It fluctuates intensely on a day to day basis as the elasticity of market forces snap into place and relax.  It’s even subject to substantive local variations where one shop may have no copies of a game and another has three they just can’t shift.

I’m also not talking about affordability because again – it can be reasonably said that is not within the scope for a reviewer to decide.   That which is affordable to me is not necessarily affordable to other people, and vice versa.  I don’t blink at buying a £50 game I want on a whim.  Other people don’t blink at buying a £5000 car.    It’s neither possible, not appropriate, for a reviewer to say ‘This is a very affordable game’ because affordability is relative.

I’m not talking then about price but instead I’m talking about value which is inherently a function of price but also of the business model.   I’m talking about reviewing a game based on what it could be rather than what it is, and then not disclosing what I’m buying when I pick up the box.    To be fair, many reviewers are already doing this when looking at weird edge cases, but not enough of them and not consistently enough for what is required.   Often when I’m reading or watching a review I’m unsure what I’m going to get for my money, and all I have to go on is a throwaway comment of only passing focus but staggering implication.  ‘Of course, to do any deckbuilding you’ll need to buy a second core set.  But on to the dice…’

What consumers need to know from a review is what they’re going to be buying and the extent of the commitment they’re making.  Are they buying a board game and it’s done, or are they basically signing up to a loot crate?  Is it a one night stand or a long term relationship?  Are they buying a deck-builder that provides too few cards to do deckbuilding?  Are they buying a miniatures game that doesn’t contain enough units for a battle?   They’re buying a box of something, but when are they getting the game that you’re reviewing?

The idea that cost and value are irrelevant to a review is symptomatic of the often deeply shallow coverage board-games get.   The idea that business models are not germane to criticism is an abrogation of reviewer responsibility.   Games don’t exist in a vacuum, and the business model of a game defines in substantive ways the undertaking a purchaser will be making.  Imagine you read a film review that said ‘This is the greatest exploration of the complexity of character interactions ever in movies’.  Imagine you went to see it, only for the movie to be fifteen minutes of Vin Diesel punching a wall.  Imagine you only got to see all the character development if you bought a second ticket.  Not in order to ‘fully appreciate’ it – but to experience it at all.   You’d be justifiably aggrieved that the implications of that didn’t get fully explored in a review that is supposed to be aimed at offering you real advice as a consumer.

‘People will pay for it so it must be okay’, is a stance that is often taken here – that fundamentally if the market forces are flowing everything must be fine.  But you know – heroin flies off the metaphorical shelves and we don’t, as a society, excuse it on the fact that ‘well, that’s the business model isn’t it?’

When review copies are provided in abundant amounts it distorts perception of this value system because cost genuinely ceases to be an issue to the reviewer.  That makes it more important, not less, that reviewers confront this bias and formally address the issue of value to the consumer.   Those receiving review copies, myself included, are experiential outliers.  Most people don’t have board games that come up with the rations.  They need to budget for them and they need to ensure that their budget is wisely spent.   They need to consider that in relation to their own circumstances – if you only buy one game a month and get four hours out of it, the decision to do that needs to be a conscious choice.  People need those of us who have played the game to tell what they’re getting and how much fun they can reasonably expect to have.

This is more too than simply saying ‘It’s a living card game so be prepared to buy regular expansions’, or ‘This is a miniatures game so I hope you enjoy having your wallet strip mined’.  There are differences in kind and in scale that go along with games even within these families of product.  You get a spectacular amount of Netrunner out of the bare box – enough that you realistically never need to buy any more.  You get a parsimonious amount of the Arkham Horror LCG in a core set.  A teardown of this is coming where I get very salty about the cost and value.  In this case, it also loses its replayability on an exponential scale.   You get enough out of the X-Wing miniatures core set to have fun, but the game everyone is enthusing about is £100 down the line.   These are not irrelevant concerns – they are fundamental to giving people the context they need to make an informed decision.  Sure, it’s difficult to approach this in a way that works for everyone.  The fun of games is impossible to pin down in a review format though and somehow we’ve all managed to make our peace with that.  It’s up to everyone to decide whether the value is sufficient for the price they’ll pay, but people can’t make that decision alone and they can’t meaningfully make it if it’s considered out of scope for review consideration.

Some reviewers, and I won’t name names here, are largely enabling a critical ecosystem which permits what is basically a bait and switch.   They convince you a game is marvelous.  They tell you they play with expansions.  They hint, but never emphasise, that you don’t get the full experience from the base game box.  They don’t come right out and say ‘This game doesn’t have everything I’m promising you.  You need two core sets.  You need four core sets.  YOU NEED EIGHT CORE SETS’.     They often don’t include discussions on how much replayability you’ll get and how much you need to spend to get more.   Most of us, at least if we’re savvy enough to be reading board game reviews in the first place, know what an LCG or a CCG or a miniatures game implies about cost.   That’s not true of everyone though, but the inaccessibility of jargon is a topic for another post.  It’s not enough though to assume value can be implied from the business model – what we also need to know is the scale, regularity and density of consumption needed to get an acceptable amount of reliable and replayable fun out of the game.

One of the useful things that comes from reviewing games you bought yourself is that you simply can’t shrug off value for money.   If a game is stiffing you on its value, you feel it in your wallet directly.  I bought the Arkham Horror LCG core game – spoiler alert, I loved it.  I bought the expansion, and then I found out that the expansion is really a quarter of the actual narrative cycle and to experience the rest I would have needed to purchase six separate mythos packs.  I sent that expansion back unopened because I think £120 is a ludicrous amount of money to spend before I get to find out how a story ends.   Other people might disagree, but if you don’t tell them upfront that’s what they’ll be spending then a disservice is being rendered.  This isn’t a game where you can wring days of fun out of the box.  This is a game where each scenario has a very, very limited shelf life.   This is an LCG where the novelty of the experience is its prime entertainment payload, and as such the business model is borderline exploitative.   That is absolutely germane and relevant to a fully considered and critical review of the product.  People need to know what they’re getting themselves into.  Sometimes people just want to know they’re getting a game and won’t be obligated to sign up for regular micro-transactions.

That this kind of system is so blandly accepted (and often congratulated) in the board game reviewer community is enabling practises that simply would not be permitted to fly in other spheres of entertainment.  Warner Brothers are planning to introduce microtransactions into Shadows of War and there are virtual riots about it.   I won’t buy the game now until such time as it is so cheap that it doesn’t even matter – I was going to buy it at release, but I was given the necessary information for me to make an informed decision.  Were I to have been offered a review copy of it, I suspect my view on microtransactions would have been somewhat more muted since I didn’t have to make any financial compromises to add it to my collection.

Value is integral to assessing and meaningfully critiquing a board game.   On Meeple Like Us, we do this for every single game within the socioeconomic section of our teardowns.  I don’t think we should accept ‘it’s irrelevant’ as an acceptable response to the effective lack of meaningful consideration it receives in reviews, and I think the regular receipt of review copies is one of the mechanisms that is substantively enabling reviewers to take this position.   Review copies are useful and overall more value to the hobby than they are detriment to consumers.  However, we also need to be mindful of what their availability does to coverage and we need to be self-aware enough to compensate for it.