The Ethical Compromises of Paid Reviews

You can read more of my writing over at the Meeple Like Us blog, or the Textual Intercourse blog over at Epitaph Online.  You can some information about my research interests over at my personal homepage, or on my profile at Robert Gordon University.

---

In a previous editorial I spoke a bit about the biases of review copies and how the semi-financial relationship implied by their presence actually does likely introduce a number of biases into coverage.  Generally though I tend to assume that in this environment – where a largely unremunerated hobbyist ‘press’ dominates – there is only a minimal risk from the most overt manifestations of favourable coverage.   Where bloggers cannot rely on financial rewards they must instead seek more intrinsic motivation.  I did though mention another issue in the introduction to that editorial – that of paid for reviews.  I’m going to talk about that today and there isn’t a damn thing you can do to stop me.  I appreciate this editorial is going to make me a number of enemies, but let’s be honest – the line for that particular job opening is so long these days that it’s unlikely I’m ever going to get around to offering new applicants an interview.

First of all, I’m going to begin with a few disclaimers.  The first is that I will not be naming names in this editorial. I’ve seen plenty of people who in response to the Thoughtful Gamer believe that invalidates any argument that’s made around this topic.  That’s fair enough – everyone gets to decide what evidence they need to form a personal judgement.  However, most of my information from this post comes from a closed Facebook group, of which I used to be a member, and comments were made by participants of that group on the expectation of a certain level of disclosure.  Their expectation of disclosure did not, I am willing to bet, include their remarks being published full, unedited and extracted from context on a comparatively open blog post.   This is exactly the same situation as if you say to your Facebook friends that you were ‘totally bladdered’ the night before and so called in sick to work only for a ‘friend’ to then pass that info on to your boss.  I’ve had posts from my Facebook profile posted to Reddit before, without my knowledge, and while I don’t particularly care about the post being made public I do care about the violation of implied trust.   While I can provide sourced quotes and evidence for everything I say, I don’t believe it would be ethical to do so.  Plus, as we say in Scotland, ‘Ah’m no a grass’.    I will however put a link to the discussion so that anyone already permitted access to the comments can corroborate what I say should they be so inclined.

Importantly too, I need to stress one point that often goes missing in discussions about this topic – this isn’t about people being paid for their efforts.  It’s about where the payment comes from.  Many outlets such as Shut Up and Sit Down, the Dice Tower, No Pun Included, Rahdo, Actualol and more source funding for their efforts directly from their audiences.  That’s marvelous, and the model everyone should follow.  I wish them well, and have no quarrel with them or any other outlet that has well aligned their work and their compensation.  Similarly with outlets like Watch It Played – they can take money from whomever they like because they’re offering purely instructional content.   Lay down your arms, good ser knights, we are at an accord here.

Before we really get started we need to first address the unhelpful conflation of terms that tends to undermine these kind of discussions.   Here I constrain my commentary to coverage that is by nature a review – that is to say a critical overview of a piece of work.   Reviews are of necessity evaluative, subjective, and driven by personal opinion.  There is no such thing as an objective review although as far as is possible a review should be informed by familiarity with the wider context in which the product functions.   Important here is that it’s not the case that a review is being rendered whenever an opinion is expressed.  ‘I like the art in this game’ is not a review – it’s subjective, but it’s not evaluative.  ‘The art in this game sucks’, is a review – a poor review in both senses of the word.

We use the word ‘review’ too freely in discourse, and that’s a problem because this definitional looseness both excludes and includes a lot of content it shouldn’t.   Definitions matter because they can be a shield against accusations of impropriety.  ‘This isn’t a review, it’s a preview’, for example, or ‘This is an editorial, not a review’.  It’s possible for a piece of work to be both.   It also means that content that is genuinely exempt from the ethical complication we’ll discuss later can get lumped in with the rest – ‘instructional videos’ or ‘Kickstarter news’ as two obvious examples.   The context in which an opinion is expressed is important here.

However, while this is a useful starting point for discussion it’s also important to understand that a review also has an expectation of audience that goes with it.   Here’s where the problems tend to emerge.   In the most common form, a review is a piece of consumer guidance – something that is made available to help others meaningfully evaluate a cultural or physical product. The hope is that it will inform buying or consumptive decision making.   Reviews of restaurants, for example, are there to tell people where they should consider spending their money when they go out for a meal.

Most reviews are ‘customer facing’, but there are a subset that are aimed at more specific, and often hidden, audiences.  For example, an internal product review or a review of business practises within a corporation.   These are also evaluative and subjective, but they are also often motivationally driven.  ‘We are instituting a review to see how we might develop more inclusive hiring practices’.  Such reviews are aimed at achieving a goal, and that goal will often put constraints over what may be considered within the scope of the work.

Taking these together means we have something of a working definition of a review – it’s subjective and evaluative, and the expectation is that it will influence the decision making process of the audience.    It is not purely factual – it is always opinion based, but those opinions will ideally be informed.     It is critical in the proper sense.   And importantly, it has an implied or explicit audience that should be trustful that the content of the review is delivered with their interests in mind.  That’s a good place to begin because it effectively discounts a lot of board game coverage as outside the meaningful scope of discussion, and also grabs by the scruff of the neck a lot that might hope to be excluded by virtue of linguistic trickery.

If you’re doing news, or instructional videos, or let’s plays or anything where you are providing primarily factual information without editorial or evaluative slant then you’re fine.  I’m (mostly) happy with you charging whatever the hell you want for your coverage.   I mean, I’m not really but it’s difficult to put my finger on why in the context of a discussion about ethics so let’s pretend I’m okay with it.   Really if I dig deep enough into my psyche I probably don’t like it because the money is going to you and not to me.

That leaves everyone else, and there are three questions that anyone must address in any discussion of the ethics of compensation for review.

Who is your review for?

Generally this is whatever the audience for your output will be.  For this blog, as an example, the vast majority of the output is public for general consumption.   My audience is everyone reading this right now.  HELLO, YOU ARE ALL THE BEST.  However, I am currently in the process of doing private accessibility reviews for publishers – these are, for the moment, completely free and done primarily for the purposes of advocacy.   The audience there is not any of you, it’s the publishers who I hope to convince to make accessibility improvements.

To whom are you beholden for the production of the review?

This is where it gets trickier.  If you are beholden to your audience, then you can proceed happily onto the next post in this blog without giving the topic a second thought.  I receive almost no financial compensation from Meeple Like Us and what compensation I receive is trivial – a few dollars here and there from Amazon affiliate links.  Even that’s not entirely pure of ethical complication – we might return to that in a future post.   I don’t run a Patreon, or a Kickstarter.  Nobody paid me to do any of the teardowns on this site, and at the time of writing all but a small handful of games that have been covered were purchased with my own money.    What I’m looking to do with this site is to give people the information they need to make informed decisions about the games they might be considering, and for that information to encompass a meaningful amount of data within an accessibility context.   In that, I am beholden once again to the audience for this blog.   My reward for this work comes in what is essentially impact – this work only succeeds if people believe in the coverage and the integrity of the work being undertaken.   My audience, and my obligations, are aligned.  When doing private accessibility reviews, again the audience and my obligations align.

We’ve got a pretty handy phrase for when these don’t align – it’s called a conflict of interest.   If the answer to questions one and two are different, then we move on to the hugely important third question.

How well does your audience understand the implications of this misalignment?

That’s the real toughie because here disclosure is a mandatory first step but it’s often not enough because by and large there are massive gaps in the media literacy of many people.  That’s why hoaxes, fake news and urban legends propagate – because criticality of media and information is not baked into the bones of the majority of the population.   If it was we wouldn’t be having debates about the reality of climate change, the authenticity of the moon landings, or the safety of vaccinations.     If you have strong views on any of these subjects and would like to discuss them may I suggest you drop me a mail at [email protected].

As a populace we are not sufficiently critical of ‘facts’ by themselves, much less the complex set of interrelationships and compromising factors that go into their production.   Some of these are overt influences, such as ‘we will pay you to produce favourable coverage’. Some of them are more subtle, such as ‘let’s pay someone we don’t need to coerce to provide favourable coverage’.    Most people directly impacted by these forces aren’t fully conscious of them so it’s a lot to assume that those at several levels removed would be more aware.   Don’t get me wrong – a lot of people are very aware of these issues.    I would assume as a reader of an article like this you’re among them – after all, you need to have a certain interest in the ethics of disclosure to have even read this far.   However, many people aren’t.

For the simplest case (a review copy) simply saying ‘a review copy was provided in exchange for <whatever kind of review>’ is probably sufficient.  By and large people understand the context of that relationship and can weigh their own understanding of what it means for the coverage.  That however becomes much more complicated and ethically fraught when it moves into the region of being paid to provide review content.

So here we come back to this Facebook thread and the discussion within.  But again, before we go too deep I want to make a few points:

The Facebook discussion began with the price-point of $400 being cited for reviewers charging Kickstarters for coverage.   By and large, that’s what the discussion was – on the affordability, or otherwise, of that sum for project creators.    Some direct (anonymous) quotes:

Note that last quote where the issue of a distinction between review and more instructional content is brought in and later reinforced by noting that reviews are often provided by request of the person commissioning the video.   But still, the primary conversation focuses around the market economy of charging for reviews:

Note here that that this is explicitly, in these extracted comments, entirely about value and reward.  That’s the frame within which we need to evaluate the discussion – that it’s a business decision with the expectation of a return on that investment.  That makes complete sense from the perspective of someone investing money into a review.  It is simultaneously utterly incompatible with the duties generally expected of someone offering a piece of commentary that has an evaluative component.  That’s obviously true because honest evaluation will often be negative.   As such, the producer of a paid review is beholden to a patron (the publisher) to produce a review that supports the bottom line of the project.   As one commentator notes:

“So I’m a little confused. I’m a creator who will be looking for reviews very soon. If I were to use one of these services, and pay $400, is it possible that the review I get in return is “This game sucks?”

And the answer, by and large, is ‘no’.  Even if the reviewer is willing to post negative content about a project they’ll find it difficult to receive funding for anything else in the future.   Much board game coverage is generally uncritical, and there are a few reasons for it:

This is a self-selection bias that skews coverage towards the positive.   However, in cases where money has been accepted in exchange for review content there’s another reason.

Negative coverage isn’t going to inspire anyone to send you a pay-check in the future.

For a review copy, the publisher is out the total landed cost of the game.  That’s not an awful lot in the grand scheme of things.  Even if a review translates into a single sale at full retail price that money has been made back.    There is such wide variation in what people consider to be fun that if outlet A hates it, Outlet B might love it.  We don’t like Love Letter, but we’re all but unique in that respect. We detest Hanabi, but again it hardly matters because we’re a lone voice of rationality in an otherwise baffling chorus of unsettling and upsettingly universal praise.

A negative paid review, usually from a company unable to otherwise distinguish itself in a crowded marketplace, can absolutely sink a project.  Who backs the Kickstarter of the game they just saw slated by a respected voice in the game ‘media’?     You’re not starved for choices – there are more games released on Kickstarter than anyone but Scrooge McDuck could possibly back.   The money you don’t spend on project A can quite happily be diverted to Project B.  That means then that those paid to review content are absolutely, indisputably incentivised to be uncritical in any meaningful sense.  That doesn’t mean their actions will align with those incentives (people are all kinds of complicated) but it does mean that it’s difficult to trust positive coverage when it came as the result of a financial transaction.  Even if you’re given a free rein to do what you like in this specific instance of this specific game, it’s vital that you consider what it means for all future opportunities.  You’re not responding to a game, you’re responding to a potential review stream.  It’s not about the box in front of you, it’s about the relationship you’re hoping to forge.

Even that would be absolutely fine though if this were about ‘to whom are you beholden’ in isolation.  The problem is that you as a reviewer are just an expensive opinion attached to an outlet.   Your only value in this transaction is in that you give access to an audience that puts credence in your opinion.   You as the reviewer are not the product.  You’re the salesperson, and what you’re selling is the audience you have.   That’s what happens when your audience, and your obligations, don’t line up. You cease to be a reviewer, and you become a promoter.  You’re no longer doing reviews – you’re doing advertorials, or infomercials.    If you’re someone doing this, you undoubtedly disagree with this perspective.  You undoubtedly feel that your viewpoints are not for sale.  Maybe you’re right, but I don’t trust you and there’s no logical reason why I should.

Disclosure then becomes the common shield against this – transparency in the exchange of money for your viewpoint is the first step, but it’s far from the only step that need be taken if you aren’t to mislead your audience.   In essence what you need to be transparent about is the explicit purposes to which you are putting the members of that audience.  They might justifiably feel aggrieved that they’re considered to be a tradeable asset on your personal sell-sheet.  I have left more than one group when it was made clear that my membership had financial value to the organiser, and that’s okay – that’s what disclosure lets me do.   Disclosure by itself though isn’t enough – that disclosure needs to be visible.

The problem here then isn’t that board game reviews aren’t worth the money – not to toot my own horn or anything but seriously $400 doesn’t buy anyone a lot of my time or attention.  The consultancy rate for academics at Robert Gordon University is £675 a day, or roughly $110 an hour.  Often those hours are a damn sight less intensive than learning a game well enough to meaningfully review it and then writing two lengthy articles.    Despite some of the responses to the discussion point that I quoted above, the problem here isn’t the value of the service provided.   I have no disagreement with those that say writing, scripting, performing, recording and editing a video is a time investment that merits that level of compensation and likely a great deal more.    At my hourly rate for an average game, I’d be expecting around $1200 to fully cover my time.  And even that skews low because years of experience have led me to write very quickly.  The average word count per game of six or seven thousand words doesn’t take me all that long when I’ve worked out what I want to say.   $1200 is passing a lot of savings on to the person commissioning the work.  Honestly, I’m excellent value for money.

It’s not the amount that’s the problem, it’s from where the amount is sourced and with whose interests the review is expected to align.  With a paid review, it is entirely reasonable for the person commissioning the review to expect that they receive a service commensurate with the payment.    If what you produce falls under a marketing line item in someone’s financial spreadsheet, then you’re a marketer regardless of how you see yourself or your role.    The expectations that fall upon you are to justify the money someone spent on your services.

Here, reviewers passing off what they do as preview or editorial doesn’t inoculate them against ethical misconduct.   Coverage by itself can be an evaluative statement after all.  That’s not quite within the scope of this discussion, but it’s still germane – perhaps fodder for a future editorial.   Your audience deserves to know that you’re being honest with them, and honest about everything that matters.   If anything you say confers a value judgement within an evaluative context, then you’re offering a review and that means that your opinions are being offered for sale.    You may be honestly expressing your view about the thing you’re covering, but I as your audience can’t trust your opinion in this case and likely not in any cases in the future.  I can’t ever be sure that the opinion you’re offering me isn’t contingent on what benefit it brings you as the result of curating a favourable relationship with those funding you.  If you’re telling me that this Kickstarter looks amazing and I should definitely think about backing it, it doesn’t matter what you call it – it’s a review.    Who are you serving with that opinion?  Are you guiding me towards a wise choice, or your patron towards my wallet?

Again I emphasise here that this isn’t a case of me pretending to be the sheriff clearing out the miscreants from the reviewer ecosystem.   This is a tiny problem right now, with an equally tiny proportion of outlets falling into the scope of the discussion.  Nobody is getting rich here.    Most people doing this likely aren’t even making minimum wage from their efforts.  But paradoxically that’s what makes the topic so important.   We’re seeing ethical lapses here even with the miniscule amounts of money being pumped into certain reviewer outlets.  What we do here, and now, sets the bar for acceptable behaviour for the future.  This is a hobby that is growing at an explosive rate, and it’s entirely possible that five or so years in the future we’re not talking about what is essentially little more than fractional compensation for effort invested.   What happens when real sums of money start to be offered?    What happens when lucrative sponsorship deals start to appear?  What happens if we start to mirror the skewed and corrupted mess of ethical violation that describes a large proportion of video game journalism?  Sure, it’s a tiny problem now but this kind of thing can work to normalise problematic practices.  It begins with ‘Nobody is doing it’, to ‘Oh, one outlet is doing and nobody seems to mind that much’, to ‘some people are doing it’, to ‘everyone else is doing it’, to ‘Oh, I’m doing it too’.

If we can’t be trusted, as a reviewer community, to be stalwart against the minor temptations how the hell can we hope to stand against the major ones?   If we fail in the most basic of ethical responsibilities now, why should anyone trust any of us in the future? It’s easy to look upon the occasional offer of compensation for editorial viewpoint as a bonus – a reward for the hard work you are putting into offering content to the community.   Nothing in life is free, and the price-tag that’s hidden on the underside of that cheque is one that I don’t think most of us would be willing to pay if we could see the interest it accrues in the future.