With around 11,000 games and over 125 million users, Steam rules the PC games business. That kind of reach and influence means developers both large and small theoretically have access to a massive amount of potential customers.

But that kind of footprint comes with a catch: such heavy reliance on a platform means that even ostensibly subtle changes to the system can affect the businesses of countless game developers.

This week, Steam owner Valve Software announced updates to the way its platform handles Steam game reviews that are submitted by players who obtained games by using a Steam game download key. “It's too easy for Steam keys to end up being used in ways that artificially inflate review scores," Valve said in a blog post, arguing the reasoning behind the decision.

Gordon Little had his first game release on Steam just a couple of weeks ago. Spell Casting: Meowgically Enhanced Edition racked up many appreciative reviews from players on the platform, leading to an overall positive average. Then Steam changed to its review policy.

"Yesterday I had a positive rating," says Little. "This morning it's negative."

Spell Casting: Meowgically Enhanced Edition saw its average score
go from positive to negative overnight 

Valve's decision has many developers puzzled and upset. It's harmed their ratings and their reputations, or put them in a difficult position in regards to their game's future.

"Our game is late in its life cycle and I don't think it will have a significant impact on sales, but 75% of our ratings suddenly vanished," says Nicholas Laborde of Raconteur Games, developers of Close Order. "It no longer looks like a game that several thousand people purchased; now, it gives the impression of 'it made it out of early access barely and no one bought it' which isn't the case at all, so that will permanently have an effect on people who visit our Steam page."

"I'm sad that my Kickstarter backers for our game Moon Hunters don't matter anymore," says Tanya X. Short of Kitfox Games. "They were our first supporters."

Dean Razavi echoed the same sentiment. Kickstarter game backers regularly receive Steam keys in exchange for their pledges. "My game Vidar was Kickstarted last year, and I'm currently in the middle of sending out Steam keys to backers," says Razavi. "Knowing that those individuals' reviews won't be counted towards my overall review score is a little sad."

It’s not just the review ratings for games that’s being affected. Developers also saw the number of reviews on their games drop drastically overnight. Customers give more credence to a positive rating with 100 reviews than one that has a positive rating but just 10 reviews.

"I have two games on Steam at the moment, and both took a dramatic hit in terms of the number of reviews, which was extremely frustrating because they didn't have a large number of them to begin with," says Nathan Meunier. Go To Bed from his studio Touchfight Games went from 33 reviews to 7. 

His solo project This Book Is A Dungeon went from 83 reviews to 55 overnight, and actually went from Mostly Positive up to Very Positive. "Does that help me? Yes. But at what cost?" he asks.

"It took a long time to build those reviews up organically," he says. "User reviews are one of the biggest at-a-glance metrics potential players use to gauge whether your game is worth buying. If you're a small developer with niche games that have less than 100 reviews to start with, waking up one morning to have those numbers slashed is brutal."

Why the change?

Valve said it made this move in an effort to curb review score manipulation. Some developers are believed to be giving out free Steam keys in exchange for good reviews. Still, will this new policy actually stop the flow of paid-for positive reviews? Many developers do not think so.
 
"I think what they're trying to do now is a first step, but it's easily bypassed by shady devs just giving money to shady reviewers to buy and review the games." says Little.

"I understand where the policy change is coming from, but the way it was rolled out was way too heavy handed," says Meunier. "I think a lot of developers, especially indies, felt blindsided. The core of what Steam is trying to do here is great. The updated user review policy as it stands isn't. A lot of players who buy our games legitimately (and obtain a Steam Key via other platforms) now don't have an opportunity to weigh-in. There needs to be some middle ground here, because as-is this hurts a lot of small developers who struggle to get reviews in the first place."

"I think it's good that Valve have started to privilege recent reviews, and if they keep doing that, it'll address some of the concerns people have," says Alexis Kennedy, a founder of Failbetter Games who's now Bioware's first guest writer. He says score of his game Sunless Sea was not significantly affected. "We ran a successful Kickstarter and gave out Steam keys, so the change doesn't automatically penalize community-centric devs," he says. "But I know it has penalized some of my community-centric peers, and obviously that means innocent edge cases have had a horrible shock."

What can be done to make this situation better? Many of the developers Gamasutra reached out to offered suggested solutions and advice for Steam to better stop fraudulent reviews while helping developers who might be harmed by their current policy.

Using the tools at hand

Steam already has some tools that could help them stamp out fraudulent activity. There is a flagging system  in place for reviews that marks them as problematic for various reasons, but several devs say that the tools seem woefully underutilized and poorly policed.

"Instead of taking Steam key reviews out of the score mix, I would like to see Valve do a better job of responding to flagged reviews," says Logan Williams of IndieWolverine, a PR firm representing games like GRAV, Goblins & Grottos, and Heart Forth, Alicia. "I run a Steam review audit service and have seen many instances of troll-like reviews, reviews that violate Steam community guidelines and reviews that attack developers directly."
 
"When I come across these reviews, I've flagged them (when appropriate) and I have yet to notice a substantial amount of flagged reviews being removed," he says. "Honestly, the best idea I can think of is to add staff to monitor/review flagged reviews."

Touchfight’s Meunier said a more focused approached would’ve spared negatively-affected developers. "It make sense to focus more on targeting people who grossly violate and abuse the system rather than throwing down blanket measures that impact everyone across the board,” he says. “That seems liked something Steam has already been doing, so why drop a hammer on everyone else?"

The audience and other developers, in using the flagging system, could point out instances of suspicious activity to Steam, helping them find the root of the problem with these bought reviews. For Steam, though, this would mean that more work would have to be put into following up on these flags, and teams would have to be put into place and keep an eye on things.

New reporting systems

Valve was obviously made aware of the problem with fraudulent reviews, either through developers and players pointing it out to them or through their own research. People are aware that this is a problem. So, one way developers feel it could be curbed is through adding an entirely new system of reporting, building upon that idea of flagging reviews.

"If the penalty for being caught rigging the system were banning, and a simple reporting system were put in place, the few developers who did engage in these activities would surely get the message soon enough," says Kitfox’s Short.

"I think a reporting system for gamers to report to Steam devs that are actively seeking out keys in exchange for reviews, and take harsh action on them," says Christian Allen of Serellan, developer of Epsilon. "Gamers will call them out if given a chance and an expectation of action. As always, it would also be nice to have a chance to give some feedback on these issues before they are implemented."

This proposed system, while similar to the flag system already in place, would in theory offer easier policing. 

Forum-posting style

Most developers clearly realize that Steam is a large company with thousands of games to police, though. Setting up systems that they will have to watch may not be in their best interest, adding further costs and employees to keep an eye on them. As such, they've come up with some ideas that will take care of themselves.
 
"To fix shady reviewers, reviews should be treated like forum posts," says Little. "New user reviews should weigh less than older accounts. Accounts with more games should weigh more than nearly-empty accounts. They could weight accounts based on play time and overall activity. The more active an account was, the more their reviews counted."
 
In giving more power to certain kinds of reviewers based on specific criteria, Steam would possibly be able to weed out some of the accounts that have only been made to leave paid-for reviews. At the very least, it would require a more substantial time investment from those leaving purchased scores. 

Changing the score's makeup

Some developers feel that the problem lies in the over-simplicity of how the overall score is put together, and that an overhaul of this system might be the key to fixing Steam's problems.
 

"The problem is that certain game developers seem to be caught up in a payola scheme of giving away free keys in order to boost their review score," says Lars Doucet of Level Up Labs, developers of Defender's Quest. "But pulling the weeds right out also takes up the wheat -- developers who just have a low number of reviews, and/or relied primarily on Kickstarter backers, preorderers, or direct purchasers."
 
"But why are they hurt?" Doucet asks. "Because the review score is prominent and visible, and because the review score has two ‘all or nothing’ critical mass zones-- at 50 and 500 reviews (the thresholds for the 'very' and 'overwhelmingly' adjectives in 'very positive' and 'overwhelmingly' positive, respectively)."
 
Steam appears to be currently taking an average score of its positive and negative reviews to find out what the overall score is. This may be fine for large sample groups of reviews, but for smaller games with fewer reviews, this makes it very easy to mess with the system. A glut of positive reviews can have a huge effect on the game's overall rating in this system, but Doucet offers an alternative.

"One solution to this is to use a different sorting algorithm to rank the review score based on the mix of positive and negative votes, so that losing one vote doesn't send you over the threshold and into oblivion," says Doucet. "Using the Wilson Score Method, as SteamDB does based on Evan Miller's famous article, ‘How not to sort by average rating.’ This way, games with a low number of scores can still achieve relatively high ratings and compete with the big dogs -- but at the cost of lower certainty in their score."

Could Valve benefit from SteamDB's "Fancy Algorithm?"

This lower certainty appears to be a problem as it may overly inflate scores as well, but for Doucet, this is not a problem for long. "This is fine, however, because the algorithm is self-correcting. A game with 50/50 positive reviews will shoot up the charts, but not quite to the top b/c the sample size is too small for complete confidence. In 5 percent of cases, the algorithm will assign the wrong ranking (because it's based on a 95 percent confidence interval)."

"In this case, games that were assigned too optimistic of a score have just gotten the boost they need to attract more samples, which will automatically serve to correct the score and move them closer to their natural place. The most important part of this rating system, however, is that it doesn't depend on arbitrary thresholds to assign ‘significance’ -- it's a smooth gradient built straight from the raw number of positive & negative samples."

Doucet's solution removes the importance of needing a number of review scores to skew your game's overall rating in one direction or another. Without this easy ability to change your overall score, the act of buying reviews would lose a lot of its appeal and fade away over time, and all with a change of algorithm. "This is the first thing I would do as it's difficult to game, and only requires a few lines of code that could easily be inserted into the existing backend,” he says.
 
Little suggests that Steam should factor in other review sources as well. "External reviews should count when the game doesn't have a lot of internal reviews," he says. "They should weigh more until there's sufficient internal reviews to balance it out. Once a game has 50-100 internal reviews, drop the external ones."

Keep calm and shout from the rooftops

Alexis Kennedy has a message for his fellow devs: If you feel that the changes negatively effect you, then speak up. "My main message has been DON'T PANIC," he says. "Valve are always pretty ruthless about these sort of changes, but they also always iterate with an eye on the long term, and they must know they have up-and-coming competitors. So let's give them clear feedback."