Ask Gamasutra: 84 Metacritic need not apply

Over on Gamasutra's job boards, an ad recently popped up that has become a topic of debate over the course of the past couple days. Irrational Games, best known for excellent games like System Shock 2 and BioShock, is looking for a new design manager. Included are several requirements: "Six-plus years as a game designer." Ok, they're looking for someone with good experience. "Four-plus years of experience managing direct reports." Alright then. "Shipped a minimum of three games from pre-production through ship." Good, good. "Credit on at least one game with an 85+ average Metacritic review score." Wait what? So here's the question posed to Gamasutra and Game Developer staff: Should game developers base any of their hiring requirements on Metacritic scores? (Apparently, Irrational answered the question just a few hours ago -- the company removed the requirement from the job ad as we were formulating this article.) Kris Graft Editor-in-Chief Twitter: @krisgraft Let's examine briefly why Irrational would even make this a requirement. I'm sure HR's reasoning for this would be to sort the wheat from the chaff -- to find proven talent in order to form a world-class team. And it just makes the hiring process easier, because there are theoretically fewer applicants. But in order to form a world-class team, you don't need to add a stipulation that basically says "84 and below need not apply" when you've already set the bar quite high with other, more rational requirements. You're actively discouraging potentially strong candidates with a Metacritic requirement, all because of this misguided obsession over averages of arbitrary numerical values. Do I really need to mention some of the top-tier games that -- gasp -- scored an 84 and below? Some really smart and talented folks have contributed to games that weren't outright critical darlings. If studios are going to start using Metacritic scores as a basis for hiring new employees, maybe their HR departments should just cut out the middleman and recruit a couple dozen video game reviewers who will play job applicants' games, score them independently, then average out the results. Isn't that essentially what's going on here? (Also, there's a key takeaway for hiring managers -- if you want your job ad to get high visibility on the internet, just include a controversial job requirement! We'll take it from there.) Brandon Sheffield Sr. Editor Gamasutra; EIC, Game Developer Twitter: @necrosofty I feel like we're all generally going to say the same thing here if we're not careful. It's obviously stupid - games are made by a team, not an individual. Holding their individual work to a group standard, and a nebulous one at that, is beyond the pale. So in the interest of diversity, I decided to try to figure out a context in which this could make sense. What does the stipulation of an 85 percent Metacritic rating for a released game really say about a person? It says they have the ability to attach themselves to a good team, and see a game through with that team. That's all. As a designer, artist, or producer, that's incomplete information, because it doesn't say much about how well you did your job. But there are two disciplines for which it actually has some meaning: Recruitment and business development. If you grew a team that generated 85 percent Metacritic games, that actually kind of means something tangible, as a recruiter. As roles need filling as aproject progresses, if you can help find developers that fit, they'll contribute to the game's overall success, and that reflects well on your ability. In biz dev, it's a bit less clear-cut, but if you show that you can attach yourself to successful developers, you at least understand how successful developers work, and perhaps even how to identify similar qualities in other teams. You may be able to translate this skill to a new job identifying the right developer for the right job at a publisher. That's all I've got! Aside from those two disciplines, the idea of ascribing metacritic score to individual value is completely backward. Sorry Irrational! Your job ad is living up to your studio name. Christian Nutt Features Director Twitter: @ferricide It doesn't make a great deal of sense to me, frankly. It's bad enough that games are reduced to a number that is an aggregate of a variety of very flawed game reviews, just to start with. It's worse when this affects the business of studios, as happened with Fallout: New Vegas developer Obsidian, which missed its Metacritic target by one point and lost its bonus. But it's even worse when you're pinning that badge to an individual whose contribution to a bad game could have been amazing, or to a great game could have been insignificant. There are a lot of reasons games are good or bad. There are also a lot of games that are either good or bad, but their Metacritic scores don't reflect that. When you square that by adding an individual's contribution into the mix, it's going to be impossible to untangle. Mike Rose UK Editor Twitter: @RaveofRavendale I remember the day I was hired by Gamasutra like it was sometime last year. I had dressed in my best video game t-shirt and tattiest jeans, and made sure to turn up to the interview a few minutes early so I could get myself psyched up with phrases like "I am the controller" and "my body is ready." Things appeared to be going well - Kris Graft kept nodding and saying "tell me more bro", and Frank Cifaldi occasionally looked up from his retro gaming magazine when I said something particularly impressive, a wry smile forming on his lips only once or twice. Then I was thrown a curveball I was not expecting. "So which magazine did you work on that has an 85+ percent JournoCritic rating?" Graft uttered casually, while skimming through my extensive portfolio of sheets of paper with words on them. It turned out I had failed to spot this particular requirement in the job listing, perhaps because I had blanked it from my memory since it was so ridiculous a requirement. In essence, this requirement completely ignored all the great work I had done at my past magazines because a) said magazines weren't AAA enough to be on JournoCritic, b) other writers at said magazines didn't match me as an individual for talent, and therefore I had to bear the brunt of the less-than-perfect team I had been working with, or c) the magazine was a bit niche, edgey and/or new, and therefore didn't sit well so well with everyone who tried, despite being of immense quality. In response to Graft's question, I upturned the table, said "Good day to you sirs" and left without another word. Because let's face it, who would want to work for a company that believes this to be an acceptable requirement for hiring? [Also, this story was completely fabricated for dramatic effect. - Ed.] Frank Cifaldi News Director Twitter: @frankcifaldi In an ideal world, of course not. Metacritic scores are in no way an accurate indicator of someone's talents, some of the brightest and most talented people in our industry happen to ship lousy-to-middling games. I don't need to tell anyone here about the things that just inexplicably go wrong when you're making a game. But we don't live in an ideal world, and hiring managers have a lot to deal with. This is just one of the many shortcuts they are forced to take, and for a design manager, I don't think it's a terrible one. Besides, seriously, how many of the "requirements" for your position did you actually meet before you were hired? Eric Caoili News Editor Twitter: @tinycartridge Judging someone's output on a flawed system like Metacritic -- and the often defective systems of the critics contributing to Metacritic scores -- is a terrible, terrible idea. As other Gamasutra editors have demonstrated, plenty of remarkable video games have not received their due on Metacritic thanks to those problems, but there are also many other reasons outside of how games are reviewed that make this an awful way to go about hiring people. What about those competent designers at studios stuck on throwaway licensed tie-ins for the most part (Powerhead and Wayforward come to mind), but could put out some stellar games given the chance to work with original or better material? Or developers who spend years working on a project only to see it cancelled or their studio shut down? Or those iOS and social game makers whose titles never even get a Metacritic score? It's bad enough that developers are missing out on bonus payments because of publishers' dependance on Metacritic's sketchy system, but turning talented people away for a job they're qualified for by referring to a number that doesn't tell a tenth of the story? Terrible, terrible. Leigh Alexander Editor-at-Large Twitter: @leighalexander I realize most publishers feel that a Metacritic score over 85 is a reliable predictor of sales and correlates for all intents and purposes to quality, and I know most of them have data that largely supports this assertion, no matter how disinterested I myself am in score brackets. It's just that on big dev teams, it's nearly impossible to tell who's to blame when that score falls short. You might be passing up a triple-A talent who's been badly managed -- or hiring the total dud from an otherwise stellar team. Then there are those folks who made major contributions to famous games without ever being included in the credits for some reason or another. I think someone's project history can give a hiring manager a starting idea of what level of team this person is used to working with, which they can then use as a baseline to ask them about their contributions and strengths. I just think hitting a number as condition for even applying is narrow-minded. Patrick Miller Editor, Game Developer magazine Twitter: @pattheflip I can certainly see the appeal of including a Metacritic filter in your hiring process; it'd narrow down the number of resumes you'd have to read to only people who have worked on a critically acclaimed title, like N.O.V.A. 2 (90), Trainyard (90), Pizza vs. Skeletons (90), Sonic & SEGA All-Stars Racing (89), NHL Eastside Hockey Manager 2007 (89), or Pinball FX 2: Marvel Pinball - Avengers Chronicles (88). That way, you don't have to deal with all the folks who worked on games that didn't do so well with reviewers: Saints Row: The Third (84), Crysis 2 (84), Marvel vs. Capcom 3 (84), Fallout: New Vegas (84), Borderlands (84), DOOM II (83), Mario Kart 64 (83), Homeworld 2 (83), Mega Man 9 (83), Superbrothers: Sword & Sworcery EP (83), Rock Band (82), Gradius V (82), S.T.A.L.K.E.R.: Shadow of Chernobyl (82), Brutal Legend (82), Tekken 6 (82), Super Smash Bros. (79)...the list goes on. Tom Curtis News Editor Twitter: @thomascurtis Last I checked, Metacritic doesn't measure individual ability; it measures games, which usually come from not just one person, but an entire team. Why should a single developer be held responsible for the collective performance of his or her previous employer? Sure, you could argue that since Irrational's looking to hire someone for a managerial position, they need someone who can take responsibility for the whole studio, but even the best management in the world can't guarantee a high Metacritic score. You still need to have a talented team, a solid production plan, and of course, a fair amount of luck. After all, your Metacritic score is really just an arbitrary number derived from the press, and it doesn't take much to ruin your chances of receiving a "good" score. Look at Obsidian's Fallout: New Vegas. That game just missed Irrational's Metacritic threshold, yet overall, critics loved it. In fact, if you look at the reviews on Metacritic, most ended up docking points for one reason: the game's numerous technical glitches. Now, are all the developers at Obsidian supposed to be accountable for these problems? Surely there are designers working at the studio who didn't have a hand in those issues, so why should they be denied an opportunity at a new employer? It just doesn't make sense.