Q&A: EEDAR's Zatkin On The Theory Of Achievements

Do better achievements in games lead to better sales? According to a new report by research group Electronic Entertainment Design and Research, there seems to be a strong, quantifiable relationship. Formerly an MMO designer on games like the original Everquest, EEDAR researcher Geoff Zatkin explained that Microsoft's Achievement system (referred to in a platform agnostic way in the research as 'accomplishments') has its roots, knowingly or not, in classic massively multiplayer design. Amongst Zatkin and the rest of the EEDAR team's findings were that games that reward points for online elements generate 50 percent more sales than those without, the same number as those that dole out points for user-generated features. EEDAR also found that a number of these options open to developers were going vastly underused, saying that "29% of all Accomplishments are Completion Accomplishments; one of the easiest to develop and integrate – leaving way for additional opportunities within the Accomplishment categories." EEDAR's message is clear -- more and diverse achievements lead to more engaged players, better reviews, and stronger sales. Gamasutra talked with Zatkin to learn more. Are you actually implying a causal relationship between better accomplishments and sales? Geoff Zatkin: Currently we have a really strong correlation for it, and we're working on the causation right now -- we're doing a lot of regression studies of the data. It's sure seeming to look like it right now. We went through every single accomplishment on the Xbox 360 by hand, putting it all in and tracking it with sales data and Metacritic and a couple of other meta layers of data, just really looking at what's causing it. In some cases, we looked at the number of players playing each one to see what's helping to really bring them in, bring in longevity, and it's been surprisingly striking. We didn't think we'd be seeing results so quantifiable. Do you think there's maybe a simpler correlation between developers that have more time to do more interesting accomplishments also having more time and a bigger budget to make games that sell better? GZ: Somewhat, but we found that it tracked across -- one of the ways we tried to figure out "better games" was by tracking in Metacritic as well, so we'll average in 50 or 100 critics worth of data. I think in some ways they are related, because people who think about accomplishments earlier, and put in a wider breadth and a bigger number -- who are obviously paying attention to it, and integrating them into their game -- are giving the consumers a better experience. People like rewards. I come from doing massively multiplayer games for ten years and I can tell you with a little bit of authority that people like short term rewards, they like long term rewards, little things that every once in a while go 'ding' -- carrots. They're just fun. So you also see a correlation that MMOs with better rewards -- visible rewards -- also track similarly to accomplishments? GZ: Yeah, one example would be the City of Heroes badges. It appeals to a very certain type of gamer. From both anecdotal and studied evidence, the people that it resonated with, it really resonated with. That's one of the interesting things about the Xbox 360, just looking at people deciding which system to purchase a game for if it's a multi-platform release. You have the option of continuing to give ongoing points for a score which, while meaningless, is still a fun thing to advance. You see a lot of consumers getting directed that way. Do you think Microsoft is breeding that certain type of gamer? GZ: I think they've already bred them in. With the exception of the Wii, which is an entirely new audience, the people who played games never really stopped -- the average age of the gamer is up into the low 30s, and we've been playing games forever. The next generation is coming in, and it's kind of expected. Microsoft hit that at just the right time with the Xbox 360 -- it was a surprisingly good move. Now you see that Sony's going to implement their trophies, and Steam has implemented their own version of the system, and I'll bet you dollars to donuts that in a year we're going to see the Wii doing something along the lines. MMOs have been using them for a while whether they're calling them by name or not. It's becoming a really interesting system, and that's why we wanted to study it. We built up a classification system and after looking at thousands of these, put them all together so we could readily group like to like and see by genre which ones correlated more with which groupings of games -- which ones were being used and which weren't -- looking for an expectation point. In the last bit of research we did, it was very broad and general. This one is very much going down into something that helps publishers and developers very specifically to make something that will track better with the people playing their games to hopefully both boost sales a little bit, and the longevity, and just give players a better play experience. Can you break down the different types of accomplishments you've identified? GZ: We broke them down into 16 different types, and we've got very specific definitions about what each one is. They go from action accomplishments which reward you for doing an action to something as specific as one related to user generated content, or games that do unlocks, time based or score based accomplishments, initiation, elimination, customization, collection, community, challenges, advancement. They really track to different player types. We've found that games that incorporate a wider diversity of accomplishments, they hit more people with the things that they really like, and they tend to do better, they tend to get reviewed better, which isn't necessarily terribly surprising. The same goes for quantity. It's interesting -- we actually have time graphs over the years since the release of the Xbox 360, showing that while it didn't start out that way, it's definitely that way now. As people learned what the accomplishments do, it's actually rather striking. When you look at sales figures for the first five or six months, it really didn't matter as much, but as the consumers have got used to them, they now really seem to be focusing on games that [have more and diverse accomplishments]. Which of those categories seemed to have the most impact? It actually really comes down to the genre, and we've broken it down on a genre by genre basis. If you look, say, at a music and rhythm game like Guitar Hero, the top five types of accomplishments that they're using are unlocking things, completing things, generating certain types of score, giving challenges like get so many notes in a row, and collection. We found that actually the ones that didn't correlate much with sales were challenges, collection, and failure/loss, so even though failure/loss isn't one of the most common ones, it's one of the ones that if you don't include -- just comparing with and without -- you're going to see that you're sales just aren't as high. We've actually made a cheat sheet, so, if you're a developer and you're making a game, you can look at what the competition is doing -- here's high point, low point, here's the ones that tracked the strongest sales wise, here are ones that don't really have any impact on sales, here are ones that users have feedback on that people aren't including. For Hardcore Eyes Only We asked Zatkin and EEDAR for specific examples of titles that seemed to truly understand the possibilities of accomplishments in game design. Said Zatkin, "We've actually pulled out really hard ones, like -- Final Fantasy XI? Holy crap! They're masochistic." "If you've never played it," he explained, "they have a level cap at 75, and for a hardcore player getting to that level is realistically probably a three month accomplishment. They also have at this point about 14 or 15 different classes. They give you 30 accomplishment points for every class that you reach level 75 with. I think we tracked three people that had gone through for all of them." Fellow analyst Shane Hebard-Massey compiled his own list of noteworthy achievements. "I’ve seen so many that the standard ones seem kind of boring," he said. "There’s lots of 'complete the game on X difficulty' or 'kill 5 other players in a row, without dying.' These stand out in my eyes." Amongst his favorites are Crackdown's 'Car Juggler' award, for keeping a car in the air for seven seconds with explosives, Orange Box's 'The One Free Bullet', for getting through the entirety of Episode One having fired only a single bullet, and Virtua Tennis's 'VT.TV Viewers', which gave players points for watching recorded matches: "something completely new in regard to Xbox Live gaming," he said. The bad, says Hebard-Massey, include Command & Conquer 3's 'No Life', which gives out points for playing 100 hours of multiplayer, and Transformers' 'Transform and Roll Out', which, he explains, is "earned by pressing the Y button." Finally, in his 'nearly impossible' category, Hebard-Massey pointed to the same Final Fantasy XI achievement, saying the group determined it was the single hardest game to earn all 1000 Points. "What we're really trying to do with this is just make something that's really useful to people," Zatkin said. "It comes back from when I used to do a lot of development. I like hard data, and it's so hard to come by in this industry. You get everybody with their pet theory on something, but it's always a theory." "That's one of the reasons my partner and I founded this company," he concluded. "We were interested in going after hard data so that before you spend 20 million dollars developing your next game, you could do a little bit of due diligence, look at what you were actually making -- the thing you're spending you're time and money on. There's a lot of waste in this industry, and I can speak from having seen a lot of it personally."