Designer's Notebook: Strange Agents Are Profiling Our Games!

In an obscure town on the northeast coast of England, in a small, anonymous office housed in what was once a Victorian school, a little-known organization is compiling profiles of video games. If you've ever published a game, and if your game has ever been discussed on the World Wide Web, they've probably got a file on it. They're simply called the Strange Agency. You'll be hearing more about them later.

Suppose you've got a great idea for a new game. You've put together a tech demo and a proof-of-concept prototype; you've created a great presentation full of whizzy movies with a pulse-pounding rock beat behind it; you have a really slick brochure about your game, all ready to hand to the publishing execs. (They won't read it, of course – publishing execs don't read much and some of them can't read at all – but if you've done it right, it has lots of pretty pictures and it'll look cool floating around their office.) You've even dug out a T-shirt without any gravy stains on it, so you're all ready for your pitch. What have you forgotten?

Oh yeah… competitive analysis. The most boring part of any product discussion. Why talk about somebody else's market share when you could be waxing rhapsodic about your vertex shaders? Trouble is, the publisher wants to know what they'll be going up against before they invest a few million bucks in your game. So one of the first questions they're going to ask when the presentation's over is, "Who do you see as your main competition?" And the next one will be, "What makes you better than them?" And you'd better be ready with the answers, or you're wasting their time.

Now if you're doing a shooter or a driving or a sports game, you can name your competition pretty quickly. They're all over the press: Halo, Burnout, Pro Evolution Soccer and so on. But what if you're in a less visible part of the market? What if you're in some funky cross-genre space… Extreme Ninja Barbie BMX Make-Over, let's say? How do you go about finding what else is out there, and whether it's any good? Trolling through the back issues of magazines, or trawling the Web in the hopes that you'll find something that looks kinda like what you're making? There are a lot of games published every year. A lot of games. In the United States alone, the ESRB rates over 1000 games a year, and of course those don't include all the indie products, web-based games, mobile games, and so on. You're going to spend a long time looking through those magazines.

I've always felt that market research is one of the game industry's weak points. For too many years we've relied on pouty women with big breasts and even bigger guns to sell our products to the young men whom we've always assumed were our major market – hell, our only market. There was always a much larger market of players who were less susceptible to breasts and guns, but we ignored them. Serious consumer or product studies are few and far between. For the most part it's done by guessing and by God, leading to arguments around the water cooler: “Listen, our customers aren't going to want any of that ‘story' crap. The only thing they care about is… [insert pet theory here].” These deep insights are all derived from what the designer himself enjoys, plus a few conversations with like-minded fanboys who, by definition, are atypical.

So about a month ago I went to the Animex conference up in Middlesbrough, home of the Fighting… well, they don't have a mascot but they do have a nuclear power plant, so let's call them the Fighting Neutrons… and there I heard a talk by an interesting new company that tries to quantify what's actually out in the marketplace in a searchable way. That company is Strange Agency, and they've built up a huge database, not just of the names of the games, or the traditional genres (which are fracturing ever more rapidly anyway), but of actual gameplay elements. Each game – over 20,000 of them so far – is profiled in terms of the player's experience of playing the game. Strange Agency can search their database based on what the player actually does, activity groups as they call them.

What this means to a publisher or a developer is that you don't have to trawl through the Web looking for potential competitors, because Strange Agency has already done it for you. Suppose you're thinking of making an underwater fighting game that involves swimming, stunt moves, and weapons – a sort James Bond-at-the-end-of-Thunderball scenario. Search the Strange Agency database for games that score high in those particular activity groups, and you'll find the games that you need to look into. Their Strange Analyst software has the ability to identify subgenres that you might have never known you were part of.

So how do they actually obtain all this information? You probably won't believe it. Say the word “semiotics” to the average game developer, and he'll go into standby mode. Semiotics is some high-falutin' academic thing to do with signs and symbols, all discussed in an obscure jargon and of no practical use whatsoever, right? Now say the phrase “computational semiotics” and he'll power down completely.

But that, it turns out, is the answer. Semiotics is about the relationship between the signifier – the symbol, or words people use – and the signified – the idea or concept that they're actually talking about. So Strange Agency's software crawls the Web looking to see what people say about games. It reads the text of game reviews and other material, digests it, mapping words to ideas, and from that determines what the key activities are in each game. The exact details of how the AI works are a closely-guarded secret, of course (although I bet the National Security Agency has something similar).

One of the brilliant aspects of this approach is that Strange Agency can profile thousands of games without having to play them themselves. A small company couldn't possibly play all the games released every year and profile them by hand. Instead, they benefit from the experience of real-world players who have already played the games and talked about them online. Other people do the work; the company mines the data.

When I first heard about this, an objection immediately came to mind: aren't the people who talk about games online atypical? Fanboys or flamers? So I asked the Strange Agent that I spoke to how they control for this, and it turns out there is some hand-tuning to exclude flamewars over trivialities. Serious game reviews get more weight than forum postings, because reviews are more likely to discuss the game's content in depth. By reading the reviews, the software builds up a picture of what activities occupy most of a player's time and attention.

Another question I had was, don't the results reflect the biases of the reviewer? And they would, if there were only one or two reviews online. But the Web is incredibly democratic, and anybody with a blog can review a game. Games are often reviewed dozens or even hundreds of times, all over the place – again, more than a single person can track down and read. So the software is reporting aggregate results, not individual opinions.

Yet another benefit of trawling reviews rather than, say, company websites, is that you get the opinions of players rather than marketers. Marketers sometimes make a mistake and put the emphasis in the wrong place in their advertising. We've all seen examples of this happening – the Black and White box gives the impression that the game is about Good versus Evil, when it's actually about repeatedly slapping a giant cow.

So what does this have to do with game design? Not much directly, but it may have a lot to do with what designs the publishers actually choose to fund. If you can walk into a pitch meeting with a Strange Agency report showing who your key competitors are and how you're planning to differentiate your game from them, you're just that bit better-prepared. And if you're a publishing exec – well, you haven't actually read this far, of course, but hypothetically – you might find that this kind of information helps to identify unfilled niches in the market and maybe give your product planning something more than a hunch to go on.

I know this column sounds like a big plug for the company, but I'm not affiliated with them and don't ever expect to be. They're so new that they don't have a track record yet, and only time will tell how valuable their services turn out to be. I'm just really intrigued by their analytical techniques, and I think they could provide some hard data in a business where most of the time hard data is conspicuous by its absence.

______________________________________________________