Using Conventions as a Reality Check

I’m sure everyone has heard stories about people who spent ages working on a game, released it, and then wondered why the sales didn’t come rolling in. The core idea seemed solid, they couldn’t find any more bugs, and friends they’d tested the game on had said it was fun. What went wrong? Maybe the game didn’t sell because of the oil crisis that happened shortly before they launched. Maybe they should have released on Thursday instead of Wednesday. Or maybe, the game had some pretty major issues that they weren’t able to see as a result of being too close to it.

When you’re working on a game 24/7, you’re used to playing in a particular way and will build up a mental map of all of the things that are currently “wrong” with the game. These are likely to be pretty different to what are actually the biggest issues that people have when playing the game for the first time. There are many ways to test against this.

Some people will swear by metrics as a means for making sure this doesn’t happen (which I will argue against another time, when creating games such as mine), and I’ve done a whole lot of testing the waters through competitions and periodically sending new builds to old and new players alike. But one of the most useful things to me has been to continuously show the game off at conferences / conventions, because they’re basically a way to put the game in front of a lot of people in a very short space of time. Conferences are especially good at testing how well the game is able to hold peoples interest, when they’re surrounded by a significant number of other distractions.

Over the weekend, I traveled interstate to a convention called AVCon to show off the latest build of Antichamber. As per usual at these things, the response was very strong, though there are still some areas that need further work. This isn’t perfectionism, this is reality. My intention with this show in particular was to just leave people playing until they’d had enough, mainly so that I could be testing a bunch of later game content, but also to continue testing assumptions and find which areas needed the most attention before I show it off for the PAX10.

When people got up to leave, I’d ask them how long they thought they had been playing for, and their guesses were always way off. People who thought they’d been playing for 20 minutes had actually been there for 50. People who said half an hour were there for more than an hour, etc. In one instance, a person had been playing for 2 hours before I finally kicked them off to let one of the people crowding around play. These are all pretty good signs that things are working well in general.

Needless to say, the priority for what needed the most attention leading up to PAX Prime next month changed a bit, for the better. I’ve known how this happens for quite a long time, because what I thought was “almost finished” prior to E3 last year was actually “quite a while away from being done” after seeing waves of people play it at E3. Every conference I’ve been to has been a way to test for this kind of thing, and though it can delay the release of the game, it means that people are getting something of a much higher quality when the game is finally done. It is especially important for the kind of game that I’m creating, where a few false assumptions can dramatically impact what information people are learning / ignoring in the game, which affects how it feels.

Whilst at this conference, I had quite a number of people picking my brain about various things, some of which related to getting their game on Steam. This is something that has come up before, and my first question has always been the same. How does the quality of your game compare to the average quality of other games on the service? When your response to this question is “yeah it’s not as good I guess”, your first point of call would probably be to go off and resolve that.

Take a serious step back from your work and analyse what other games are doing “right” (as they’re on the service already) that your game is currently missing. If the art clearly looks cheap and dirty, either hire an artist or find a style that looks refined without costing more than you can afford. If your game looks really nice in screenshots, but has wonky controls, go off and work on that.

If you can’t spot any immediately obvious flaws such as these, though, start putting the game in front of people. Not just friends and family, or people who were already interested in the game for one reason or another, but people who have no reason to play your game other than the fact that it is in front of them. Random people who have better things to do with their time, and won’t continue playing for more than a few minutes if there are some pretty major issues with it. Conferences are full of these kinds of people.

Moreso than just putting the game in front of people, though, actually watch them as they play. If your game is being showcased at an event, but you’re not there as it happens, you’re wasting a pretty massive opportunity to understand how people are actually receiving the game. This is why I’ve made the effort to attend almost everything that has had the game on display around the world. It’s expensive, but when the alternative is releasing something that doesn’t actually work as it should, I’d just consider it an investment.

When people are playing Antichamber at an event, I spend more time watching their face than I spend watching the game. I already know how the game works, and an occasional glance at the screen will tell me what the player is looking at. But their face tells me what they’re thinking. This is important, because there’s generally a pretty obvious disconnect between what people say if you’re asking them questions, and how they were responding whilst playing the game.

Doing this can make it easy to work out why players are having trouble with something at one stage in the game, because you were able to see that at another stage, they completely ignored something important. They may have run past it, or briefly looked at what you wanted them to see, but didn’t take the time to actually understand it.

The other important thing to keep in mind at conferences (but also applies to any feedback that you get, even from just sending builds of the game around) is to listen to 100% of feedback and criticism. This isn’t suggesting that it is all relevant or that you should apply it all, and you may end up throwing away 90% of it, but you should only do so after you’ve seriously processed it all and have worked out what people meant underneath what they said. If someone says “this game is great but I was a bit confused about the controls”, don’t just hear “this game is great” and brush off the other comments.

Likewise, if someone asks how your game is going to target 4 year olds, appreciate what they’re asking, process it, and then be content throwing away that question, having at least thought about it seriously. The worst thing you can do is to only listen to comments that line up with your preconceived assumptions, or hear nothing at all. For the record, I already knew the answer to this, but still got a few young children to play Antichamber at AVCon, just to be sure. The game definitely isn’t for young children, for reasons I can clearly identify.

I shouldn’t really have to point out that there are always going to be exceptions to this advice. Yes, some games may genuinely be more difficult to test at a conference / convention. If you’re developing an iOS or Flash game (which I’m not), it may be cheaper to just release the game and update it as you go. But even in those instances, make sure you’re well aware of the specific reasons why a conference may not be worth your time, and are finding other ways of effectively testing against assumptions, rather than just throwing around statements about how “it’s not the right environment”. Technically, conferences are the wrong environment for almost any game, mine included. They’re still incredibly valuable.

The overall point of this post is this: When was the last time you bought a mediocre game and then raved to all of your friends about it? Whether your game costs $1 or $100, constantly putting it in front of people who know nothing about it and being realistic with what the issues are and how to fix them will avoid creating something that no one actually wants. I know this, because I could have released the game over a year and a half ago, and it certainly wouldn’t have been as worthwhile as an experience as it is today.

On that note, come by the PAX10 booth if you’re in Seattle. I’ll be there to show the game off once again, with a significant number of changes since AVCon.

This post was reposted from my blog.