Testing: Over the Horizon Radar for Game Dev

If you can only read one post today and haven't yet read Laralyn McWilliam's piece relating to testing, ignore me and go read it instead.

When I was working on my latest game I had an epiphany.  I realized that I had found something in game development I could not do.  Over a decade of sofware engineering expertise and I was stumped.

From the title of this post I bet you've guessed what that impossible task is: yep, testing.  The realization that a 7 year old girl could do something I couldn't was like a bucket of ice water in the face.  

Player testing is about the most important thing we can do for our game development: its like the over-the-horizon radar that is going to spot that iceberg in time, its the crystal ball that will tell me if my game is going to be a hit with players or be perceived as a bug-riddled flop.

When it happened I felt like that character in your favourite spooky tale who tries to walk down a mysterious road but keeps finding themselves turned around and going nowhere.  

I thought I was testing, it felt like I was making progress getting stuff tested, but in fact it was an illusion.  When I smugly ticked off another feature as working on my project plan, in reality the work that really lay ahead was accumulating, lurking hidden, gathering in the wings ready to blow out the development schedule by weeks or months.

And I know I'm not alone in this.

In the picture below, that's me on the right there watching a youngster help me out with testing my game at River City Labs in July 2014.

Well, no-one wants to do testing right?  So what is the problem?  Its not my job, right?

Trouble is I am an indie developer and as such everything is my job.  What's more part of my coping strategy for the stress of taking on that big risk that is developing an indie game is to tell myself that hard work will get it all done.  That doesn't work if there is work I am not able to do.

In my game development work here's my current list of duties:

See something missing?  Sure enough, I have removed testing from the list, because, as it turns out I cannot do it, even if I want to.

Even if for those who are not self described Indie Developers, being stuck back in the denial stages of believing self-testing is possible means pushing out work that is ready to blow up or be dead-on-arrival.  Taking the "its not my job" approach is just going to mean that we push functionality out which is not as good as it could be, and hoping the others in our team or company find those issues before the players do.

One quick thing: developers absolutely can do what is called developer testing.  That is, anything from writing unit tests (something that doesn't happen too much in game dev) through to writing scripts for an auto-testing tool to click, tap and swipe its way through functionality.  Developer testing absolutely is in our job description as a developer.  

But the kind of magical, invaluable testing done by actual players getting hands-on; while its not in our job description we ignore it at our peril.  As developers we have to break through our natural fear of watching the un-annointed and un-washed public getting their chaotic fingers on our games.  It is only then that we see them doing the things we never dreamed of, and never could dream of.

OK, OK, I get it - you're having trouble testing your stuff: have you tried not sucking at testing?

This is where I smile and point the finger one-eighty back at you, dear reader, because guess what - you are in the same boat as me.  Or at least (like me) you cannot test your own code.  It is just not humanly possible.  

Still think it's possible to do your own player testing?  Maybe if you're a split-brain patient.  More likely not doing player testing means shipping a game, which has a bunch of flaws that authors are just not aware of.

Let me describe the phenomenon of the enchanted road of testing failure, in case you have not experienced it yourself.  Let me illustrate with an example:  You build a game play level which is intended to show your players how some aspect of game play works, and throw up a hint balloon which is dismissed by tapping on the screen.  Then you can command your character to go and do stuff in the game.  So you test this stuff you just coded.

You launch the level, dismiss the hint and play through and its looking good.  You try a few different things, and there's no problems.

Now you figure it is all working, and its great, right?  So you hand your game to someone and the first thing they do is swipe across the hint balloon, the first touch dismisses the it, and the rest of the touches in the swipe get queued up causing the character to instantly start moving before your controller that responds to the hint balloon being dismissed has initialised.  Boom, crash, opera.

Another one.  You go to play testing and find that players complain of lag, even when you're getting 60FPS reliably.   What's going on?  Turns out that you built in a nice animation which shows the character moving from one modality to another.  As a developer you check the animation occurs correctly and the appropriate commands can be issued in the different modes.   For you the game is super-responsive.

But the players don't even see the animation and expect their commands in the new modality to work immediately.  This doesn't mean all animations are bad, and all players ignore all animations - look at Starcrafts tank going into siege mode or vikings moving between flying and ground mode.  If the "lag" is there for a purpose it needs to work in the game, and you as developer need to manage how players perceive it so it doesn't seem like lag.

And that is just a couple of examples of many.

Well, don't write stupid bugs.  Test better - you should have known someone could do that!

If we write code, we write bugs - end of story.  

The trick is finding them.  And to find them we need to be someone else - not ourselves.  Trouble is I cannot actually be someone else.  People can't lift themselves up by your own bootstraps, and I can't replace my brain with a freshly laundered version just when it suits me.  If you designed that button to be tapped, until you know better - that is until a brain that is not yours gazes apon that very button - you will do exactly to that button what is intended and not give it a second thought.

And until you've been confronted with the phenomenon a few times it is almost impossible to come to terms with it.  If you've been writing a game and have not gone through the experience of putting your game in the hands of an out-of-the-blue player, a complete neophyte to your game, you're in for an out-of-body experience.

I've seen other developers (I work out of a co-working space) go through this too.  First comes the look of consternation as they see the person testing their game "doing it wrong".  Then comes the mouth - I can see their jaw flapping as they try to explain to the player "their mistakes".  Finally in exasperation the developer grabs the device and shows them "how to do it properly".  If they haven't lost the player back at stage 1, then the player is dead to them by stage 3.

No testing is going to find all problems.  We don't have time for that anyway.

Given that you have a game at all, testing is in my opinion about the only way you can make it better.  If you don't test it with real-life, non-game-dev actual 100% human beings your game will go out the door with problems, many of which you might be blissfully unaware of.  We don't test because we want to find 100% of bugs - we test to make it better, to find the obvious show-stoppers, deal-killers and causes of day 1 bad reviews.  Its something that you have to make time for.

I know its hard to do, in part because its personally challenging and so tempting to think of testers as clueless noobs fumbling their way through your works of genius.  Its demanding on time, and requires organisation.

But there's a few ways now to do it; Apple's new beta program for example now joins TestFlight, for testing of iOS games; and you can organize test days by throwing on some pizza and drinks in return for willing test players coming in.  If you're at the big end of town, or can somehow spring for the cash required there's usability labs, pro-test suites and convention booths as well.

And, chillingly enough - its not something you can do yourself, no matter how diligent, resourceful and independent you are.  So here's a word of hard-won advice, make time for testing.  Its your over-the-horizon-anti-iceberg-radar.