Why beta testers are so valuable - examples from real life

Our next game Hannu the Hunter will be released within a month and one of the last effort has been balancing the game using beta testers outside of our own dev team. We feel that this final testing phase has turned out to be possibly the most important phase of the development. That's why we wanted to share with you these examples of vital information that we have acquired in Hannu the Hunter beta testing.

Difficulty is hard to balance
When the beta test phase was about to start our honest hypothesis was that the game is likely too easy. After we had seen couple of players tackle through the first quarter of the game, it was clear that our assumptions were very wrong. It was in fact too difficult. Our mistake was to overestimate the players' speed of learning in motorics/accuracy type challenges. Here's a figure of what I mean:

It's fair to assume that the player's motoric skills develop fast in the beginning and there is some rather constant maximum skill level that most player's will achieve at some point after enough practice. Simple right? So all we need to do is increase the game difficulty gradually and after that offer the player varying but equally difficult challenges.

The problem is that Motoric skills tend to take time to learn. They develop when you sleep and it takes days, weeks or even months to master games that require alot of accuracy. Our game was increasing the motoric difficulty level too fast and the players got furstrated. Our developer team - that has been working with a game for months - was a poor test group when it came to determining a proper level of difficulty. They were just too experienced and could no longer imagine the fresh player's learning experience.

Often the only way to find the right speed of difficulty increase is to use beta testers. Especially when it comes to estimating player's speed of learning motoric skills, pure guessing is unlikely to work.

Intuitiveness
Good game designers can put them selves into player's role and make quite accurate guesses of what will feel intuitive for the player and what will not. 

Naturally we had to make tons of such intuitiveness guesses during Hannu the Hunter development and I dare to claim that most of them were pretty correct. The thing is that some of our game's intuitiveness based hints are extremely vital to the player's experience, and we couldn't just assume that our guesses were correct. 

That's where the beta testers were of great help. When you have seen handful of player's acting in game as you intended, you can be far more sure that your game is in fact intuitive. Then of course there are those few places where you notice that the player's get lost, and it is the fresh beta testers that are the only people who can show you those errors.

Device related game bugs
Halkologic Games is a smallish game studio and we have limited capability to test our mobile games with different devices. The beta testers (that usually use their own devices for testing) offer us a wider sampling of the game's performance on different devices. As usual, once again the beta phase revealed atleast one serious bug in our game. 

It's a little embarrassing to admit but in our alpha testing phase we forgot to test what happens to our game if the frame rate drops. So in beta testing phase some devices were not running with the typical ~60 frames per second rate and what happened was that on those devices the game physics got really messed up. It turned out that we had programmed some of our physics updates in wrong update loop (we are using Unity). That caused physics to fail on some devices, but to work correctly when played with our own devices, that were running the game with maximum frame rate. Luckly the issue was easy to fix after it was realized.

Summary
Our conclusion is that beta testing is extremely important. It's the last mile stone before release - the phase when it's time to come out of the developer's bubble and see if the game is actually good. Besides, it can be very inspiring to finally see people interacting with your creation.