[ This blog appeared originally on the Flashman Studios website. ]

One of the things we love about working at Vector Unit is the intimacy of developing games with a small team -- we just added our fourth employee in January.  Small teams can pivot quickly, and of course you get the fun and efficiency of having a close-knit group of people working together on a single project.  

Small teams do have their weak points, however, and one of those is testing.  By testing I primarily mean bug testing, which is when you play every little part of your game over and over again looking for glitches.  Bug testing can be a time consuming process when done properly...time that gets sucked directly out of our tightly-packed 4-person development schedule.  For Android developers it can also be an expensive process, as you struggle to maintain a library of all the latest phones, tablets and phablets that your game might be played on.

The other piece of “testing” is playtesting -- playing your game from beginning to end to get a feel for how it plays.  Is it too hard or too easy?  Are the tutorials easy to understand?  Is it fun?  It can be a real challenge to step back from the game you’ve worked on for a year and try to answer these questions from the fresh-eyed standpoint of a new player.

At some point after the game is functionally done but before you launch it, you really need to get the game out in front of an audience and solicit feedback.  Developers call the stages of development leading to launch “Alpha” and then “Beta”.  Coordinating a public beta can be a daunting task for a small developer, but fortunately Google has an answer for you, and it’s one we took advantage of this year when preparing to release our 5th mobile game, Beach Buggy Racing.

“Beta Testing” Setup on Google Play Developer Console

About a year ago, Google added “Alpha Testing” and “Beta Testing” tabs to the Production section of the Developer Console where you upload your APK for publishing.  Instead of just uploading it directly to the public, you can push your build to a smaller whitelisted audience.

In practice, “Alpha” and “Beta” on Google’s dashboard work the same way, except you can maintain separate audiences for each.  For example, you can push your Alpha build to a small set of internal and external testers, and then promote it to Beta where a bigger audience (like the members of a Public Google Group) gets access.  Rinse, repeat.  Then when you’re ready you can promote the build to production and have it go out to the world.

To manage who gets access, click “Manage list of testers”.  You add testers via Google Groups or Google+ Communities.  We use two primary groups:  One group is private,  invisible to the Public and invite-only.  We use this for our internal teams and B2B partners who might need access to the build in early stages of development.  Another group is our “public” group, which can be joined by fans, or really anybody (we have the join permission set to “Anyone can ask” rather than “Public” to control inflow.)

The group-based user management makes adding and removing users a breeze, since they can sign themselves up without direct management from the developer, and you can moderate the list and its permissions via the Google Groups dashboard.  

The beauty and flexibility of the beta system is that once players are part of the group, the app appears in each tester’s Google Play Store just as a regular app update would.  It’s completely transparent and comfortable for the user, with only a couple of limitations -- for instance they can’t write reviews of an app that hasn’t been fully published.  But they can get automatic updates, look at screenshots, see a list of “What’s New” -- the whole kit and kaboodle.

NOTE: If you are using Google Play Game Services (like leaderboards and achievements), there’s one important step to remember:  You need to go into a separate Google Play Game Services testing panel to enable testers to access these features before they are published.  (In a weird UI quirk, you have to first enable the Alpha test group before the switch to enable Beta testers appears.)

The Beach Buggy Racing Launch

Using Google’s beta system, we were able to push periodic builds of Beach Buggy Racing to about two dozen partners and friends while the game was still in early development.  About a month before launch, we created the public beta group, and opened up invitations to our social media followers.  Within a few days we had about 500 Android testers in the group.

The feedback from this group proved to be invaluable.  The initial 500 were (mostly) motivated fans who played new builds as we posted them, and came back with fairly detailed bug reports and creative input.  A few testers took the time to post logcat crash reports and other more detailed test results, and many of them regressed bugs after they were reported fixed.  A few bugs were region specific -- they only happened to players in Europe for example -- and would have been impossible for us to test and regress internally in the U.S.

In addition to allowing us to test the game in a safe, supportive environment and elicit feedback, the beta program also allowed us to engage closely with some of our biggest fans.  From their perspective, they got to play our new game before anyone else and give meaningful feedback before the game became available to the rest of the world.  We tuned game difficulty, tweaked the economy and modified our IAP strategy based on feedback from the forums.  

Some of the changes that came out of these dialogs were significant.  For example, the “Disable Ads” IAP morphed into “Go Premium” after a detailed discussion in the forums.  Instead of simply disabling advertising, it now strips away most of the F2P trappings in the game, removing ads, disabling the energy meter, unlocking Split Screen on supported devices, and making some items like car painting free.  “Go Premium” is now one of the most popular IAPs in the game.

Now that Beach Buggy Racing is published, we’ve kept the forums open and continue to rely upon the feedback and interaction with our beta test group.  We provide them with advance builds before the general public, and in return they give us great feedback and a more interactive dialog about upcoming game features.

Tips & Gotchas

Doesn’t really work for paid games

We tried a limited version of this last year with Riptide GP2, our premium game.  But a big limitation of using Google’s system for premium games is that beta testers still have to actually buy the game before they can download it.  For obvious reasons, this substantially limited the available test pool.

Real world KPI will differ

Beach Buggy Racing is a free to play game, and the beta group was our first opportunity to test out our analytics and get an initial sense of performance and monetization.  We were hugely encouraged by the data coming in, and figured we had a major hit on our hands -- until the game went live and we realized that (of course) the hard core fans who signed up for our beta would play more, and pay more, than the typical average user.  Still -- thanks, fans!  We wish everyone was more like you.

Set expectations early

In addition to pinning installation instructions to the top of the group discussion list, we also included a disclaimer and list of expectations.  Not everyone is familiar with the risks of beta testing (software may not be stable), or the expectations (in exchange for early access we want clear and detailed feedback from you, with X, Y and Z information.)  Mostly I credit the quality of our testers, but it can’t hurt to provide clear guidelines up front.

Expect and even encourage criticism

As with all playtesting, be prepared for negative feedback.  In fact, you should encourage critical feedback, because that’s how you’ll make the game better.  Thank people for their criticisms when they’re thoughtful and clearly explained, and try to steer any less-helpful feedback in a more productive direction.  

Respond quickly to posts

The more people see actual responses from the game developer, the more encouraged everybody is to participate.  Keeping the feedback loop tight makes the group feel alive, and ensures that participants are getting fresh comments in their daily digests.

Consider adding security failsafes

Typically when we publish a new game, the APK shows up on pirate sites within 24-48 hours.  For this reason, we cautiously added a “kill switch” into all our beta software which we could use to disable the app if the APK ever made it out of the beta and into the wild.  Fortunately, thanks to the loyalty and dedication of our testers, it never did.  Again, I believe it helps here to control membership and limit publicity to your existing fans, particularly in the sensitive weeks before the game launches.

Moving forward

The beta test feature in Google Play has become a welcome part of game development here at Vector Unit, and I expect we’ll expand its role in our process going forward.  Its flexibility, ease of use, and integration into the standard Developer Console flow really make it a no-brainer for any free to play app developer.

Other platforms are coming around to this model as well.  Just last week Apple announced Test Flight Beta Testing, which provides similar functionality on the iOS side.  While this is a welcome addition, and one we’ll surely use, it does come with a few caveats:  You’re limited to 1000 testers, you have to manage the list of testers yourself, and you need to go through Apple Review before the testers can get it.  The last one is a biggie, although we’ve been told you might not have to go through a “complete” review every time you update the beta.  We’ll definitely give it a try going forward!

References

Using alpha/beta testing & staged rollouts on the Google Play Developer Console

Beach Buggy Racing in the Google Play Store

The Beach Buggy Racing Public Beta Google Group