Testing Farm - Making mobile games testing easier on the cheap

Testing is an essential part of game development and mobile titles are no exception. The market is filled with devices of all form factors, performance levels and compatibility. The more players there are the more successful the game is and the larger the income. In order to get as many players as possible, you need to support as many smartphones and tablets as you can. To achieve that you need to “polish the build” so that it works well even on the most budget devices.

Since testing is vital, it begs the question: how do you conduct it? The simplest way is to acquire a pool of devices and launch the game on them, tracking bugs and fixing them. It’s reliable but depends greatly on the number of testers and requires higher spending, as there is a risk of losing or damaging the devices. Tracking errors is also harder this way, since developers only get the information which their testers provide, and the level of detail is only determined by the latter’s level of responsibility.

There is a way to speed up, optimize and automate the process. How? Set up a testing farm.

What is it?

In short, a farm is a testing PC, similar to those used to test computer games, but with more budget hardware and without a GPU (it’s not necessary). The purpose of this setup is to create a place for an employee to get a testing device without physically accessing it.

This is especially useful for companies with large device fleets, which would be harder to track otherwise. The more employees you involve, the more difficult it is, as each one would need a gadget. The farm allows you to keep the devices connected at all times and issue them without excess bureaucracy.

How does it work?

From a user’s perspective working with the farm is similar to remote play functions available on some gaming consoles. All smartphones and tablets are connected to the farm, deployed on Ubuntu. In our particular case we based the setup on specific source code available for free on GitHub, which we upgraded and optimized to the company’s requirements, as well as updated it, since the original developer ceased support 4-5 years ago. At the moment the farm supports all versions of Android up to 12. We’re also working on iOS support,

In detail, the farm operates like this: we run an Ubuntu server that uses Rethink as a database. The reason for such an OS choice is simple - its functionality makes it the most suitable. Users connect to the devices via a web interface, while the gadgets themselves are connected to the PC via Android Debug Bridge. The latter shows every session of every user on every device at any given time. The administrator also had access to full usage history. The web interface is accessible not only from PCs but also from any smartphone, so you can perform test runs on an Android tablet from an iOS phone.

Since this work is done, essentially, through a web connection, one might ask “wouldn’t slow internet cause problems?” Our testing showed that such a setup doesn’t require an overly powerful channel, even a fairly budget 100 MBps connection will work fine, with up to 120 users testing games simultaneously.

What’s in it for the developer?

First, as mentioned previously, the more devices support the game, the better. The best way to achieve that is to test a build on as many devices as possible. The farm helps considerably accelerate and simplify the process, while also making it easier to access the devices.

Second, with the farm you no longer have to test solely at the office. Employees don’t need to take the devices physically, meaning no more staying at the workplace with it to avoid losing it or breaking NDAs. You can access the farm from both your work PC and your own device, provided security procedures are followed.

This, it’s easier to control the devices themselves via the farm. It tracks who, when and where, takes which smartphone, which helps optimize the schedule and determine reasons for bugs and crashes.

Fourth, the farm supports macros. This allows running the same session on multiple devices at once, which dramatically decreases the time and effort spent on a test. Alternate ways, such as emulation, do not fully cover these requirements.

Finally, with the farm you can record videos and take screenshots automatically. This helps add evidence to bug reports immediately after encountering errors.

Of course, there are drawbacks, but those are custom. For instance, each form factor and devices of different resolution require individual macros. The more macros there are, the more support they require. We’re currently working to resolve this.

We also aim to develop the system further so it will be able to copy sessions from one device to the others via wi-fi and collect logs simultaneously. This will allow us to test on all devices at once from one anywhere in the office.

How do I set up a farm like this?

"The recipe is actually simple”. It’s accessible to indie developers and large companies alike. We use a PC with an i7 CPU, 32 GB RAM, 500 GB SSD and a 2TB HDD for logs, screenshots and videos. You’ll also need Ubuntu 20.04 LTS and a docker image to hold a container with the server so you can restart the farm in case of malfunction. The latter is easily doable even for beginners. The server can be deployed on even less powerful systems, depending on the number of devices and frequency of use.

As for customization, it can be done by any system admin level employee. The work is done in iterations and is a constant process but most of the setup and installation work can be completed in a small amount of time. In our case studying the information took 1 month and the server itself was deployed in 4 business days.

What’s it like to work with it?

From the user’s point of view, working with the farm is similar to using the devices themselves. It looks very much like using an emulator but with the device’s hardware rather than that of the PC. At the same time, the administrator sees all the sessions in real time and can access the logs at any moment, which saves a huge amount of time, since you can find out what caused a crash or an exception almost instantly.

All of the smartphone functions are supported, including keyboard, languages, multi touch and gestures. The devices themselves are stored in a special box next to the farm and can be tracked physically.

Won’t this “kill” the devices faster?

Surely there is additional strain, primarily on the batteries, due to using the debug bridge. According to manufacturers, the devices can lose up to 20% of their lifetime. However this doesn’t play much of a role in reality. The market updates the devices so fast you’ll likely update your fleet faster than you’ll have to replace any broken devices. Still, it would be wise to have a stock of parts and batteries.

Whom will the farm suit best?

A farm would be suitable for everyone save for targeted teams, but whether or not such still exist in this day and age remains to be seen. Android supports a wide lineup of devices and an equal range of OS versions. iOS is pretty much the same. So the farm is a much more convenient option. A properly selected fleet of devices will help optimize work for the widest range available. You will be able to spot and eliminate errors faster, which is especially important in an industry where the game has to be perfect before release, or players lost at launch might not come back.