One of the most common problems in mobile game testing – similar to app testing – is how to access all possible devices that your game users are using. That said, despite thousands of unique models already out there for Android (and with all possible iOS variants with different OS versions) only few hundred of them are really relevant in the big picture.

Let's take a look at different mobile game testing approaches, as well as infrastructural and architectural aspects of the mobile game testing...

Architecture and Infrastructure Aspects of Mobile Game Testing

Let’s start from software architecture. Many of today’s mobile games are based on open source or commercial game engines, such as Unreal, Unity3D, CryEngine, Construct, PlayCanvas,  Cocos2D, and many others! (DISCLAIMER: I’m not taking sides of which game engine works the best or delivers the greatest features, just listing few popular ones for reference.) Furthermore, many of these game engines provide great tools and frameworks to get you game quickly built, with all those nice graphical features.

Looking this from the traditional software point of view, this is like a “tool – app – middleware” offering that these products will provide you, as well as compiling your game to specific platform (many of these products are cross-platform and enable you instantly to compile the game for Android, iOS and other platforms as well). For the platform part, let’s use Android as an example: an “open” operating system with a stack of software components which can be roughly divided into four layers: Application, Application framework (manager layer for content, resource, package etc.), Libraries (e.g. OpenGL ES software, Fonts, WebKit, SGL etc.) and the Linux kernel (drivers for graphics, audio and keypad, power management etc.). In addition, platform here includes the actual hardware running whatever chipset on it, with GPU and displays of different resolutions.

After that your mobile game needs to communicate with back-end servers using your own or Google/Apple/other services via WiFi, radio or some other type of communication channel. The need for testing the connectivity between your mobile game and back-end services is nowadays very important. For example, ads are shown via connectivity and without those running properly on your device, you are taking an extra risk in front of your core audience – mobile gamers. Sure, they would probably like it better without ads, but from the monetization point of view, you surely want to test this thoroughly before submitting your app to app stores – and make sure you are not losing revenue because of faulty implementation.

As said, testing connectivity should be one of the key things to do when building mobile games. Test automation can bring you a lot benefits for automating the connectivity testing, and scaling the infrastructure from one user to hundreds of thousands of users. 

Testing Hermetic Environment

Today, nearly all mobile games have a server-client interaction, some requiring a login, uploading of data (e.g. game results/progress) and downloading of data (e.g. ads). When adopting these kinds of services to be part of your games, developers need to ensure that all their changes done in code, to either server side or client side do not break the functionality of the service. Furthermore, this type of environment can be also set up on a local machine/server where critical connectivity and interaction between user and server can be tested. In fact, testing for some of these types of use cases could be done efficiently with a hermetic environment – however, not jeopardizing the integrity of your game.

Hermetic testing can be seen and done as easily as switching your device to flight mode. Nothing gets send out and nothing gets in. 

Testing Compatibility - and Doing Porting

When your mobile game is getting developed, you should always test on the real platform that the end-users are using as well. This means that you must test your game on real devices. Emulators won’t simply provide you as authentic environment as it would be required to get full understanding of real behavior in hands of real users. What makes a difference between using an emulator or real hardware? Typically, the lack of real hardware is slowing down and forces developers to use emulators, but this can shrunk into 4 categories why games should be only tested on real devices:

1) SOFTWARE. It’s just not only about vanilla platform software that makes the difference, but nowadays OEMs customize their software a lot: UI layers, customized legacy software, only certain widely used apps are pre-integrated with the device (e.g. Twitter, Facebook).

2) HARDWARE. There are LOTS of differences in chipsets, GPUs, even connectivity hardware, displays, sensors, and memory. This is very hard to emulate appropriately – and even it would get as close to real world scenario – all of that is executed on PC x86 hardware, and not ARM-based hardware.

3) USER EXPERIENCE. This is heavily related to performance. The graphics performance of the system makes the user experience, either good or bad. And there is an absolute must to get mobile game tested on real hardware.

4) NETWORK. The system can experience network issues and slow network is not possible to be tested on emulation. In terms of network configuration, emulators run on the PC, connect to the LAN and access the internet via your corporate firewall. Using real handsets, the network is connected to the radio interface and from there to the internet. 

Whatever platform (e.g. native mobile platforms, web etc.) you are building your mobile game on, remember that only real devices give you real results. Furthermore, the depth of testing (e.g. how many devices are used, how many different configuration those cover) is important as you don’t want to minimize your target audience who have potential to play your game – make you money!

Smoke and Reference Testing

When the mobile game is in the state that is testable in the context of 1) game itself, 2) platform-compiled and 3) connects with back-end services (or just platform APIs) the full-blown reference testing can give great results and understanding of how well game does. But even much earlier than that, it is possible to start doing smoke testing and using even hundreds of real Android/iOS devices for that. When bugs are revealed and fixed earlier, it will save you a lot of time (and money) and gets you earlier to publish your game for gamers. In addition to regular smoke and reference testing, it’s a good idea to include real performance and battery drain tests as part of your testing pattern.

The reference testing gives great understanding of how your game does, but it doesn’t cover everything. For example, huge number of users can quickly take down the back-end, slow down the game-play and even worse, force users to stop playing the game. More about this in the next chapter…

Stress, Load, Spike, Soak – Performance Testing

Good performance is very closely related to good user experience. User wants to see constant progress with the game, do the smooth game-playing, graphics performance needs to be up to snuff and work across all different mobile devices (phones, tablets etc.) the very same. In addition, it is important to understand how well back-end(s) can handle massive number of users and significant load.

There are lots of different ways to build back-end testing and scale up the performance so that simulates the huge load for your service. One easy and very straightforward way to do this is to use device farm on cloud to get all of those devices to do the same thing for your service. It literally takes minutes to set it up and get results back instantly.

Conclusion

It’s important to highlight that with mobile games, the graphics performance capability is the most important thing in order to form either good or bad user experience of that game. Naturally, not all devices are capable to run smoothly that high-end game, but even with smaller changes and optimization you can really make a difference and get your game running properly on wide range of different hardware. The more devices you get supported, the better chance you have getting your game played by hundreds of millions of gamers.

Building a successful mobile game starts with understanding how intuitive and great game-play can be enabled for gamers. Making it sure that users understand that they are fully capable to use all great features (and graphical stuff) you’ve built in your mobile game, makes a difference. The usability, user interfaces, graphics assets, OpenGL ES content and implementation are actually those that form the playing experience. And eventually, when game-play is great and gamers stick with it, you’ll see that as an increased potential to monetize that enthusiasm.

Have a great testing for your mobile games!