Differences between Software Testing and Game Testing

What is the major difference between testing an application or a software system compared to testing a game?

July 21, 2014

8 Min Read

What is the major difference between testing an application or a software system compared to testing a game? This is the question I asked myself when I switched jobs after ten years of working with software testing and quality assurance within mobile phones, to start working with games. I have read almost everything there is to read about software testing in general, but I have no real knowledge about game testing. So I have bought two books on game testing to get some basic understanding, and worked a few weeks to get some practical hands on experience.

To aid in my own competence development I tried to compile what I have learned so far.

Software testing is an engineering discipline. This is also true for game testing. Yes, you can play games and find bugs without structure or expert knowledge just as you can test that a mobile phone works, but if you want to exceed you need to become a professional tester.

Being a professional game tester means that you must understand both testing in general, but also the unique discipline of game testing. This is no small feat, as software testing is very complex and multifaceted, and game testing requires very specific skills.

Before we can explore the differences we need to agree on the similarities between software testing and game testing. Basically everything that Cem Kaner has written in his course Black Box Software Testing [1] is in my opinion applicable to both fields. Almost everything Alan Page has written about test automation [2] can also be applied to game testing. Most of what is included in the ISTQB testing certifications [3] should also be valid. Game testing basically inherits almost every aspect of software testing in general. Even the mindset of James Bach and his rapid software testing [4] can basically be tailor made for game testing. How Google and Microsoft works with software testing can teach game testers a lot about how to structure and organize testing in a good way [5][6].

Game testing is every bit as complex as any other testing, and should be treated in just the same way. It is not something that can be left to untrained laymen. There is a place for live user test, or alpha and beta test, but those are just a few pieces in a much larger puzzle. To reduce game testing to something as simple as just playing the game to see if you find any problems, is to seriously undermine the quality of your game.

So how is game testing unique? What aspects make game testing different from normal software testing? I tried to compile a list of types of testing that makes game testing especially troublesome. [7][8][9]

Fun Factor Testing
Balance Testing
Game Level/World Testing
AI Testing
Multiplayer/Network Testing
Audio Testing
Physics Testing
Realism Testing
Modification API Testing

Usability or user experience is something that is tested for all software, but Fun Factor Testing is something unique to games, since they are an entertainment product. Games are not only supposed to work intuitively and provide a good user experience – they also have to be fun to play. How do you assess if something is fun or not? How do you know if a certain experience is something that will entice the proposed target group? This requires a unique insight into game design, and vast experience and data about the user group and what that group enjoys.

Creating balance between different options, as well balance of difficulty for different levels, monsters or events, is also something unique to games. Balance Testing can only be done in a good way with a vast knowledge of game design and how the target audience responds to different difficulty levels. It also requires many hours of actual gameplay of the game under test.

One of the most complex aspects of game testing can be the testing of the actual world or level, especially if it is a vast, sprawling, 3D world, such as for modern MMOs. Some parts of this can be automated in interesting ways that are also quite unique to game testing, such as having bots move randomly through the game world to see if they get stuck or find other problems with the world. As the complexity of the task grows, it becomes more and more important to find ways to reduce the complexity with the help of tools. For puzzle games it is important to make sure that all the graphics for every level looks good, but also that each level is passable, and that game mechanics that have previously been tested in isolation, actual work in different level implementations.

Testing that computer-controlled opponents are working correctly, that the artificial intelligence is behaving according to design, can also become very difficult as the complexity of the behavior increases. Chess is a good basic example, and enemies in a first person shooter is a more modern one. This type of testing requires the tester to understand what triggers different types of behavior, and how these triggers can be confused by different parameters. Understanding of AI and game design is critical to succeed in this field.

Multiplayer testing is a whole other beast in itself. Many players simultaneously interacting with the game world, with computer-controlled opponents, with game servers, with each other. So many things that can go wrong. And it often requires a whole team of testers. Many difficult risk-based decisions to make if you don’t want to spend unlimited amounts of time testing different scenarios. Understanding of multiplayer game design, and how to test efficiently as a team is required knowledge for this type of testing.

Audio testing is common in all software that creates some kind of sound or plays media. However games have a unique aspect that other software does not have to care about to the same extent. Game music has to involve the user in the game and enhance the game play. Not only should the audio play without stuttering or missing elements, it should also add to the gameplay. This requires extensive audio skills and specific understanding of game audio. Very specific expert domain knowledge.

Many modern 3D games have physics engines. Modern shooters such as Battlefield or Crisis and RPGs such as Skyrim, have destructible environment and possibilities to throw objects. Testing if the physics engine works requires an understanding of physics as well as how to implement that in a good way into a game. The complexity of the testing grows with the complexity of the engine.

Especially in simulators or racing games it is very relevant that the game feels real, but many other games also incorporate elements that need to feel real. Testing for realism requires specific domain knowledge needed for the game or game element. An airplane simulator requires an understanding of airplanes. Cars, weapons, human movement, animals. Each poses their own problems.

A lot of software has open API that can be used by third parties. But there are few other instances where the players will try to exploit these open API to gain unfair gameplay advantages. This requires outside-the-box thinking. How will the modders use the API, and how will they be able to alter the gameplay in significant ways? Being one step ahead of the entire community of modders is a daunting task indeed.

In addition to these different types of testing, categorizing users can most likely also be done in a unique way for games, to help prioritize and select what tests and use cases are most relevant to run. How to do this categorization is another article in itself, but I leave you with an example of how you could think about categorizing different types of gamers. [7][8]

I have tried to cover some of what makes games a unique testing experience. I am sure there is even more, and I hope that I will be able to identify even more aspects as my experience in game testing grows.

One thing is already certain however. Game testing is very complex, and the complexity will grow as games become more and more complex. It is a combinatorial explosion that can only be handled by testing in a very smart way. Underestimating the competence and effort required to test games is something that can undermine the quality of any game.

Any tester who takes their game testing seriously need to master a large number of techniques, methods and processes, and understand their product in a way which is not required for many other software testing fields.

A daunting task indeed.