Why game testers should be integrated into the development teams

Jeff Sutherland recently wrote about that being agile means getting rid of separate test teams. He talks about how tools development at Microsoft reduced their bug count from 30000 down to 3000 by integrating their testers into the development teams. [1]

Of course there will always be instances where a separate test team will be necessary, or at the very least equally good. Running mandatory customer acceptance tests or certification tests is one example. Localization test is probably another. But we will disregard these instances in this article.

So why should game testers not be a separate team? Why should almost all game testers be integrated into the development teams, and part of the development process? The short answer is “complexity” and “combinatorial explosion”. At least these are my thoughts on the subject.

Large game worlds, multiplayer, AI, unpredictable users, and many other factors add a complexity to games, which is rarely seen in other software. Games are unpredictable and can sometimes feel random. Bugs appear even though the changes done should not have affected that area or feature. A complex system [2].

If game testers are siting in a separate team and are not an active part of the development process this means they have less insight into the how the complex system works. To them it is even more unpredictable and random.  This is further compounded by the fact that game testers usually have less clear requirements to work with than other testers [3]. This means that when something changes they basically have to regression test everything if they want to be certain that nothing has been broken. Since most modern games become larger and larger, and more and more connected to other software, the number of possible tests grows and grows. There is a combinatorial explosion [4] of tests to execute every time something changes.

With a separate test team this means that the number of tests they must execute will continuously (and not linearly) increase with feature growth. This means that the team must either grow, increase lead-time, or not test everything. Since the separate test team is dealing with a complex system, they do not, with any certainty, know what tests to select and can easily choose not to run tests that would have actually revealed critical bugs.

Since continuously increasing lead-time or growing the test team is probably not an option, the only way forward is to be able to select test cases in a better way.  To be able to do this I think that you need two things. Better communication with the system experts (game developers and system architects), and a better understanding of the complex system, which reduces the unpredictability and perceived randomness.

As you can imagine my opinion is that this is achieved by having testers integrated into the development teams.

When discussing this with a friend, I was asked who would perform system test if all the testers were part of different development teams. Who would make sure that all the different development teams’ features and changes worked together? Wasn’t there room for a system test team that tested that the entire game was actually working?

My answer was that we should not confuse the need for system test with the need for a system test team. System testing is probably the most complex testing, and I don’t think the answer is to make it harder for these game testers to communicate with the development teams, and give them less insight into the complexity of the system.

With increasing complexity and combinatorial explosion we need to test smarter and more efficiently. The answer to solve this conundrum can never be to move the game testers further away from the development process.

That is my opinion anyway.

References

[1] Agile means get rid of test teams

http://scrum.jeffsutherland.com/2014/07/agile-means-get-rid-of-test-teams.html

[2] Wikipedia: Complex system

http://en.wikipedia.org/wiki/Complex_system

[3] Cowboys, Ankle Sprains, and Keepers of Quality: How is Video Game Development Different form Software Development

http://research.microsoft.com/pubs/210047/murphyhill-icse-2014.pdf

[4]Wikipedia: Combinatorial explosion

http://en.wikipedia.org/wiki/Combinatorial_explosion