Do we really need game testers?

Do we really need game testers when we move to a more Agile way of developing games? If the development teams own the quality of their product and do their own testing, is there really a place for dedicated game testers? Play tests with real users/untrained testers is one thing, but here I am talking about professional testers performing testing throughout the game life cycle. To answer this question we need to talk about competence instead of roles. In a cross-functional development team, is there a specific competence that a game tester can bring to the table, or can they just be replaced with more coders that perform the testing themselves? I will try to answer this question based on my own experience as a software tester, and introspection into how I approach testing problems.

To find an answer, we need to break down test competence into its components, and see if this competence is unique in any way, and if it adds value to the development team.

Let us start this breakdown with trying to think about some general competencies we expect game testers to have, but which are not unique to game testers, and is something we expect from any developer or engineer. This will obviously not be a comprehensive list, but let us try to cover many competencies we usually associate with game testers, so that we can exclude them from the elusive test competence, to get down to the core of what a game tester actually brings to the table in a development team.

Competencies we (might) expect game testers (and most other people) to have, which have nothing to do with test:

Even if you had all these skills (and you would be a great person if you did), there is nothing in this list that specifically requires a tester. A great developer probably has many of or all of these skills, and many additional development-related skills.

So if test competence is not covered by the list above, what is it then? To understand this, it is my belief that we must look in the Cynefin Framework [1]. There are many different types of problems, and according to this framework they can be divided into four categories; Obvious, Complicated, Complex and Chaotic.

Sidebar: How does complex behavior arise?

“The behavior of a complex system is often said to be due to emergence.” [3]

“Emergence is a process whereby larger entities, patterns, and regularities arise through interactions among smaller or simpler entities that themselves do not exhibit such properties.” [4]

It is my belief that higher levels of test competence are related to the ability to handle complex test problems. So what does this mean? First, let us define a complex test problem.

Obvious Test Problems: Tests in which the relationship between cause and effect is obvious to all

Complicated Test Problems: Tests in which the relationship between cause and effect requires analysis or some other form of investigation and/or the application of expert knowledge

Complex Test Problems: Tests in which the relationship between cause and effect can only be perceived in retrospect, but not in advance

So a game producer should have no problems solving an obvious test problem, and a game developer should have no problems solving a complicated test problem, but it is my belief that a game tester is better suited for solving complex test problems due to their specific core competence.

Let’s take a step back and look at the implications of what I just wrote. This means that I see no reason to have dedicated game testers to handle obvious and complicated testing problems. This can be done by developers, and in some cases even producers, designers and other roles. Obvious and complicated testing problems do not require high enough test competence to justify a dedicated game tester on the team. Sometimes you might however need a tester to coach the members of the team to raise their test competence to a level that allows them to handle these obvious and complicated problems.

The next step is to define this test competence I keep mentioning. I believe it consists of (at least) the following components:

So what does this actually mean? What is it that I expect an experienced tester to achieve with the four bullets above? How does this add value to the development team?

If an experienced tester is confronted with a complex test problem, then this is how I expect them to approach this problem:

By approaching a complex test problem like this, I believe that the game tester will provide value to the development team by providing them with information about how the system actually works, and by helping them understand and disperse the complexity of the system.

So a game tester needs a wide array of skills to be effective (the first list of competencies), but they also need a strong test competence, which is what makes them unique in the development team, and allows them to provide value in the form of a better understanding of the complexity of the system.

Everyone in a development team can test, but not everyone is equally equipped to handle complex test problems.

So do we really need game testers? The answer is yes, and no.  What we need is test competence. If we can get this competence into the development team in some other way, then no, we don’t need dedicated game testers. If we cannot, then having game testers on the team may be the only solution.

References

[1] Cynefin Framework

https://en.wikipedia.org/wiki/Cynefin_Framework

[2] Heuristics Cheat Sheet

http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

[3] Complexity

https://en.wikipedia.org/wiki/Complexity

[4] Emergence

https://en.wikipedia.org/wiki/Emergence