What makes a good game tester?

In this article I will explore what I feel are the most important skills and attributes of a good game tester, and what type of mindset I believe a good game tester should have. This is based on my experience and my context of working with mobile applications and games.

There are a lot of valuable skills and attributes that can be good to have as an Agile cross-functional team member, and there are skills you need to have just to function as a professional in a corporation, but these are not the skills I will focus on here – here I will focus on the skills and attributes I think I person needs to have in their specific role as a game tester.

An example of this can be “critical thinking”. This has historically been mentioned as a valuable attribute to have for a tester. And I absolutely agree – but it is equally so for basically everyone involved in developing complex software of any kind. Another example could be “good communication skills”. These kinds of attributes and skills I will not bring up in this article. I will focus more on those skills and attributes which makes the tester role unique. So let us begin.

My definition of quality intelligence is:

“Quality intelligence (QI) can be described as "a set of techniques and tools for the acquisition and transformation of raw product data into meaningful and useful information for quality analysis purposes””

So what does this mean, and how does it connect to being a good game tester? In your role as a game tester it is not your job to defend quality, to make sure a certain bug is fixed, or to fight for a higher level of polish in your game. Your job is to make sure that other people with a higher pay grade (or just another role, but that is less provocative) than you have the information needed to make those decisions.

We are working on complex products. There is a lot of uncertainty and risks built into this. A good game tester reduces uncertainty and risk and helps stakeholders and team members (developers, artists, game designers, etc.) make the right decisions, by providing them the right information.

Now this can be done in many ways, both reactively and proactively, which leads us to the next topic.

Before we talk about exploring complexity, we need to define complexity. The Cynefin framework has a definition that resonates with me:

“The complex domain represents the "unknown unknowns". Cause and effect can only be deduced in retrospect.”

Cause and effect can only be deduced in retrospect. This means that no amount of planning or discussions upfront will save you from complexity. The Cynefin framework prescribes “probe-sense-respond” as a way to tackle complex problems, and this to me has a lot in common with exploration. So what does it mean to explore a complex game?

For each test session you have a different starting point; what you are testing, how much you know about that complex system, requirements and expectations, historical data, knowledge about the underlying architecture and functions of the system, and so on. This is dependent on what information is available, but also the domain knowledge of the tester.

Once you have a starting point you can begin your exploration of the complex system, or in our case, the game or feature.

But before you start exploring, you need to prioritize your work. What information is of most value to the stakeholders, and by extension: which tests are most valuable to the stakeholders? 

When you can set your test scope and prioritized it, you can start designing and executing tests. 

So to be more concrete, these are abilities I expect a good tester to have:

So what do I mean with the last bullet? Once you have performed a test, the starting point for your exploration has shifted because of the new information that test has uncovered. You must now incorporate that when selecting the next test to perform. But there is one more ability a good tester needs to have when it comes to exploring complexity:

What I mean with this is that you have to know when to stop testing. When your time is better spent doing something else. When the product risk is sufficiently low.

As you may understand the above is far from trivial, and should in my opinion be part of the core competence of every tester. I will try to break it down further.

To be able to create a proper starting point a tester needs to be able to understand the system to some degree. Yes, you can go in completely blind with only an understanding from a user’s perspective but, in my opinion, that is a last resort. Looking at requirements, understanding stakeholder needs, and have some degree of understanding of the underlying software architecture is important to be able to get an overall picture of the complex problem you are facing. Does this mean that you have to be able to write code? No. But you need to understand software, and you need to understand stakeholder requirements and customer needs. And with all this understanding a good game tester needs to be able to create a mental model (that may or may not be documented somewhere) of what they are testing, to use as a basis for scope selection.

Based on knowledge about the system and it’s complexity and risks, and an understanding of what information is valuable to stakeholders, a tester needs to be able to do a smart selection about what to test. This can be done on different layers of abstraction, from testing performance for a very specific function, to just stating that usability testing for a feature is needed. This is where understanding of testing heuristics and system quality attributes come in.

Testing Heuristics Cheat Sheet: http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

System Quality Attributes

https://en.wikipedia.org/wiki/List_of_system_quality_attributes

Once you have concluded what you are going to test, you have to think about how you are actually going to test it. You have to design the tests you are going to perform. Now this can be done up-front, or in parallel with test execution, but one key aspect is to never create wasteful artefacts. If you don’t need a test case for the future, don’t write it down, just have it in your head. In general, my view is that test cases are not an efficient way of transferring knowledge. There is definitely a place for test cases, but it is limited to specific circumstances.

Test design includes looking at what tools are best for the job. Are there any tools that are required to execute the test case? Are there any tools that could facilitate test execution?

When designing a test you need to understand what you are trying to achieve. What piece of information are you trying to uncover, and to what degree will performing a certain test help you achieve that goal. The more knowledge and understanding you have of the system at this stage, the better. Trying to provoke a complex system without any knowledge can be very time consuming and inefficient.

Once you know what to test, and how to test it, it is just a matter of performing the test. Of course you should continuously look for ways to make test execution more efficient. That can take many forms. One is test automation, but more on that later.

Being good at test execution is all about achieving what you set out to do in the scope selection and design phase as efficiently as possible.

You will never be “done” testing something. There will always be more to test. Therefore, it is very important to know when to quit.

And this is all based on risk, and what level of it is acceptable to stakeholders.

You must always look at the value provided by performing one more test, and ask if it is the best use of your time.

Releasing a complex system, such as a game, to customers is always a risk. But the more information you have of the complex system and how it behaves, the lower the risk. And at some point you need to decide if the risk-level you face is acceptable.

To be an efficient tester you need to know when to quit, or at least when to ask stakeholders if you should continue.

What I have describe above is all reactive. As a good tester you also need to work proactively. What form can this proactive work take?

Writing automated tests is software development, and should be done by software developers. If you are a tester/software developer – awesome – but unless you are proficient in software development, leave it to those who are. Some types of automation requires less software development competence, and some require more

Of course. There are many attributes and skills, just like software development skill, which can provide value to an Agile team. Being good at game design allows you to do play testing and balance testing. Knowing about software security allows you to do security testing. Understanding UX helps you find value detractors in that area, and so on. Depending on what a team needs, they may expect you as a tester to know some of these things, but in my opinion, it is not the core of your role as a tester.

Here is a table to further explain my thinking:

I believe that what I have described in this article is the core competence that every good game tester should possess, but it is of course based on my experiences and context, and should not go unchallenged.