Codifying Game & Test Knowledge in Tests?

How much information should your test cases (or test missions, charters, or other types or similar test artifacts) include? What are the pros and cons of adding lots of detailed information in your test cases? These are questions I will discuss in this article, based on my experience with testing.

So imagine you have a test case, which just states that you should test the feature, and confirm that you experience the desired outcome. If you have no previous experience with the game and it’s features, performing such a test will be very difficult, and most likely counter-productive.  If you are an experienced game tester who has worked with the game and it’s features extensively, it may be enough information for you.

So the key point here is:

What if you are very experienced in general but do not have any game specific knowledge? Most likely you would run some exploratory testing on the feature before the actual test to understand how it works, and then discuss your findings with a developer or designer.  This way you get the context-specific domain knowledge you need to perform the tests using the very vague test cases.

Key point:

But there are obviously exceptions. If you have to enter certain settings into test tools or environments to be able to test the feature, and this information is not readily available, it will be difficult to perform the tests. This could be part of the test case, or of some other test documentation, depending on the context.

Key point:

So why not just write very detailed test cases, with intricate step-by-step instructions for how to do every single thing, so that even someone without any experience could do it? That would make all of the above irrelevant.

There are a few reasons.

Firstly it is very costly to write these kinds of test cases. The game testers have to spend a lot of time creating these test cases. Time they could have spent doing more valuable things, like actually testing the game, or other preparations for testing.

Secondly these test cases will require much more maintenance. Every time a small detail in the feature changes, the tests need to be updated. This is an additional cost apart from the upfront creation cost.

Thirdly these types of detailed, scripted tests restrict the tester and reduce the variation and innovation in the tests performed. You will test everything exactly the same every time, and there is a greater risk of missing bugs at the periphery of the test. The tester is executing a test case, rather than focusing on testing the feature. Running the exact same test over and over again is often not an optimal use of resources.

Fourthly it gives false confidence that inexperienced game testers can just perform these tests, and that the confidence in this results will be the same as if an experienced tester performed the tests. Even though an inexperienced tester can run through a detailed, scripted test, this does not mean that an experienced game tester wouldn’t have done it better. And I don’t mean faster – I mean finding tricky bugs in the area, which the detailed test artifact intends to cover.  You cannot separate test design from test execution and think that test execution is something mindless which can be outsourced to random people. Good test execution requires experience and opportunities to utilize that experience when you perform your tests.

Key points:

But what about if you are an inexperienced tester – isn’t detailed test cases a good way to learn the feature/game, and get the initial experience you need? This is a tempting proposition. An experienced tester transfers his knowledge to an inexperienced tester through a test case. Personally I am unconvinced this is a good way to transfer knowledge. I think the best way is either through pair-testing, mentorships, or through good communication channels. I just don’t think detailed test cases are a cost-effective way of transferring knowledge.

With this in mind I would like to edit the first key-point I presented:

So how do you reach the optimal level of detail in a test artifact?

But even experienced game testers have to understand what needs to be tested, so that they do not forget to test specific parts of the game or feature. So a test case that states: “Test the game”, would be problematic for anyone, regardless of experience.

The goal with a test case (or similar test artifact) should be:

If an experienced game tester knows what needs to be tested, then the game tester can use experience and knowledge to deduct how to test that in the best way, with regards to the current context, using minimal documentation.

These are my views right now in any event.

/Johan Hoberg