An Indie Guide to Quality Assurance Testing and Procedures

My name is Ryan Keable and I am a co-founder of Anomalous Interactive, a new indie development studio out of Melbourne, Australia. Before I began running my own studio I was many different roles in the Australian Games industry. One of those roles was lead Quality Assurance manager on Wicked Witches latest title, Wii AFL.

After this role I began QA consultancy for The Voxel Agents, another indie dev team in Melbourne who are responsible for the successful Train Conductor iPhone series. Working with these guys I realized that many indie dev teams don’t know enough about correct QA procedures and operations!

I found that many small teams fully realize is that QA affects production more than you think. The more efficiently you can manage your QA the faster your games will be developed and they will release with a higher level of polish. We followed this QA process in development of our latest game, Slingshot Justice and I would like to share a few hints and tips with you.


Bug Tracking and Ticket Management

This is the most important and most obvious part of the QA process. When you are writing down bugs, you are identifying and categorizing an issue in your game that needs to be fixed. Correctly reporting bug information and categorizing it clearly will make the job easier for those who will be required to solve it.

Identification:
The first step is to identify Who, What, when, Where and How (why is the same thing as how respectively).

It is also beneficial under certain circumstances to list out an array of issues that are covered by the same category. This is helpful with areas of the game such as UI. A list of bugs found in a single UI screen can be grouped into the one ticket and displayed as tasks. This helps to prevent bloating your ticket system with too many tickets and the annoyance of having to look at multiple tickets that all cover the one area you are currently working in.

Categorization:
Now that you have an informative bug written up, many tracking systems will allow you to create development categories such as ‘code’, ‘art’ and ‘design’. Categorizing your bugs with these kinds of keywords will help identification and sorting in your tracking system.

Priorities:
Identifying the priority of the bug is very important for work flow. When you are working with a system that involves upwards of 100+ bugs, organizing tickets into levels of priority will help both production and the ticket owners responsible for fixing the issues determine where to start. Organizing priorities also helps establish the overall work flow for Milestones and Sprints.

Submitting for Testing:
This was a system I loved to work with. When the owner of a ticket has decided that their bug is resolved on their end, that bug is now ready for testing. Create a ‘testing’ resolution for tickets to group all the tickets that need to be tested together. This allows production and QA to see just how much work is required on the re-testing end. Allocated tasks should also be put through this process so QA is aware when a new feature is implemented and ready to be tested.

Communication and Commenting:
Your ticket system and work flow should allow for a good deal of back and forth dialogue between the reporter and the owner. All transfers of tickets between reports, owners and testers from allocation to testing and to final resolution should be accompanied by comments. This does not just signify ticket sign off but is also great for iteration.

One of the best experiences I had on QA was creating comment based dialogues back and forth between myself and the UI programmers as we worked through specific bug or usability issues. This simple dialogue freed up our time from having to do specific 1 on 1 sessions over simple issues or creating multiple tickets for the same category and bloating the system.

Here is an example from Wii AFL:
My bug report:
- Injuries’ button hover state is scaling incorrectly
- ‘Player name’ is injured and is out for 1 weeks
- There is no notification when an player has recovered from injury

* can we add textual information in the injury field to display recoveries?

This was returned by the Programmer with:
- ‘Injuries’ button hover state is scaling incorrectly - fixed
- ‘Player name’ is injured and is out for 1 weeks - fixed
- There is no notification when an player has recovered from injury

* can we add textual information in the injury field to display recoveries?

What information should we display and how do you want this information to be displayed?

In this fashion we polished the bug and maintained a design and production dialogue without any unnecessary meetings.


Google Spreadsheets are Awesome
As you probably know already, Google docs are awesome. They are also amazing for Test Matrices, on the fly QA testing and Asset Tracking.

Test Matrices:
Test Matrices should be used as final comprehensive checklists to ascertain whether your product is functioning correctly. Test Matrices are checklists that cover all functionality within a given system. You can use as a resource domain for bugs or as a checklist to write bugs from.

Data Entry and Analysis:
Spreadsheets can also be used to record data and information generated in your game overtime. The collected data can then be analysed and compared in order to determine if there are bugs or design flaws with the related feature.

This is fantastic for tracking statistical data in your game such as:

With a recorded overview of data you have the capacity to compare and contrast the data accumulated data and find any errors or idiosyncrasies. It will also help you to evaluate user progress and user experience, depending on the feature

For example, in my Wii AFL, football players gained experience after every match. The rate at which XP was gained and spent (amongst other things) was tracked in this Spreadsheet.

From this data we reported numerous bugs that would of been difficult to spot through general observation. It also provided us with an overview of the user experience of these systems and allowed us to adjust the features accordingly

Asset Tracking:
Google spreadsheets can also be used as an asset tracking system, either for task development tracking or as a general asset checklist. Again on Wii AFL we used this process to track the current level of implementation, exportation and development of over 1000 animations.

The shared access to google docs disabled any bottle necks that are normally created by version control ownership by allowing all our animators to post their updates simultaneously into the document.

Bug testing for the animations was also integrated into this google doc system by creating columns dedicated to checking specific aspects of the animation; both technical and aesthetic. All bugs related to an animation were eventually handled by this system and written as comments inside the related column and coloured correctly for the animators to interpret.

Final Notes on Google Docs:
While I have really only covered spreadsheets, don’t forget about google’s word processor and drawing tools! Use these tools to create on the fly, dynamically updating design docs and specs. You can support design docs with simple illustrations that can all be worked on collaboratively.

Make them pretty! Use headers, columns and row breakers to make your docs easy to read. The colouring rules for spreadsheets are really easy to set up and work dynamically and should be used to display the different states of development. A simple colour system is red for not done, yellow for in progress, blue for done, green for approved!

Google docs are shareable. I cannot stress the importance of this, it completely removes production bottlenecks created by version control ownership and lockouts that limits only one person to work on a document at any given time.

Google Docs are automatically saved, revisions included!

Finally there are exporting issues when converting from google docs to excel so please do not attempt to create game data in google docs then export it into your engines!


Builds for Testing

It is always important for development and production to have working builds. Do not let broken builds stop your artists and designers from continuing their work. Try to get one or two clean builds out each week for testing. When writing bugs, the build date should be reported on every ticket - this is very important! Build dates should be displayed using some form of debug text on the interface

Debug builds should also be used rigorously during the testing process to minimize time spent playing the game to unlock data. If you want to test a feature at the end of the game, don’t force your tester to play through an hour of content before they can unlock it and test it. Displaying memory and the frames per second rate in a debug build is very useful when looking for memory leaks (derr).


The Do’s and Don’ts of QA Reporting and Testing

A final few notes on QA reporting and testing:

If you follow the recommended practices and process detailed in this report you should be staying on top of your QA process on a week to week basis. You will have no fear of massive amounts of QA grind work to be done just before release and you are more likely to release bug free.

I hope these tips and hints have been useful, these practices certainly changed how I approach development. If you have any more questions hit me up at [email protected], tweet us at @anoamlousint or join us at www.facebook.com/anomalousinteractive.