Top 10 Tips on How to Improve a Game Quality Assurance Department

I'm director of quality assurance at Alberta, Canada-based BioWare Corp. BioWare is the developer of upcoming titles like the martial arts action-RPG Jade Empire and the fantasy RPG Dragon Age, among others. I've been meaning to write a piece like this one for a while now - hopefully, some of this information will prove useful to some of you regarding practical tips for improving quality assurance regarding approaches to video game bug and standard testing.

1. Define "Quality" and "Fun"

So, what is "quality" anyway? Some of the common definitions are: "a degree of excellence" or "a distinguishing attribute," which aren't very useful to those of us who have to assure that it's there.

Let's begin by defining quality within the context of the products we make (note: defining quality will be a common theme in this article).

Games are complex pieces of software. We can start by breaking them down into smaller, more manageable chunks. For example we could start by looking at the software side; we can devise test approaches to our games by applying many of the standard techniques from the software quality and assurance industry. These "best practices" are worth examining in more detail: A few examples could be: functional testing (does it work as intended?), performance testing (does it meet our performance targets?) and usability (ease of use) to name just a few.

Now comes the fun part… literally! For such a small word, "fun" is a vast topic. Actually, I'd go so far as to call it a science. This means applying an approach of observation, identification, description, experimental investigation, and theoretical explanation to help define the seemingly simple word, "fun". As we did with "quality", let's define "fun" by asking some simple questions. This process will lead to some answers, and probably even more questions. A few examples could be: why do people play (our) games? What is fun? What isn't fun? Note that the specific questions you ask may well vary depending on the kinds of games you develop.

2. Make a Plan and Set a Vision

Planning is one of the four pillars of management. We should all have plans, right? It is said that "when you fail to plan, you plan to fail." This will also serve as a roadmap for our departments. Planning ability could be used as a measure of our progress as leads (scary isn't it).

The vision is simply over-arching goals we hope to achieve with the plan. Our current QA department's vision is to develop a team of workers whose goal is to improve product quality while reducing the cost of bugs by finding them earlier.

3. Provide your team with clear Job Descriptions and Responsibilities

As a profession QA can provide a challenging and rewarding experience, and for some it can lead to a lateral movement to other departments. We are often required to be jacks-of-all-trades and masters of at least one. Testing requires an understanding of a lot of different systems and their interactions as well as the ability to communicate our ideas to the developers.

We have a variety of roles within QA. Some are dictated by corporate goals, other are dictated by the types of games we make and by the type of information we seek. For example, our full-time testers provide expert testing services and developer support. Currently our full-time roles include QA leads, technical and design gurus (sub-leads), QA tools leads, technical testers, gameplay/story testers, application testers and QA programmers.

Our contract testing roles include the proverbial playtesters, focus testers, beta testers and user test specialists. Let's talk about the playtesters (who by the way are an essential part of the QA team, albeit at a later stage of development): they are energetic and provide fresh perspectives. It also helps me sleep better at night knowing that a small army of playtesters are taking the boot to our games.

I've been mulling over for a while now how regression testing works with regard to our games. Regression testing is simply ensuring changes introduced do not adversely affect current functionality. It's near impossible to perform full regression on games such as the ones made at BioWare. By using automation and build acceptance tests we can cover what we need to. At a certain point we add a test case which requires playing through the entire game's critical path. This is a good task for a playtester.

4. Find Good People, and Treat them Well! - Quality in the Workplace

Another of BioWare's core values (along with "Quality in our Products") is "Quality in our Workplace". Basically, we believe that a company's greatest asset is its people. A good place to start when creating a high performance team is to surround yourself with talented and passionate individuals.

Let's look at the hiring process. Hiring is serious business which has a direct and high impact on team performance. What is it that carpenters say? - "measure twice, cut once." When we bring someone aboard we are confident they will be a valuable addition to our team. (The idea of spending the majority of one's time coaching low performance employees or remaining in hiring hell isn't appealing nor productive).

Retaining great people is just as vital for any company. Investing time in finding out what motivates (and de-motivates, so you can improve things!) is just as important. A good manager listens attentively; remember that good ideas can come from anywhere and anyone, anytime. Career planning is not just nice benefit but essential, and in some cases succession planning may be required.

5. Training and Development

The products we create are the results of our collective intellects and a goodly sum of our blood, sweat and tears. Training is one conduit by which we can hone our knowledge, not to mention that it's mutually beneficial to both employee and employer. Employees share and apply this new found knowledge to the products they help to create. Employees feel valued and appreciated when their employers invest in their training and long term career development - this helps to build long term loyalty to the company.

Discussion can be a powerful ally. This prompted us to develop a few programs with that very idea in mind. Our Design Group focuses on gameplay and design discussions. Our Reading Group looks at writing and story elements. The QA critique session is where the team watches gameplay in the BioWare movie theatre and openly discuss their thoughts. We are in the planning stages of a programming discussion group as well. Additionally, we maintain a library of books covering multiple disciplines such as game theory, quality assurance, leadership and programming.

Professional development should also be considered. I'd describe these as universal skills - the type of skills we can apply at work or in our everyday lives. A few examples would be decision making, time management and communication.

6. Tools of the Trade

Labor is by far the biggest expense. That's why employee productivity should be at the top of our list of priorities. Tools have been around for eons; it's a simple concept really: tools help us work more efficiently and achieve better results. With that said, are we providing our teams the tools to make them better workers?

The problem with tools is that they usually have a high initial cost or the benefits aren't immediately obvious. I take it we're all familiar with lost productivity (this seemingly innocuous spectre of inefficiency). It doesn't show up on balance sheets, and to borrow a words of Verbal Kint from the Usual Suspects, "the greatest trick the Devil ever pulled was convincing the world that he didn't exist." Seek and ye shall find! Go straight to the source. Talk to the employees and find out what frustrates them. Observe them in their natural habitat. The project management tool can be a powerful ally, if the probes are set right. We can apply our experience and understanding of the process to identify the real problem. Once we gathered this information all we need to do is pull it all into a Return on Investment (ROI) or similar report and present it to management to help back up the case for improving the tools.

Tools can also help eliminate repetitive and monotonous tasks. Bill Gates summed it up well when he said, "it's hard to measure when you make your knowledge workers more effective, but it's just common sense that knowledge workers who are not distracted by or burdened by routine matters will do better work." (Bill Gates, Business @ The Speed of Thought)

In our department the popular tools are project management, debugging and logging, development, automation and benchmarking tools. In addition we've created a game research station where we can perform competitive analysis.

7. Create a Quality Working Environment

The corporate culture will play a large part in defining the overall environment. At the department we play a significant role in defining our local environment. We must again ask ourselves what a quality work environment is (and isn't), and believe in fair and respectful work environments which focus on collaboration. This includes ergonomics as well. Ergonomics has to do with providing a safe and healthy working environment for your team.

In all companies, it's important to develop a culture, and this drives the behaviors we feel are important such as collaboration and fairness. Again, these values are quality in our products and quality in our workplace, in the context of humility and integrity.

8. Develop an Accurate and Fair Monitoring System

What is our purpose other than playing games for a living? In quality assurance, we provide feedback. Unfortunately for us, this creates a significant challenge when trying to qualify and quantify our performance. I'd venture to think this is sometimes or in some measure due to the lack of visual or functional components in our results. Our feedback is supposed to improve product quality and save development time by finding issues earlier - bugs are lower cost when discovered earlier because fewer dependent systems are impacted. The challenge with live business environments is that we often can't perform controlled experiments to prove the value of our contributions.

When developing a system like this we should strive to make it fair and accurate. Monitoring is yet another pillar of management. Let's look at possible sources of information that can be monitored. A good place to start would be the people to whom you are providing the feedback to (the "customer", if you will). We can also look at the project tracking tool and last, but not least, the resulting quality of the product, which should be verified at specified intervals. We've even asked our team what they felt they contributed to the project.

What about a bug quota system, you say? The systems I've seen were simplistic and of little value, not to mention being counter-productive and frustrating for everyone involved. Granted, these systems can generate many "units" (some of which might be considered valid bugs). My advice would be, "Do not put faith in what statistics say until you have carefully considered what they do not say." (William W. Watt) We recommend reading The Goal by Eliyahu M. Goldratt, a book which covers flawed measurement systems.

9. Improve the Feedback

If our product is feedback, and our department has the word "quality" in it, then we should damn well make sure our output is of high quality! (It would seem kind of pointless if we didn't practice what we preached). To assist in this let's define "quality feedback".

To help in this task I'll use the "Can Pig Ride" acronym as a guideline. This provides us with 10 points to keep in mind when providing feedback. They are Condense, Accurate, Neutralize, Precise, Isolate, Generalize, Recreate, Impact, Debug and Evidence (due to space considerations, I cannot go into detail on this acronym in this article but I've provided the reference to the original article that covers this concept: Kelly Whitmill, "Writing better defect reports"). Here are a couple of points to keep in mind when providing feedback: know your audience, don't make assumptions as to what they know and provide them with the information they need to perform their jobs.

In games we often deal with the "opinion" bug, which is subjective in nature. As an added bonus, the creator may have a personal attachment to his work (which is perfectly natural but adds another level of complexity). There are some common problems associated with this type of feedback. First off, they're not facts, they're opinions - some of which will conflict with other opinions. Often as well, too much emphasis is put on suggesting a fix and not enough on isolating the underlying problem which can result in treating the symptom instead of the problem.

Here are some suggestions on how to improve opinion bugs: try to remove the subjectivity of the feedback (neutralize). Provide references to support your opinion. Get additional support from other people (e.g. we often have discussions or email polls for these types of issues). Focus on isolating and defining the problem as opposed to providing a suggested fix (which is an opinion of its own - while welcome, this is not a critical element to a bug).

10. Make QA an Integral part of the Development Team

At one time or another everything passes through QA's hands. The more often it does, generally the higher the level of polish in the final product. Our effectiveness increases significantly when we have the information we need. It is said that, "a man's judgment cannot be better than the information on which he has based it." (Arthur Hays Sulzberger)

One thing that has helped is letting people know what we do and how we can help them; by way of comparision, if I don't know what someone is selling and how it can be of use to me, I probably won't want to buy it. This opens up the lines of communication and gets the information flowing both in and out of the department. We all need to be evangelists for QA and spread the good word about what we do (sounds cultish, but it's true).

What does QA bring to the table? As people who spend most of our time meticulously scrutinizing the project in whole and in part we can provide valuable feedback from different a perspective.

Along the lines of the Six Thinking Hats system (de Bono, E., 1985. Six Thinking Hats. Boston: Little, Brown, and Co.). I'd say from we could provide the Black Hat (Logical Negative), White Hat (Information and Facts), and Blue Hat (Steps and Process) perspectives.

However, we do need to be a respected part of the team (as we are at BioWare) to be effective.

Conclusion

Conventional wisdom states that, "an ounce of prevention is worth a pound of cure" - why not apply that to game development? Try to consider the points raised in the article above, apply the ones which you can to your own work environments, and above all else make QA an integral and respected part of your team - you'll be pleasantly surprised at the results!