How much people, time and money should QA take? Part1

Project planning is one of the most crucial and difficult steps of a videogame development project. It sets the scope of the game and its features and, through those, the amount of effort, time and money that is planned for it. With this entry I’ll share a couple tips and techniques my team and myself use to calculate how much people, time and money QA should take. 

The most basic external factor rules we rely on are:

The main internal factor technique we use is the “Puzzle technique” which considers:

I’ll discuss the Puzzle technique in a future entry; this one will cover the two external factor rules mentioned above. And before I start and get the question: “Matt! Does this apply to Dev QA or Publisher QA? Or outsourcers?”, this applies to all videogame QA!

The 10% rule

The 10% rule is based on the budget value that you assign to QA. It’s called as such because for most mid-sized projects, approximately 10% of the development budget would be assigned to QA testing. Bear in mind that this calculation should only be based on developer and tester expenses. It doesn’t take in consideration Art, Music & SFX, Marketing, Legal, etc., but solely what can be considered development or “engineering” expenses. Below is an example of what it would look like typically in terms of expenses; 10% of costs to testing, 20% to Debug Devs (essentially your developers fixing/debugging, or the amount of time spent fixing/debugging, the issues found by the testers) and the rest, 70%, to your core development team creating and implementing technical assets in your game.

As a general rule, the bigger the project, the less QA should cost in terms of total proportion. This can be explained by a few things:

Below is a guideline for that cost. As you can see, the concepts explained above can be clearly identified. I can already hear some of you complain that your QA cost experiences differed. Remember, this is a guideline, and if you see a drastic difference, there might be a few points you didn’t take in consideration. One may be that your dev team, or yourself if you are an indie, have been doing that testing instead of an assigned QA team. Another big influencer might be what I call “multipliers”, factors that can impact QA efforts and costs drastically (we’ll have a look at multipliers in my next entry).

Here’s an example from a developer that was kind enough to share information. His game cost him 571K$. So, based on the above cost you’d expect QA to be close to 20% (between Indie/B game and A type).

If we solely take the development budget of 278K$ (Dev and QA costs), we get 22.3% for testing, which is indeed very close to what we were expecting.

The QA/Debug Dev rule

While the 10% rule is based on expenditures, the QA/Debug Dev rule is based on average staff volumes while testing occurs. Essentially, on average, the amount of testers you assign to your project should not exceed the amount of developers assigned to fixing/debugging bugs on that same project. The biggest exceptions to this rule can sometimes be right after the beginning of testing, when testers are starting to understand your game, get productive and find all of the most obvious issues, and when nearing the end of your project, when quality is already quite high and the amount of bugs found per tester drops drastically. Both of these usually even out as an average and are partly why test teams are quite small at a beginning of a project and a lot bigger near the end.

The reason behind this rule is, on average across a whole project, the average amount of time it takes for a tester to find and regress an issue is about the same as the amount of time it takes a developer to debug that same issue.

In the example above, I’ve taken the same 3 core roles and the same “10% cost QA team” example (mid-sized AA project). The reason why the above is showing an 18% staff ratio rather than 10% as before is because a Dev usually costs about twice what a tester would.

As you can see, the effort or volume value between testers and the developers fixing/debugging the issues found is balanced. You will want to maintain a relative balance for most of the project. If you don’t, one of two things could occur:

A good example of the above is something that happened to me a few years back. A big publisher approached us with a project; 2 guys in their basement built a great prototype and the publisher decided to back it up. After reviewing their timeline of 6 months and the game’s high amount of features, we evaluated that the project would need about 5 testers to get everything checked and fixed. 5 testers, 2 developers… After 3 weeks of starting QA, the testers had nothing to do and we suggested to stop QA until the development team could fix most of the issues reported and add in additional content. I won’t go into all of the details but, unsurprisingly, the game project was completed after more than 2 years. A lot further away than the initial requested 6 months!

Obviously, the second situation is a lot better than the first IF your development team is flexible regarding roles and IF you’ve actually assigned enough testers to your project. It also proves the proposed balance by forcing it on you naturally, when no bugs are left to fix, by adapting the role of your development team.

For the curious types, here’s what the volume ratio graph would look like:

So, that’s it for the basic rules to calculate how many resources QA should take. In my next entry, I’ll show you how to use the Puzzle technique, more specifically how to prepare the high-level QA strategy part of your project plan, step by step, and calculate how long your QA rounds should be based on:

It’s the main internal concrete analysis of your game’s content and features which provides a concrete QA cost calculation to your project. Be there!