We speak to experts from small, medium, and larger game companies to find out how they handle the oft-under-appreciated discipline of quality assurance.
Quality assurance is an often underappreciated and misunderstood part of the game development process. Depending on the studio, team sizes can vary from a single person to a whole team of individuals, responsible for testing, tracking, and communicating any bugs they may find.
Because of this, the workflow can sometimes be markedly different between studios, with teams preferring certain software or processes over others.
Lesleyann White is the principal QA specialist at Failbetter Games, a studio specializing in interactive fiction games like Sunless Sea and Sunless Skies. Before that, she worked on Runescape at Jagex where she worked alongside a QA department of roughly 30 people. But at Failbetter she is the only person responsible for quality assurance.
“The hierarchy is so different,” explains White. “I remember before I arrived, Failbetter didn’t have a QA department, so it is kind of an all-encompassing role, working across all the projects, doing all of the work, all of the planning, whereas in a larger company – for example, when I worked at Jagex…we had junior testers who would be able to be promoted up to a tester, a senior, a lead. Then we had the QA manager who was responsible for the whole of the QA department, all the personnel side as well, such as people management 
Failbetter Games' Sunless Skies
Team17 is a Wakefield-based developer and publisher. The QA department is responsible for polishing their own games, such as Worms W.M.D., but also helping out smaller teams QA test their own projects as well, including hit games like Overcooked and Overcooked 2. The QA department at the publisher is currently 43 people strong, with a range of roles within that team.
Chloe Crookes is a senior QA analyst at the studio. During a recent talk at Yorkshire Games Festival, she explained her role as follows: “The difference between standard QA and senior, in my experience, is generally senior QA have their own team of testers and will oversee their work as well as completing their own. You might also complete test plans and organize test cases.” Crookes' current project is Moving Out, a new co-operative game developed by Devm Games and SMG Studio, and published by Team17.
SMG Studio and Devm Games' Moving Out
Cambridge-based studio Frontier Developments is another larger development studio, which primarily develops simulation games such as Jurassic World Evolution and Planet Zoo.
Much like Team17, they also have a range of different roles within their QA department. Colin Davis, the head of QA at the studio, explains the advantage of this: “Our size allows us to offer a large variety of testing. We have a number of specialized teams working alongside the more typical functionality test approach. Our Functionality testers are responsible for testing all gameplay and features that form the game they are assigned to, while our specialized teams are focused on whichever game requires their particular type of testing at that given time.”
As well as different roles, there are also a number of different tools available to those in QA and these can differ depending on the studio and their own internal workflow.
“We pretty much perform all types of testing at Failbetter,” White explains. “System, integration, smoke, regression, performance, compatibility, compliance, etc. It’s easier to say what types of testing we don’t do at Failbetter: localization and unit testing.” White explains that the former is due to the large word count within Failbetter’s games that would make the process costly and inefficient, while the latter is something that she would like to introduce in the future.
For both bug reporting and project management, White uses Jira as her preferred tool, or has done for the last year or so. As a system administrator, she enjoys how configurable it can be and how flexible it is for both smaller and larger projects -- an opinion that’s supported by Frontier also adopting the same issue tracking software at times for their own needs.
As for other tools, White uses software such as Unity for testing and bug fixing and various other bespoke testing tools, including macros like AutoHotkey and Failbetter’s in-house CMS: Storynexus. That’s in addition to Confluence or Google Docs for writing technical documents. Given the small size of the studio the team like to keep things simple and cost-effective to avoid creating more issues for themselves or overcomplicating their workflow.
Team17, meanwhile uses Excel for test plans and Redmine as their database system of choice. Redmine allows them to give a summary of each issue they encounter, a description of that issue, and enter custom fields to elaborate on which format or platform they encountered the problem. They can also add any relevant screenshots or videos to give developers more information to go on.
“Generally, we’ll always attempt to reproduce issues as many times as we can to get a solid reproduction for it before we give it to the developers to fix,” Crookes elaborates in our interview following her talk. “More often than not if a bug is happening, it will be obvious why it is happening so usually that’s just a very easy thing to do…But sometimes you have edge case issues where you’re like I’m not quite sure why this is happening...In times like that, we just have to provide the developers with as much as we can give them and then kind of see if we encounter it again.”
Frontier Developments' Planet Zoo
According to Davis, Frontier has gone a step further to create their own bespoke tool that can interface with several bug databases. This way Frontier developers can template much of their bug writing, making everything more efficient and compatible overall. On projects, they apply a variety of testing based on the requirements of a particular game. This includes everything from certification to compatibility to localization, user experience, destructive, and automation.
Davis explains, “Automation is an important part of what we do, as it allows us to automate some of the more repetitive work like placing every single asset down in a game like Planet Coaster and have the machine check that they can be placed in the game. This frees us up to look a little more closely at what the game experience is like.
“In addition to this, we will also use automated testing to create a range of tests to help us with each game we work on. These tests can range from simple things like ensuring you can start the game on any given platform on a newly compiled build, to more advanced things like flying a ship around in Elite, landing at various stations and then working out what the best trade route might be from the information presented. This type of automation is closer to what a player might be doing in the game and can aid in catching issues towards the more experiential side.”
Both White and Crookes also elaborated on the importance of ‘regression testing’ or ‘halo testing’ as it is sometimes called. That is, looking around a fix to make sure a change hasn’t impacted the build elsewhere.
“What we would do is directly test the issue at hand and then, for example, if they’ve changed the way an achievement unlocks you would then shift your focus to a broader look at that achievement,” Crookes states. “So, for example, if you need to complete a certain level to unlock it, maybe you test failing the level and make sure that doesn’t unlock it. So, it is kind of blind testing around it, but you also have to pull things from past experience, so if you know something similar has happened in another game you can test around that.”
One thing the teams all attest to is the value of putting a game in front of a players. This is especially true in relation to the issues of accessibility and balancing.
White explains, “With regards to accessibility testing, it’s important for QA teams to perform it, but I feel there’s no substitute for putting the product in front of the target audience. Having specific test groups evaluate a feature often raises aspects about the design’s accessibility that you hadn’t considered. It’s one of the reasons we put our games in Early Access. It allows us to get the game in front of players with a wide range of accessibility concerns and gather their feedback.”
Recently, for instance, Failbetter ran a new beta for an interactive map (embedded below) for Fallen London. As part of this process, they reached out to those within their community who use a screen reader to hear their input about this new addition.
Team17 meanwhile perform regular usability tests, where they invite members of the public to get hands-on with their games and test how they are balanced. This allows them to get feedback from individuals unfamiliar with the game being played, which has proven an invaluable asset.
“We get people from the general public to come and test the games and provide feedback,” says Crookes. “I think it’s really useful for us, because the QA teams can be on projects for like sixteen months and sometimes because we’re like so good at the game sometimes we can make it a bit difficult as well. And it’s hard for us to take a step back and go ‘Right, this doesn’t actually play very well.’”
For those thinking of getting into QA, I asked what steps people can take to make themselves stand out and what studios are looking for in terms of a desirable applicant for a QA position. This is what they had to say:
“When it comes to starting out, try not to treat QA as a foot in the door for the games industry,” argues Davis. “Quality Assurance is an integral part of the development process, it’s a great career choice for anyone who’s interested in games and wants to make a difference.” He gives some examples of how to develop your skills. “If you’ve ever tried speed running a game, try enhancing these skills without relying on information you find online. The skills and abilities you pick up from doing this can be extremely useful in QA!”
“The best thing is to sell the fact you want to be in the industry, not the fact that you want to play games,” answers Crookes. “People will apply thinking it’s just playing games all day and they won’t last very long, because it is quite a tough job when you have submissions coming up and deadlines. I think being able to prove that you want to be there because you have a passion for the industry is probably the main thing. If you’ve made games at home, show them, and say, ‘Hey, look at all this effort I’ve been putting in. This is where I want to be. This is what I want to do’. I think you just really need to be able to sell it.”
You May Also Like
