Part 4: “Adopt, Adapt and Improve” – Agile QA BioWare Style

Adopt, adapt and improve was the motto of the Round Table, a social networking and charitable organization for men in their 20s, 30s and early 40s, founded in Norwich, England in 1927.

Its motto was to bring professionals together around the table, adopt methods that have proved sound in the past, adapt them to the changing needs of the times and wherever possible improve them! Sounds familiar?

If you work in agile development you know what I’m talking about. This basic principle of Plan-Do-Check is part of our core BioWare QA Philosophy. Lisa Crispin and Janet Gregory wrote a book in 2009 called 'Agile Testing' (published by Addison-Wesley) which provides some guidance in agile testing. I was surprised to read that the core QA principles we had been following at BioWare were actually recommended standard in the software QA community. It was surprising, but also encouraging in a way.

In this context: Our 10 Agile Testing Commandments are:
 

1. Deliver value to the customer
QA in our studio focuses on defect prevention, enabling/unblocking developers, and providing valuable business intelligence to leads and decision makers in the studio. QA will also communicate important metrics and data about test coverage, game and system stage and quality to applicable development customers and stakeholders. A strong QA department will give a studio confidence that the product will meet the standards that are expected by the developer and most importantly, by the end-user.

2. Provide continuous feedback to the team
Continuous feedback should be provided to the producers, developers and your own QA team. Good feedback is about risk management paired with root cause analysis; it is about keeping the communication/feedback channels open with all relevant teams. The feedback provided, and its visibility, is critical for the success of all teams involved.

3. Enable better communication
In many ways QA acts as mini scrum masters and as facilitators, because production and QA are probably the only two departments that talk to every other development discipline; they need to connect the dots and always keep the big picture in mind. In this role QA helps to ensure better communication between developers and ensures that no miscommunication exists regarding standards and quality expectations within the team.

4. Have courage
to raise risks to quality. QA is neither a gatekeeper nor should they be a bottleneck. We provide information to development stakeholders who truly own quality. Feedback helps development to reach their goals and to react more effectively. QA also needs to advocate for the end-user as much as possible, even if that means we ruffle feathers, we often help to keep developers on their feet :)

5. Keep it simple!

6. Practice continuous improvement
Inspect, adapt and maintain practices and processes - the motto of the round table!

7. Respond to change
There is no "one solution fits all" model. We will do what is best for the project and adapt current strategies and come up with innovative ideas wherever there is an opportunity or a challenge to be resolved. There is one thing that is certain in game development: change. As QA we aim to always stay adaptable in order to react to change quickly and without (m)any casualties.

8. Be self-organized
At BioWare our QA is encouraged to develop new tools, processes and skills to improve work and workflow for QA and for development. Development of new tools and automation is not limited to QA Programmers - we train our Analysts to be able to detect these opportunities.

9. Focus on people
QA often ends up having to define and figure out what a particular feature is intended to do - our wealth of experience across a variety of titles helps us to achieve this. 

10. Have fun!
Evaluating the fun-factor of a game can be a real challenge, but at the same time it can be a lot of fun! QA is a department that is always in the product, constantly using it and seeing it change. QA also often sees the worst sides of a product, so keeping a sense of humour is often very useful and healthy, even if only to keep the developers spirits up and to keep them focused. If work becomes too repetitive without real challenges, we can run the risk of becoming complacent and will eventually make mistakes. 

My secret 11th commandment and our QA Department Mantra is: Keep calm and continue testing. Many of our developers genuinely appreciate the fact that our QA can stay calm and doesn't panic despite the fact that QA typically sees the worst side of everything; this can give developers the confidence to focus on making awesome games!

To quote Clinton Keith, who introduced scrum to game development 10 years ago, scrum is about people over processes - just like a good leader of creative people is more like a gardener than a process mechanic.

QA teams at BioWare stay agile to the point that the organizational structure depends very much on what the project teams need and how it is structured. Yet our main goal remains the same: to ensure that quality is built into everything developers are working on.Â

During prototyping and pre-production of our games, our analysts are mostly involved in the verification of systems since there are not yet many assets to test. These analysts are typically testers that we refer to as Tech QA who have an understanding of the systems that are used to implement the content.

They identify issues in the tools, systems and workflows and mostly support the programming teams. We also have analysts that have expertise in what we call Content QA, focusing mostly on Level Art, Level Design, Cinematic Design, Gameplay and Narrative.

QA Analysts are often embedded with the scrum teams, with one analyst sometimes supporting multiple scrums. As the project starts to progress, more content comes in to test and more platforms start coming online. The growth and complexity of development often requires more testing of added features and developer tools; more regression testing is required to ensure newly added features have not broken anything.

We made the decision that during production it is more efficient to move QA into a pool system to even out the workflow. In the current development of our products we are actually experimenting with different versions of embed and pool models or different team set-ups and techniques such as Rapid Software Testing.

We want to converge our QA processes between our franchises wherever we can and the benefit of running different franchise game-teams in parallel is that we can learn from successes and failures of the other game teams, quickly apply these lessons learned and continuously help our organisation and people evolve.

There is just too much to talk on the subject of the different types of support and specializations we have in our studio (in the context of RPG-QA) that I will not be able to cover in a single article. So, stay tuned and watch this space!

I like the analogy of comparing testers to Military Intelligence, leading people away from seeing testers as gatekeepers (QC vs. QA). In many ways the objective of a tester and an intelligence officer are very similar as it is implied in our mission statement (read Part 2 of this series).

Testers are tasked with providing information to developers and production to do their work better and to make the correct decisions. Just as the Intelligence Officer is not tasked with fighting the enemy in close combat, testers are not tasked with fixing a bug in the code as an example.

We help the developers on the front lines and not being a developer doesn't mean that QA does not have an intricate and highly technical job!

QA often has to think about and simulate strange but realistic scenarios where customers will be using our product in unusual ways. We often have to work with incomplete information, making assumptions and explaining the risks of situations that may or may not happen in the wild.

Personally, I know my team prefers the ninja analogy to that of Military Intelligence, and they like to consider themselves as a "network of ninjas" who work in support of developers and producers of the product. My QA Ninjas have mastered knowledge of a product's systems, its content and tech goals and report on both the functionality and the subjective quality of the product, enabling the rest of the team to make decisions with the best information possible.

Let's take the pillars of game design for an RPG as an example. Imagine, that exploration, combat, progression and story are all gears in a machine - if one of these gears malfunctions, they tend to grind (for example when a story element is missing, the player can't progress). In this case, Quality Ninjas act as the lubricant that makes those gears turn more smoothly. QA support exists at every level in our studio.

Predictability is important to guarantee success, especially during production. In theory you should be able to ship a quality game earlier the more knowledge and experience you have, therefore knowledge is the greatest asset your studio can create!

My blog posts so far have been from the QA perspective. In the spirit of partnership that we are talking about here, I asked some developers at BioWare Edmonton to share their thoughts about our embedded QA and I would like to conclude this entry with some of their responses:

Embed QA is a good fit for the agile approach that we try to take at BioWare, where an interdisciplinary group commits to delivering a feature. The verification of that feature should be considered part of that work, and not as an afterthought. I think that embed QA is a key strength in how we approach game development. If anything, we should be pushing farther with it. [Technical Director]

Having QA team members who are kept throughout development over multiple projects has created a field of QA experts in many fields, including Audio. These people know what they are doing and our review scores reflect that. They don't waste our time, and genuinely care about the art of game development. [Lead Sound Designer]

QA offers reporting on our bug lists, immediate support for things like build failures, and can provide suggestions for improving the game that go beyond just fixing the actual bugs. [Technical Director]
 

I believe that QA provides an objective viewpoint over the project as a whole. As our projects tend to be so large that we often will only have visibility over certain areas, usually the ones that you are working on and the transitions into and out of them, QA provide the missing link. As well, I appreciate that they are not married to the content, as they had no hand in producing it. This better sets the QA team up to appreciate and critique the project as a consumer, rather than a content creator." [Audio Artist]
 

PS: And here's a video for those of you who didn't get the Monty Python reference in the title :) http://www.youtube.com/watch?v=zt8PDTUNyPE