Evolving Test in the Video Game Industry - Part 2

In my last offering, I touched on the move from a waterfall to an agile development philosophy. Over the last ten years we've seen game studios attempt a number of approaches including SCRUM coupled with Kanban. For most, the current iteration is a hybrid of the two heavily influenced by company culture. The goal is straightforward: create iterative development processes that do not overly impact the pipeline. With ever evolving platforms, waterfall wasn't getting the job done.

You can read a more comprehensive discussion started by Seth Sivak about creating an iterative development culture here.

Agile with smaller work-unit control works for faster content development and distribution. But moving test to this new paradigm has proven problematic, as traditional game QA's static workflows and often cumbersome overhead cannot adapt fast enough to meet the needs of iterative development.  

QA's response to agile pressures: assign select game testers to sit with development teams. Conceptually this is on the right track, and has realized some returns over the last few years, but this approach is limited and fraught with risks:

I know the last item seems counterintuitive, but without inclusive retrospectives, the assigned QA tester is accountable only to the development team, with no QA communication required on the critical path. This makes it tough for QA leadership to make adjustments to testing methodologies and requirements, let alone to share best practices across testing groups.

Collocated game testers alone are not enough to create sustainable QA processes, regardless of how talented your QA leadership team may be.  

Center of Excellence (CoE) adoption in the game industry

Over the last few years, as testers are distributed to feature teams, we have seen how QA leadership oversight degrades and best practices aren't shared across teams. One solution is the creation of a Center of Excellence. Other industries use this concept to ensure practices are current and relevant.

For the purpose of this discussion, we want to focus on best practices standardization and management.  Rarely will you find game development studios with a staff dedicated to the maintenance of practices and process. A 2008 post on Agile Elements defined  the CoE as "a team of people that promote collaboration and using best practices around a specific focus area to drive business results" serving five needs: support, guidance, shared learning, measurements, and governance.  This is one of the better definitions as it provides a framework for organization.

A proven approach

With a solid process foundation (CoE) in place, we can take a look at the makeup of the testing group deployed to the feature development team. As noted above, recent history has shown us that collocated testers alone are not the ideal solution. If an Engineer in Test and Software Engineer are added to the mix, test strategy, planning, and automated test creation get addressed.

Here is an example of what an effective organization might look like:

Note: this can look different from studio to studio due to culture

The team makeup can adjust to meet the needs of the feature, but the core group of Engineer in Test, Software Engineer and Tester should remain in place. This type of integrated testing team is a unique position to visualize the entire development pipeline and trace issues from the test cases that uncovered them all the way back through to the feature design phase.

Here is the workflow:

When we are done, we should be in a position to continuously run automated tests against main and script new tests based on issues identified during the initial evaluation. This can be a robust regression suite’s foundation.

While large QA departments at the end of the pipeline had some success in the past, they are ill-suited for a more aggressive deliverable expectation. Small modular teams of Engineers in Test, Software Engineers and Testers are better suited to meet the needs of an Agile development pipeline.  By integrating test (as detailed above) into the pipeline as early as the feature design phase, we lay the foundation for identifying issues an hour after creation versus a year. This will reduce agreed upon risk, increase decision maker confidence, and build the foundation of a robust automated regression suite.

Next up … leadership and effecting change without disrupting the landscape.