Pitfalls In Automation

Invariably, when we start talking about quality assurance and testing it’s not long before the talk turns to automation. That’s understandable. Testing a modern piece of software is a monumental task by any measure and the increased scope that comes with developing an AAA title only magnifies the challenge. Headcounts rise, costs expand, and coverage seems impossible. It’s only natural to look for novel solutions to these problems and automation instantly becomes alluring. However, as is so often the case with magic bullets, things are not so simple, and there are many pitfalls inherent in most automation strategies that so often have the effect of further starving you of resources and sending your engineers into a black hole of ROI. So, to that end, allow me to present a Cliffs Notes version of my GDC 2012 talk,‘The Automation Trap: How BioWare Engineers Quality.’ It is my hope that this somewhat contrarian view of mine, which I’ve evolved over the last 9 years at BioWare, can better help you be successful with your own project’s automation strategy.

Common Assumptions – Our First Mistake

On my early projects I had my hand in numerous automation strategies both at BioWare and at other studios. These have ranged from the quick and dirty to the elaborate and painstakingly architected. Some I’d be embarrassed to show you today, while others I could wax poetic on into the wee hours. Technical proficiencies and deficiencies aside, they all shared a similar trait – they didn’t really work! Oh, to be sure they did perform their job. Multiple machines would spin up around the office of their own accord, birthing automatons that would diligently run their suite of tests before reporting their results and fading into oblivion. But they didn’t really do what we needed them to and not with any meaningful cost efficiency. We had spent valuable engineering resources building and maintaining a system that failed to approximate the simplest manual testing.

It wasn’t until Dragon Age: Origins that I realized that I, like so many engineers I think, had been operating under a number of mistaken assumptions. Obviously, we know that quality comes from testing and fortunately testing is a simple process that can be easily automated so we can use it to detect all the defects in our software, because that is what’s most important. Now, if you’ve been reading Tulay’s articles or if you are a seasoned QA professional you know that everything I’ve just said is utterly wrong. Sadly, this is an all too common view in the industry and if this is where you start your automation plans then they are doomed from the get-go.

The Cost of Testing

For starters, testing is a function of Quality Control, not Quality Assurance.  Is it essential? Certainly, but QA is a much broader set of activities that support you in building a better game. It’s easy to equate a manual test with an automated one. They look the same from the outside, but the automated test is so much cheaper to run, right? Well, not so fast. It turns out that manual testing, owing to the human capability to perform countless cognitive evaluations at the same time and its ability to adapt to novel circumstances, far outpaces the rigid, brittle limitations of automated tests. 

tests

Perhaps automated testing might win out on cost, but certainly not on value. I’d like to think that a tester’s salary is more than fair value for access to the full capabilities of the human mind. Given test-script writing and maintenance, infrastructure costs and, recognizing that time spent developing an automation framework is time not spent developing the game, can you engineer a better solution for less?

Additionally, automation is software too, and that means it is as prone to defects and in as much need of testing as any other software. Not only that, but given the rapid schedules of game development, along with system overhauls and continual changing of scope, you can expect to see your tests break… a lot! By the time developer churn has settled, and your automation framework has hardened, the game has probably already shipped.

Defect Prevention, Not Detection

Has your project ever suffered from a lack of bugs? Chances are, despite your best efforts, your product is going to ship with countless known defects. Many automation strategies are focused largely on bug discovery with little overall effect on the project. It’s far better that we employ our technology and engineering know-how in defect prevention strategies. I’ve often found that the earlier in the development pipeline that we apply our automation the more effective it has been.  You will find more bugs than you can fix, so focus on fun. Support initiatives that allow you to decrease iteration time and increase developer velocity.

Defect

A Brief Thought on Automated Regression Testing

Most of the automated tests that I’ve seen would be best classified as regression tests and their effectiveness is usually trumpeted in terms of how many times they’ve been run on a project: “Look, here! We’ve run over 200,000 regression tests! That’s a half-million dollars in savings!” This absurd measurement ignores the necessity that your test provide useful and interesting information to be of value. This goes for both manual and automated tests. Verifying a button press, for example, hundreds or even thousands of times provides very little information and even less value. When a test becomes dull and repetitive, that is a sign to move on, not to automate.

New Assumptions and the Road to Success

So, what if we assume that quality comes from iteration and that manual testing is not particularly costly and that it is wholly distinct from automated testing? Additionally, let’s assume that defect prevention is our primary goal and that while we will use automation, we will use it sparingly and effectively. Where does that put us? Well, here on my team, we think it puts on the road to success. Here’s why…

Focus on Improving Iteration Times

Iteration is where quality comes from. Improving developer velocity is like discovering additional development time in your schedule. This is why automated smoke tests are usually so successful. By catching defects before the code goes out to the rest of the development team, your developers will be able to keep working without fear of getting blocked or bogged down in broken code. Defects are typically introduced during the earliest stages of development, by focusing our automation on code as it’s written and on the resource pipelines, we can resolve issues before they have significant impact. Unaddressed defects accrue technical debt and knock-on effects.

Build a Better ‘Rock Star’

When we do target testing, we’ve consistently found that tools assisted testing yields more value than test automation. As an engineer, what tools can you provide your testers to make them more effective? By giving your testers software that allows them to expand their capabilities you can turn them into ‘rock stars.’ Can you write a tool that allows them to file bugs faster, and more accurately than ever before? Or perhaps, give them a way to test multiplayer by themselves? These are places where you can have a huge impact on the project.

It Only Counts If You Finish

Automation is a big investment. It’s easy to lose sight of the fact that you are trying to make a game and not an automation framework. You don’t ship your automation. It doesn’t pay the bills. Certainly, it can make the difference between hitting your deadline at quality and falling flat on your face, but you want to set modest goals that return meaningful results for your project. Identify primary development challenges and target them. Your major risks might not be what you think they are! Finally, remember that time spent developing automation is time spent ‘not-developing’ the game. Make sure it’s worth it!

Final Thoughts and Four Rules to Live By

I like to run any ideas we have through our Automation Axioms. These are just four simple rules designed to ensure that any work we take on will have positive impact on the project. They are a sanity check, and an idea must be well thought out to justify departing from them. Put simply, they are as follows:

If you are considering using automation on your project, make sure you have a solid test strategy in place first. It’s important that you understand the needs of your project and the risks you face. Test might not actually be a top priority. Oftentimes, what benefit we hope to get from automation we can more easily get with robust telemetry but, of course, that’s another conversation…