The illusion of skill: how to make a better game in less time

The following article is a converted script from my video “How to Make a Better Game in Less Time”.

Hi, I’m Casey “Boz” Weeks and I’ve been making games for over ten years now.

Way back when I started making games, I spent two years agonizing over every detail of my first game. Once I had a map, skill tree, boss fights, hours of content, and a tutorial, it was finally ready for playtesting.

Now if you’re an indie dev and that last sentence didn’t make you cringe, this article is for you.

You can also watch the video version of this article here.

Many Indie devs, especially beginners, underestimate the real value of playtesting.I wrote this to amplify the importance of getting feedback and give you some ideas on how to approach it.

The short answer to why you should playtest: Playtesting increases the quality of a game design per hour of development.

In other words, you can make a better game in less time.

Quality of Game = Dev Time + (Dev Time * Playtesting)

Before we begin:

Playtesting often gets mixed in with Quality Assurance, so I want to point out the distinction.

Quality Assurance testing (or QA), is about Functionality. Finding bugs, typos and other errors.

Whereas playtesting is about putting your game in front of fresh players to see where the magic happens. Playtesting is for proving appeal.

If you don’t playtest, my hope is that the first thing you do after reading is schedule one. And if you do playtest, then I hope this article helps you playtest even harder.

Most indie devs seem to know they should be playtesting, but, whenever I ask if they are, I often hear a lot of excuses - myself included. The excuses usually center around the following sentiment: There’s no time!

But what if I told you that the designers of the most successful games consider playtesting the highest priority in game development?

Part 1: Everybody’s Doin’ It

Mike Ambinder, Valve’s former Principal Research Scientist says "For us, playtesting is the most important part of the game development process. ... it is the dominant factor that shapes our decisions about what to release and when to release it.”

I reached out to him to see if he had more to say on the subject.

Mike Ambinder:

Mike emphasizes that playtesting is a scientific tool to assess uncertainty and that the answers lie within your players. He had a lot of other amazing things to say, so we’ll hear more from him later.

I often compare game design to blindly stumbling forward in the dark. In this metaphor, playtesting is like a distant lighting strike, briefly illuminating the landscape, allowing you to check your path and make adjustments. (or reveal you’re about to walk off a cliff)

Playtesting is like checking to make sure you’re not about to walk off a cliff.

Schell Games has the following guiding principle: “Playtest Constantly. It’s the Only Way to Know.”

In a 2024 GDC talk, Francisco Souki, the Principal Game Designer of Schell Games credited the success of their games to their rapid player testing workflow. (We’ll come back to their workflow later)

“Playtest Constantly. It’s the Only Way to Know” exemplifies this idea of constantly checking your path in the dark. Playtesting steers your development process towards what players connect with. What the Schell team realizes is that playtesting prevents you from planning too far into the future on just a hunch.

Richard Lemarchand, co-lead of the Uncharted series, says that while his focus in game design today is playtesting, it wasn’t always seen as an invaluable tool.

In an gamesuserresearch.com interview he said, “in the ’90s and even in the early 2000s, playtesting and especially user research, wasn’t very prevalent.”

In fact, it wasn’t until he was working for Naughty Dog in the late 2000s that he says the importance of play testing really “clicked” for him.

The struggle to make playtesting a part of the dev process is touched on in the book “The Secret Science of Games”. The author, John Hopson, has been a research lead for Halo, World of Warcraft, Overwatch, Destiny, Fortnite, and more.

In his book, he writes about how during the development of Halo, Bungie had an initial hesitance to bring playtesting into the fold. But as they began to see the positive results, successive titles allowed playtesting to become more and more integrated with the development process.

Knowing it took the industry a couple decades to catch on to playtesting makes me feel a little less dumb for waiting two years to start testing my own game.

I reached out to John to see if he had more insight into why it took the industry so long to embrace playtesting and he pointed to the growth of online feedback post-release.

Before internet reviews, game developers used to ship games and rarely hear from players.

Now, when a game is released, they receive an onslaught of player feedback on the internet.

He believes that today it’s easier to convince game developers to incorporate a more structured version of player feedback earlier in the process to avoid problems with angry players at release.

This may be more of a AAA problem than an indie problem, but if you needed another reason to playtest: just imagine an onslaught of angry reviews of something you could’ve easily avoided by collecting feedback before launch.

Part 2: How to Trust Yourself

Daniel Kahneman was a nobel prize winner who (despite being a psychologist) became known as the “grandfather of behavioral economics”. If you’ve ever heard of cognitive biases before, he and Amos Traversky were the first to introduce them.

Kahneman says, “Intuition feels just the same when it’s wrong and when it’s right”

Kahneman writes that academic researchers have repeatedly confirmed that professionals often contradict their own prior judgments when given the same data on different occasions. Kahneman calls this “The Illusion of Skill”

But that doesn’t mean you can’t trust your intuition.It’s not until you are confronted with failure, adapt, and then find what works that you begin to build an intuition you can trust.

Kinda sounds like playtesting, right?

Mike Ambinder:

We can turn that into the following algorithm:

Trust in your Intuition = Time Spent Gaining Experience * Speed of Feedback * Clarity of Feedback

More feedback and experience, better intuition.

Imagine a computer programmer and a wealth advisor, both with 20,000 hours of experience in their respective fields.

Which of the two do you think has accumulated the most expertise in their field? Who can make the best predictions about the results of their recommendations?

Kahneman was once given a rare opportunity to analyze 8 years of data for 25 wealth advisors.

Here’s what he discovered:

“...differences in skill were not to be found. The results resembled what you would expect from a dice rolling contest, not a game of skill.”

The reason for this? Clarity of feedback.

The stock market is unpredictable by design. If it was possible to predict the market, it would be completely upended - it wouldn’t work. The feedback is too ambiguous to know exactly what you did wrong or right. There’s little room to adapt.

The computer programmer on the other hand? The compiler will almost immediately tell them if the code worked or not. And they can measure how code revisions improve performance data.

The data is unambiguous and clear.

So whose expertise do you trust more?

Let’s turn that formula into terms of your current game design:

It’s probably obvious from looking at the algorithm that the more often you test and the clearer the feedback, the more you can trust your game design.

Think of playtesting as a multiplier buff on your insight score per hour of game design.

Conversely - if you test too infrequently, you can counteract the amount of time you’ve spent working on your game. (remember that bit about a crack of lightning revealing you’re about to walk off a cliff?)

In other words, playtesting helps you get more productivity out of each hour of game design.

Okay, we know we want rapid and clear feedback, but what does that look like?

Part 3: Speed & Clarity

How often should you be testing?

Well, that depends where you are in the project and your resources.

Remember earlier when I mentioned Schell Games and their rapid player testing workflow?

Francisco Souki says their team aims to have a build ready by the end of every week. Their testing team then puts the build in front of outside testers over the weekend. They start every Monday discussing what the test sessions taught them and then plan what to focus on for the week. All of this will culminate in a new build on Friday, ready for testing.

Others wait til they’re nearing a milestone on a feature and try to iterate as fast as possible at the end.

In my projects, we often enter milestones where we are working on a feature for a month or two, and as it NEARS completion, I begin scheduling tests.

I then aim to playtest as often as I can make updates between playtests. This could be anywhere from a new test within an hour to a test each day to taking a week to make more serious changes.

Obviously, your ability to test will depend on how long it takes for you to build what you’re wanting to test. But you also have to consider balancing the time you spend managing playtesting with the time you spend building your game.

In the case of Schell Game, while they have people who handle the testing and create reports for the designers, the designers still need to watch or attend the tests for best results.

You choose a style that works best for you, but my advice is: don’t get caught up thinking everything has to be perfect before you can test. (more on that later)

How many people should you test with?

Mike Ambinder:

Fortunately, the more often you test, the more perspectives you bring to your design.

Clarity of Feedback

Unlike the stock market, Game Development can receive much clearer feedback. But, there’s a trick to it.

Adriaan de Jongh touches on this topic in a 2017 GDC talk called Playtesting: Avoiding Evil Data. To summarize, there are different ways of testing that can give you useless, confusing, or incomplete data. In the talk  he discusses how to get the most from your playtesting.

He talks about ways to collect analytics:

Surveys, in-game player data, and heat maps.

These can tell you where players are having problems, and it’s good feedback to collect, but it’s far from the whole story. What this data doesn’t tell you is WHY players are having problems.

He says, by far, the best way to get clear feedback, to understand the WHY, is to watch people play your game.

Here’s a link to his video: https://youtu.be/6EUeYu0aPn4?si=N-Iw_EIRbjiN__hy&t=0s

I reached out to Adriaan to ask if he had more advice specifically on how to improve clarity in feedback. He emphasized two main tips:

These are powerful tips, and I want to dig into each separately.

1 - Be Open to Feedback

This involves getting out of your own way, which is a lot harder than it sounds.

Getting clear feedback is a journey of self-discovery. It’s about building skill with listening, empathy, player psychology, and overcoming your inner biases.

Adriaan says

“...it's likely that some parts feel 'core', something that shouldn't change. it takes real courage to look at someone playing, recognize that that core isn't good enough or what you had hoped it would be, and make (drastic) changes to improve the game's experience.”

You have to balance what you wanted from the design, and how the players are reacting to it. It’s a compromise between your vision and what your audience really wants. There’s a dangerously fine line between your ego and good results, and it’s experience and data that will help you navigate this problem.

Mike Ambinder:

2 - Don’t focus on what players say, focus on WHY they are saying it

While everything playtesters say is valuable, it’s also important to take their feedback with a grain of salt to prevent yourself from being misled by what players are saying.

Mike Ambinder:

When you combine watching someone play while listening to what they say, it puts everything into context.

Adriaan says

Testing for WHY

This is where the ethnographic interview methodology comes into play.

This methodology is built to get the most WHY out of a test session. Here’s the key mindset: The playtester is the teacher, and you the student. They aren’t the ones being tested, YOU and YOUR DESIGN are being tested. Make sure they know that. Empower them to be the teacher.

If my playtesters feel dumb, it’s because *I* am dumb.

As they play, encourage playtesters to speak their thoughts aloud and to ask questions, but don’t tell them how to solve problems, at least, not immediately. Just say “mhmm” and scribble notes like a student attending a lecture would. When a player tells you what to change, ask them why they are suggesting the change - you want to understand the WHY.

While they play, you can stop the player to ask some questions, but your main job is to shut up and watch. Be very careful with your questions. Avoid prescribing solutions - don’t ask questions like “What would you think about a double jump?” Instead, ask something like “How did you feel about the movement? What problems did you encounter?”

You want to tease out information to understand the real problems rather than accidentally confirm your biases. Though, feel free to break free of these rules once you feel like the test is over. I would love to talk about playtesting processes and tips more, but I think that’s a topic for another time.

Okay, now you know that successful game designers base their process around playtesting calling it, “The Only Way to Know”, and how to aim for frequent, clear feedback.

Maybe now you’re thinking “Okay, I need to playtest, but....”

Part 4: No Buts, No Excuses

It’s time for common excuses for not playtesting!

Excuse: “My game isn’t in a place that it’s ready to playtest yet”.

In the example I started this article with - I waited 2 years to start testing my game.And after just a handful of tests, it immediately became apparent that the core assumption of the mechanic was just plain wrong, and that I had to completely redesign. At the time I thought the game needed to be “done” before I could test it. That assumption added a year of development.

Test Early

Your FIRST goal when designing a game is to get it to a place it can be tested - first by you and then by someone who has never seen your game before. If you’re months into development and you haven’t tested your core assumptions yet, you’re in danger of having planned too far into the future on a hunch.

Test Earlier

I once spent three months programming a design idea only to finally realize how painfully awful it was when it was up and running.

I threw it out, moved to paper and programmed a tiny tool to track data.

Within 2 days I had a way better design.

Within a week, I had an amazing design.

I sometimes test games before even debugging. So long as there are no show-stoppers, I can guide players around bugs.

In super early testing, if I only want to test the core mechanics and game loop, I’ll break a golden rule of playtesting and shamefully skip in-game tutorials altogether - I just explain like it’s board game night during the test session. This saves me the extra dev time of updating a tutorial alongside a design I’m fumbling with - allowing for faster iteration cycles.

While writing this, I found the following quote by the Interaction Design Foundation:

The key here is: you want to spend as little time as possible chasing the wrong ideas and as much time as possible nurturing the best ones. Always ask yourself: What is the smallest testable version of my game or feature?

Test early, test often.

“It’s the Only Way to Know.”

Excuse: People will take my idea

No one wants your idea. Ideas are free. They’re useless on their own. Everyone believes their own idea is better. It’s execution that counts.

If you're worried you don't have a better idea, I have an exercise for you: force yourself to make a list of 50 new ideas every day until you overcome this fear.

If you’re still concerned - have a lawyer create an NDA for you and have testers sign it.

To find out if your idea is even worth something to players, show off your game early and often.

Excuse: No one will test my game!!!

Finding testers may not be as hard as you think.

Ask as many people as you can to be play testers, because after all:
“It’s the Only Way to Know.”

Excuse: I’m afraid to receive feedback

No one ever actually admits this, but it’s a common concern hiding in all of us. It may be the core concern fueling the previous excuses discussed. Showing something we’ve put our heart an soul into to others can make us feel very vulnerable.

But remember: You are not your work. And your work was created to share with the world. To enrich other people’s lives. It’s not about you - it’s about connecting to them, but... in a way that brings you satisfaction, so stay true to yourself. Your work doesn’t have to be for everyone.

In the book “Board Game Design Advice: From the Best in the World” more than 60 game designers were asked what it’s like to have a bad test session and all of them said a variation of “Great! That’s when I really learn a lot.”

That means your worst case scenario “they hate it!” could also be viewed as the best case scenario “Here are things to work on!”. The true worst case scenario is you don’t get any feedback at all.

The more you protect yourself, the weaker you become.

When Enough is Enough

So the question really isn’t SHOULD YOU PLAYTEST and more when should you stop play testing? How do you know your game is done?

One way to know:

If after the playtest, people don’t want to stop playing, or start asking when they can play again - that’s a good way to know your design is working.

To summarize, why should you playtest?

Mike Ambinder:

Now get out there and playtest!