The Uncertainty of Design

This was originally posted on http://www.gamedesignstuff.com

I was facing a frightening dilemma in my first years as a designer. I was supposed to make decisions. Decisions that would create great value. That would create interesting gameplay and perfect retention KPI's.

I had the feeling that I could never know whether what I was designing is “right” or “wrong”. I felt lost and that every decision I made as a designer was merely a coin-flip, a bet on whether that design decision went into the intended direction or not. How could I even justify getting a salary for being the "coin flipper"? I finally came to peace with this uncertainty of design and I try make its implications work for me, not against me.

Let's go

To tackle the problem of "design uncertainty" I would like to introduce the concept of "unself-conscious design" and "self-conscious design". I will briefly explain those two approaches and how they apply to games. Then I'll assess how accurate a design (decision) can actually be by comparing game design to social sciences. At the end, I'll try to share some learnings on how to actually deal with design uncertainty.

Un/self-conscious Design

The Austrian architect and a promoter of "Design Science", Christopher Alexander, describes two approaches to designing: “selfconscious design” and "unselfconscious design".

Katherine Neil wrote a fantastic article on that matter in the game design context. A brief summary: Unselfconscious designs advance a product by modifying it (replacing, adding, removing elements), and then assessing the outcome.  Is it better, keep the change? Is it worse? Roll back the change. This means gradual improvement to a product, step by step.

On the contrary, there is "self-conscious design", where the "design is freed from its reliance on making", because it is based on methods and knowledge of the substance.

 Katherine Neil wrote a fantastic article on that matter in the game design context. She describes in the article that working purely in unselfconscious design (or should I say "as an unselfconcious designer"?) would make it impossible to make intentional changes. Predicting the outcome is only possible by looking at the results of previous modifications of this product. A good example for "unselfconscious design" is A/B testing which is heavily used in Mobile games. A change is made, and a group of players is exposed to that change. That demographics' behavior is then compared to that of a control group without the change. After a time-span the numbers say whether a change had the desired effect or not. This creates a learning about how the system behaves when changing certain condition. But the experience remains a black box.

This is why it is dangerous to copy systems/mechanics from other games. As they can and probably will work differently. Learnings from unselfconscious design methods such as A/B testing, rarely translate to other games. How a system works, depends on the context it's never standing by itself, it is connected to other parts of the game, other systems, and content which is geared towards a certain experience. Shoehorning a system or mechanic into that ecosystem will not create the same experience as in its origin.

As soon as we start asking ourselves why a certain change leads to a particular outcome, we step into the realm of "selfconscious design". The result would be that we can abstract the reasons for an emerging behavior of our design. By knowing why something works, we have the power to apply generic rules to intentionally instigate a certain user-behavior or emotion through our design.  Now we can take a system and try to recreate that experience within the context of our game. This makes Design more formal and discussing it objectively. Those principles, when passed on from generation to generation, will evolve, leading to a sustainable advancement of the whole field.   

Between the knowledge gained through observation, and formalized design rules lies the  "design-instinct", which is developed by a designer undergoing many iterations of modifications and observation and then transformed them into maybe generalizable models. The problem is that designers might draw different conclusions from their various projects. This knowledge is shared by collaboration and discussion. There is no guarantee that the "correct" models are passed on. The evolution is rather slow.  I have observed many people (including myself), making the mistake of generalizing too rash. A good reminder to bring up the Dunning-Krueger effect; which describes our overconfidence due to lack of knowledge!

To counter this, we should try to transform the learnings from observations to generalizable models. This was already postulated by Dan Cook in one of my all time favorite articles on that topic, where he compares game design to alchemy.. A process in which by combining arbitrary elements, hypotheses were formed about the inner workings by making observations (Similar to the beforehand mentioned unselfconcious design). This describes how games are built: there is an idea, we build it, then we see if it works. Based on the outcome we repeat or iterate.

Since we are human beings with limited resources, this process is very inefficient as only a limited amount of insights are possible.

Especially now as designing games has already come a long way, it is harder to gain profound insights with that method. He, therefore, concludes, that the field should mature from alchemy to chemistry. Where the deep understanding of the underlying parts, assumptions can be made about the behavior of those parts (selfconscious design).

But Chemistry has an advantage: it deals with a consistent substance. A hydrogen molecule is always exactly the same, the same experiment can be repeated as every scientist in the world has access to the same hydrogen molecule. With games we are dealing with one of the least consistent and least understood substances in the known universe: The human brain, psychology. "Design" is facing the same challenges like social sciences. Both fields are based on the behavior of independent actors (people) under varying conditions. Unfortunately, people are more complicated than non-meant-physical-objects and behave differently and “irrationally” under varying conditions (it is probably only called irrational because we do not fully understand it yet).  Studies of social sciences are often inconclusive or only apply to specific situations. Does that uncertainty sound familiar?

Economists, for example, are formulating models that try to explain and predict economic behaviors. The more enclosed the subject is, the more accurate the model can be. When going to a higher level problem, models have to be combined, and their uncertainty multiplies, as multiple assumptions and experiments are and interacting with each other. That’s why we have lengthy discussions whether minimum wage has a positive impact on the economy or not. What happens long term? What happens short term? What side effects will there be? All of these things are very hard to predict. Because in reality, a colossal number of variables go into a complex system that, over time, and under the influence of outside factors behaves in a chaotic way.  I would say we are dealing with a similar situation when creating games. There is an endless amount of permutations of the experiences that the game is creating in each individual player's head.

Where the science becomes the craft is when it is decided what model to apply to what situation. And what models to combine depending on the situation. In science, models are not perfect, but they don’t need to be. In science, we always must assume that our understanding is imperfect. But it is good enough to get us to the moon, or to predict the existence of the Higgs boson.   

In the same manner, the design of a game is not a coin flip, by improving our models we improve the accuracy of our predictions when it comes to the players’ behavior or emotions.

The Current State

Katherine Neil writes: "No language of game design, nor anything that could be described as ‘formal, abstract design tools’, has been adopted into mainstream game design practice".  From my experience, this is quite accurate. Every Designer that I've met so far, has some sort of technique, or method, or ways of abstracting and describing things. But each one has their own. It always takes some initial effort to find a common ground when diving deep into a game design discussion. Explorations into the formality are being done by designers like Jason VandenBerghe who is iterating on a psychology based model that tries to explain and predict why people play and what keeps them going: Engines of Play. Also "Project Horseshoe", a "think tank", explores the opportunities of selfconscious designs.

Those are more or less still based on individuals or single groups, creating brief islands of knowledge and insight. In other fields, such as psychology or political, schools form, exploring a specific angle on a topic by which they hope to fully grasp their field. I've not seen that happen in game design yet.  

Designing for the Uncertainty

What became clear to me is that my initial problem of not knowing whether I was "wrong" or "right" was flawed, to begin with. There is no right answer, there is no 100% confidence if I just invest enough time. This can be tough. Many people, including myself, want to mitigate uncertainty and risks. I want to explore all options before making a decision. Reaching a high confidence seems like a desirable goal, but if taken too far it becomes a problem. Some decisions just need to be made, and the confidence level in those decisions is limited.  Time spent on investigating has a diminishing return. At the end of investigating there will be no "perfect design", therefore it is important to know when to stop. Also, the further you wander into the design space in relation to what you already know, the more uncertain decision become. On the other side of the scale: The more accurate our models the higher our confidence can be, yet, validation still needs to come from the test as any model that we can have is far too inaccurate to predict the players' behavior.

Never trust a designer that is overconfident in being right. Especially if that designer can not explain what the applied model or assumption is.

Like a scientist, designers must operate under the assumption that they might be wrong. This is not just a rational precept, it is also vital to keep an open discussion. Chances are high you have had a discussion with someone who had a fundamental belief in his or her idea, without the ability to rationalize it. Don't be that person! And also keep in mind, the more certain you are about something, the more silly you will look when you're wrong :)

Which leads to another important topic. In a team, in production, the designer can also not be the person that is never confident in what they're doing. So how can you deal with the dilemma of knowing that the direction everyone is walking towards might be a mistake?

Conclusion

Designing games is tough. If you feel frustrated with not being sure what you're doing is promising, keep in mind that you can never be sure. Try to see it like a scientist, like an experiment. Also rationalize your hypothesis, then you can communicate it to others. This will earn their trust and leaves enough wiggle room for changes and discussions. More experience as a designer will lead to more accurate predictions, it will not make you more right. Try to have as rational design discussion as possible, the argument should matter, not the person who is making it. As designers we are not entitled to are a more valuable opinion. The only thing that differentiates the designer from any other person on the team is that they get the time to apply their problem-solving skills and abstract thinking to the problems of the game and hopefully end up with a good repertoire of arguments and rationales that can stand on their own feet.