Deep Management #2: Genealogical Decision Analysis and Game Post-Mortems

To develop a strong decision-making culture throughout our game studio, we need to evaluate our past choices and learn from them. We must interrogate the taken-for-granted assumptions that shape our actions. This is part of a sound post-mortem process. In our book Deep Management, we cover this as well, but here in this blog, we wish to also explore the philosophy behind it.

As we mentioned in our previous post, people evaluate past choices with a modern perspective, which causes hindsight bias. We often aggravate the situation by adopting an ideological view of our past. This relates to the problematic notion of progressive history (a Hegelian view, which we’ll argue actually matters for games!). We’ll come back to these points shortly.

To escape these problems, we recommend an approach that has been used for decades in anthropology, sociology, history, and archaeology — basically in any field that examines human behavior in large groups. By adopting this mentality, we avoid narrative models of history that presuppose outcomes. Instead, we examine the past by deconstructing decisions, which requires seeing the world as the decision-makers saw it at that historic moment. By examining the “game” our employees play as they make choices, we can improve the rules, and thus refine our decision-making culture. We can also learn how to recognize faults, intervene, and correct deviant choices as early as possible. Our method was inspired by Michel Foucault’s work on genealogy and the archaeology of knowledge. In turn, Friedrich Nietzsche’s work inspired Foucault’s, as best epitomized in his book On The Genealogy of Morals. Some of these ideas originate from Thomas Kuhn, as expressed in The Structure of Scientific Revolutions.

Core to our discussion is the concept of a paradigm. A paradigm is the set of beliefs that compose our conceptual model of the world, including its rules, penalties, and incentives. We select our actions based on these taken-for-granted assumptions. While it is possible for people to imagine behaviors outside their paradigm, these behaviors appear nonsensical from within. Revolutions in paradigms change the axioms that constitute the model, which can reclassify behaviors as valid or invalid. Oftentimes, we do not question the fundamental assumptions of our world view. When someone does and produces a sufficiently compelling alternative, a revolution can occur.

As an example of a paradigmatic rule, according to some western liberal traditions, society necessarily tends to justice and/or freedom over the long term — true or not, that is a taken-for-granted assumption. Another example would be the Confucian and Protestant Work Ethics, both of which emerge as an unquestioned intrinsic relationship between work and reward. In game companies, endemic crunch possesses a similar cachet. Many will claim that good games arise from engaged workers, and engaged workers necessarily crunch (in Deep Management, we point to research that subverts that understanding; moreover, even if we ignore the validity of that statement, we point out that many managers reverse the causal link, and claim that crunch creates engagement, which is an altogether different assertion).

Some philosophies — structuralist ones in particular — presume that there are universal abstract laws that must be obeyed by any valid paradigm. This is also characteristic of ideologies, systems of rules that govern large-scale human behavior. Some such systems are teleological, by which we mean that they presume an ultimate outcome (that the arc of history is towards freedom). For these, the individual is irrelevant to the ultimate arc of history. What matters instead is the historical force (for Marxism, class struggle, for Hegel, dialectic clash). A historical ghost guides us to some inevitable fate. When people speak of progressive history, this is what they mean.

This should recall hindsight bias. In many end-of-milestone high-level post-mortems, we look back and impose a narrative. The outcome is known, and we as high-level managers envision the past as if we were disembodied intelligences making choices, instead of as individuals with compromised perspectives on the ground; of course, we as historical ghosts also possess privileged knowledge of the result that the decision-makers lacked. The equivalence between progressive history and common post-mortem processes is not total, but it is close enough that we should be suspicious of methodologies that do not address the biases this engenders.

In most companies, post-mortems proceed in a problematic fashion. They commence with the individual teams, usually in the form of meetings that last an hour or so. Here, symptoms tend to be the focus, rather than getting into the mindset that generated them. Post-mortems at this level devolve into venting sessions as well, which can be good for stress relief, but does not resolve the faults in our decision-making processes. Oftentimes, we never even teach our leads how to conduct post-mortems, nor do we explain what types of results would be valuable. The leads then summarize their notes into a digest, which they then take to a higher-tier meeting of leads. Typically, top management isn’t even at this leads-level post-mortem, but instead receives a digest created by project managers. Eventually, there is a post-mortem at the highest level. In our experience, when managers enter this final meeting, they do so with an already existing narrative. That makes it even harder to perceive the feedback that they are given, as limited as it already is. In many companies, these leaders also interact more with each other than with the floor, so end up even more distanced from conditions at the time. Good results can come out this final post-mortem filter, but rarely have we seen epiphanies. The result of this last step generally amounts to a progressive history, which hopefully has some correlation with actual events. Typically, leadership then tweaks this story and reflects it back to the team in a project-wide meeting, hoping thereby to shape the mood of the floor. While some improvements come from this process, it lacks the radical iterative improvement that we require in the high-stakes innovative game landscape.

To do better, we must pinpoint why significant decisions were made, where we sometimes don’t know which decisions are critical until long after the fact. The high-level “corporate ghost” view of top management provides little succor here, as they are distant from the point of choice. Only the people with an understanding of the original context can properly illuminate what happened. Rather than gathering summarized notes and passing them upwards in digests, the transformations of decision-making culture should arise immediately from the ground-level post-mortems and then proceed to studio management, not project leadership. The outcomes of these meetings should not be the symptoms, but the ultimate diagnosis of the root causes and their resolutions. Naturally, this information should be echoed to project leadership for awareness. It’s okay to mention problems such as insufficient hardware for the local cloud, but what we want to understand is why we didn’t allocate more hardware in the first place.

Now, some companies have explicit policies that high-level leadership must be fully integrated with those working on the ground floor, rather than operating as a distinct core team. We advise this ourselves! It is a Lean viewpoint that all value comes from the floor, so it is dangerous to be separated from day-to-day activities. Good leaders working in this manner will have built trust, so it can make sense for them to participate in these ground-level post-mortems directly, as long as they are confident they won’t suppress the voices of the quieter team members. Bombastic and boisterous leaders, however, must recognize that they’ll probably need to give their people space, regardless of trust.

In these ground-level post-mortems, the team should select key decisions for examination — ones we consider important at that point, which is admittedly a form of hindsight bias. The first problem will be determining whether that decision was conforming or deviant with respect to the decision-making system that existed at the time it was made. Then, whether it was conforming or not, we must determine whether it was a good decision. Here, we must bear in mind that success involves luck, so just because an outcome was positive or negative, that does not mean the choice was good or bad necessarily. If the decision did not conform to decision-making standards, we need to establish the cause. Were the decision-makers unaware of key guiding principles? Were the principles unclear? Did they decide to ignore those principles and if so why? Once we identify the causes, we must determine how to resolve them. Then, we ask whether the designated decision at the time was the best choice. If not, we need to isolate the flaw in our decision-making system that failed to capture this. What error in our assumptions led to an incorrect prediction? If the error is a profound one, we might need to question the foundations of our entire decision-making paradigm, which would no doubt reverberate through the entire studio. Far more often, however, the fault is a small one, or the specific team has unique challenges and so requires unique principles, which previously never arose as a problem.

For a canonical example of how valuable this process can be, consider the NASA space-shuttle Challenger launch disaster. After the explosion, the Rogers Commission Report ascribed the blame primarily to middle management, which they claimed had violated norms. In particular, the report stated that middle management disregarded escalating problems with the O-rings, and then ordered the launch despite risks raised the night before. Supposedly, pressures of funding, public relations, and scheduling drove them to these ends. There the story might have ended, except that NASA possessed a robust decision history. The sociologist Diane Vaughan became interested in this case and delved into it. Using the history and interviews with the staff who had been there at the points of decision, she established that all conduct occurred according to norms within NASA. While hindsight bias and the outside view of the investigators suggested negligence and poor decision-making, from within NASA, in the middle of all the noise of development, the story was less clear. She established that the actual roots of the catastrophe arose instead from faults within the culture. These issues were not rectified after she published her work, and some argue that they were root causes of the later Columbia disaster as well. Her book The Challenger Launch Decision provides a thorough case history that shows the distinction between the two mindsets that we have discussed.

The NASA case suggests some steps we need to take if we want to produce better deconstructions of decisions. Human memory is faulty, and post-mortems happen after the fact — sometimes long after. As we’ve said, there is little advantage to be had unless we understand the context of the decision at the time. This means we must be able to recreate the machinery that generated that decision. We need to understand the principles that were used to reach that choice. We also must understand the perspective of the people who made it — what did the world look like to them?

In many companies, little thought is given to the principles that guide choices. Often, decisions proceed from gut calls. That approach is fragile for numerous reason (as we’ll discuss in later posts or see Deep Management). This is doubly true as the key decision-makers can be busy elsewhere in the company or leave for other adventures. We cannot afford to depend on particular leaders — that is always an eventual failure. We must distill our decision-making reasoning into rules that everyone can understand. Thus, we should write down our principles and evolve them over time as we learn more. Because principles change, we should keep them in our version-control system. In this way, when examining a past decision, we can simply pull up the principles that prevailed at that point in time. Ray Dalio’s book Principles provides an excellent example of how this arose naturally at Bridgewater.

For historical context, we also require a decision history. Many companies already keep the decisions, but frequently do not include the assumptions that gave rise to them. What beliefs did the decision-makers possess that motivated the choice? Did they believe the project was going to ship in six months? When, in fact, it eventually shipped in three years? Along with the choice itself, the decision-makers should record the key motivations that drove it, including how much they believed in them (in a post-mortem, seeing a 50% versus a 90% belief in some condition might indicate different things for corrections to our decision-making system). For this to make sense, we must have an organizational culture where decision-makers can be honest about their beliefs. We have known leadership teams that operated with perpetually fake release dates and forbade any mention of more realistic ones. And yet the decision-makers have to make choices as best they can, against reality as they perceive it. If they cannot list their true motivations, the post-mortem process will be undermined from the start.

Thus, the decision history should consist of entries that contain the decision taken, date, and the artifacts that led to it. Any data store can be used for this, but we strongly suggest that the decision records either be immutable or have modification histories (suggesting a version-control system). In rare cases, someone with admin privileges might need to delete a record because it contains privileged information, but in a transparent organization, that should be unusual.

In fact, while we have discussed these processes for the purpose of post-mortems, in a transparent culture, these systems allow anyone on the floor to continuously deconstruct decisions. This can be used to catch contradictions early, but also helps grow our newer employees, who can learn why and how we make decisions. Any confusion about goals, deadlines, quality expectations, customer profiles, etc., naturally can get resolved via this process. 

We should never reach the end of a project and wonder at past strange decisions. In hindsight, their outcomes can be good or bad, but we should always know how those decisions arose.