Goodbye Postmortems, Hello Critical Stage Analysis

"What's wrong with the games that are out there today?"

I was asked this question one day by a student of mine at AI Centre for Digital Imaging and Sound, where I have been teaching a course called "Business Management for Games" for a number of years.

"Well, where can I start?" I responded. A good place might be is to investigate the way we develop our games. On that note I dug into all of the postmortems I could find online, in various magazines and books, and from discussions with people who worked in the industry.

After many hours, days, weeks and months of reading articles and talking to many game developers about how they produced games, I realized that the game industry as a whole was continuously repeating the same mistakes over and over again. And not only were the same mistakes being repeated over and over again, everyone was using the same process to analyze those same mistakes as if it was some kind of "blind religion". I determined that the process that was being used to seemingly fix these issues was actually sabotaging the efforts to repair them.

A "Blind Religion"

What's this you say -- "blind religion"? C'mon, we've got a lot of really smart and creative people in the industry working with multi-million dollar budgets and with games that now take years to develop. We have loads of people with BAs, Masters Degrees, and even some MBAs and more than a few PhDs. All of these smart people are working away to come up with the next major commercially viable title. And some are even successful at it. Please note the word "some", since the ratio of success to failure is pretty small -- I've heard estimates ranging from 1 in 7 to 1 in 25. What I do know for sure is that the top 20% of games generate about 80% of the revenue in sales.

On one game I worked on, one of the team members complained to the producer that months of working 12-hours-per-day, seven-days-a-week was getting to him. He said he'd like to have at least one weekend off. The producer replied, "This is the game industry -- crunch time is a fact of life." I have a question for all of you game company owners, producers and leads out there: Ever wonder why you have such a high turnover rate at your company? Studies have shown that most game developers will put up with a severe crunch mode for one, or at most two, projects and then they start looking for another company. Believing that crunch time is a fact of life is a result of following the "blind religion" that's so pervasive in our industry.

Why are these same really intelligent souls continuously making the same blunders over and over again? We all work like slaves under constant crunch time, sweating each design idea into a proof of concept. That design idea requires more art for the modelers and animators to create and rework, more code to crunch, more sound effects and music, further iterations of the game, and so on. Then a GTA: Vice City hits the street, everyone pops up their periscopes and does a feature comparison analysis, takes a deep breath, and tries to cram a few more goodies in to make it more fun and commercially successful. All this, and the game still has to ship on the pre-agreed upon date.

Once the game's done, those still standing take a bleary-eyed look in the rear view mirror to try and recall what went wrong. Typically, someone actually has enough energy to do something called a "postmortem," the purpose of which is to presumably never relive this same craziness again and come up with a much better game the next time around under much better working conditions. "We'll not make the same mistakes" someone writes in the postmortem, only for the postmortem to never be seen or heard from again. Then the next production repeats many of the same mistakes.

Law of Diminishing Returns

While by no means is severe crunch time the only problem with game development, I'd like to speak out about the relationship between complaints about the lack of innovation in games and the expectation that working overtime is normal. While I'm sure there are many successful titles that have required some crunch time, my feeling is that the "crunch" is at least partly responsible for the lack of innovative titles we see today. A team may be able to "crank" for a considerable portion of the first iteration of a title, but eventually it will get burned out.

There are numerous studies that show how the Law of Diminishing Returns affects productivity. Other long-term studies have shown that health problems result from a diet of night-shift work, even if the schedule is periodic. Other studies show that creativity is one of the first things to disappear when the Law of Diminishing Returns sets in.

A "Postmortem" on Postmortems

So what is a postmortem exactly? Typically, it's a document written near the completion of the project (or shortly thereafter) by one or more people on the team, who gather information on what happened over the past eighteen months or so during the game's development. The postmortem is designed to document what went right and wrong during the game. But a more cynical definition might be the following:

"A common artifact of the game development process whereby the game industry documents the fact that everyone seems to continuously make the same mistakes"

During my presentations at the Game Developers Conference (GDC) this year, I asked the participants to list both the positive and negative things about the postmortem process as they have used it on their games. I'll start with the negative aspects, since these far outweigh the positive.

The postmortem only occurs at the end of a project. I don't know about you, but with the fast pace of game development, I have a difficult time recalling with a high degree of accuracy what transpired last week, let alone last month or last year. In addition, the phenomenon of selective memory kicks in. We like to remember what is most important to us and not necessarily what is most important to the project. Also, once the game's done, many people are already off the project on holidays, working other games or at another company. If not, the developers are heading in this direction and their motivation to fill out postmortem forms is pretty low.

A postmortem offers no opportunity to solve problems for the current game since the game's development is over. I thought we were trying to make better games while we were making them? What is the point of a system that can't provide any impact to your current project? Not much.

Some team members don't want to "rock the boat" by critiquing senior team members, team leads, the producer or management. This is a normal human tendency. New team members are just getting their feet wet and getting the lay of the land. However, they may have excellent, fresh perspectives on how to improve the process. Whether new team members are experienced developers or new to the industry, I've found that feedback from recent team additions is very valuable.

We all need feedback to improve our skills, both professionally and personally, and it doesn't matter if you're at the top of the totem pole or the bottom. Without a mechanism for constructive critiquing, and an atmosphere that supports it, your team is missing a golden opportunity.

Some long-standing team members feel left out of the process, or feel that their views are just archived and forgotten. We have a term for this: "jaded". It's an unfortunate fact of life that highly creative people who don't feel they are getting heard stop contributing and deliver the bare minimum to get their job done. It's not just a question of working the minimum number of hours each week - you need that all-important emotional commitment that helps create a high-caliber game.

Some veteran developers who have repeatedly filled out postmortem forms feel that their views were quickly forgotten for the subsequent project. Often this isn't management's fault; it's because of the postmortem process itself: it's performed once at the end of the project and then archived, never to be seen again. When the next project repeats the same mistakes, those who contributed to the previous postmortem quickly say "I told you so" and ask the all-important question - "why are we doing this again"?

Postmortem questions/responses are too technical or not their area of expertise. Artists aren't coders, coders are rarely designers, and sound designers don't program. I've seen many postmortem questionnaires littered with questions that only apply to one of these disciplines. Yet someone from another discipline could easily have contributed valuable information, if only the right questions were asked and presented in a different format. Most artists who are responsible for rendering cinematics could probably not answer the following question: "What in your opinion contributed to the failure of the surround pre-encoded LT/RT 48K 16-bit FMV audio files on the PS2 build?" However, the artist might have provided very valuable information, reporting that the audio files he received were of a different sample rate than the video compositing software allowed -- had a different feedback process been used or the question phrased differently. Postmortem questionnaires often ask detailed questions long after the details have been forgotten.

No scoring values were assigned to feedback - there was no way of communicating the importance of the feedback. How can one tell which issues had the most or least impact? Postmortems ask questions, but what about the importance of these questions with regard to their impact on the game itself? We all know that breaking the build is important, but how important was that to the animator trying to get his art visible on the target platform during integration week? Was it more or less important than the poorly designed tool he was trying to use in the same situation? Only the animator knows for sure. Postmortems don't ask these kinds of highly important questions, and even if they did, it would be way too late in the process. The game was late and the art not as stunning as it could have been, all because of a poorly designed tool or unstable build at critical times in the project's development.

Once development starts on the next project, everyone is too busy to go back and check previous postmortems. There is an old quote by Santayana that goes, "Those who ignore history are condemned to repeat it". This is an industry truth. During my GDC presentations, when I asked how many developers actually read postmortems from the previous game they worked on, less than 2% raised their hands. Developers are excited (rightly so) about the next project, and fresh from some needed time off, but who wants to dig up all of the nasty issues from the last project? We know we can do better, right?

Nobody is responsible for fixing or dealing with the issues raised in postmortems. This is a very important point, not just from a personal and professional growth perspective -- also from a game, team and company growth perspective. As I said earlier, postmortems are too late in the process for anyone to fix the problems, much less nip them in the bud before they become larger issues. Because it's too late, many problems just get buried, only to rear their ugly heads as bugs during the release stage of development -- if the project has gotten that far.

The same mistakes keep getting repeated. I talked about this in the introduction. The same problems crop up in subsequent games developed by the same team and across teams in the same company. Apparently very little learning is taking place. The lack of knowledge sharing is a huge tragedy for everyone and stunts team and company growth. For teams and companies to make better games, the team and company must grow together. Postmortems don't facilitate this kind of growth since the information gathered is neither high quality, timely nor utilized in any real positive fashion.

Here's a brief summary of postmortems from Game Developer magazine. See if you can recognize how many of these you've experienced first hand on the projects you've worked on. Note that each one of these issues occurred multiple times in different games:

All of these issues could have been reduced or avoided altogether!

Postmortems generally deal with high-level issues. Many of the seemingly "small" day-to-day items that end up causing larger problems are long forgotten or are fuzzy at best by the time the postmortem is tackled. Yet the micro details are the ones that often had the most impact.

Generally games are late or don't realize their true potential because the day-to-day issues were not properly dealt with early in production. A cumulative effect sets in; many of the small issues that don't initially seem large snowball and became monsters at the end. For instance, a world that's way too big halfway through production suddenly requires lots of sound assets near the end of the project, but by that time the budget is already blown. Later, when the postmortem is being done, people criticize the game audio for being late or sounding poorly. People tend to focus on the high-level results, not necessarily the seed that germinated into the problem.

Feelings and perceptions are reported, and facts are forgotten. When I joined a previous project late in its development, I heard that the game team hated one of the game's designers. At the end of the project, having arrived too late to fully apply the CSA process, I did my own postmortem to determine the reasons why this was the case. I interviewed most people on the team and heard complaint after complaint. I interviewed most people on the team and heard complaint after complaint. The complaints were generally laced with considerable emotion. Perfectly normal, reasonable people began spewing negatives as soon as this person's name was mentioned. It seemed as if a switch was turned on and the negative energy began flowing as soon as the discussion turned to the question of design.

After listening to considerable negative comments for extended periods of time, I finally asked the all-important question. Why? Why did you feel this way about this person? Interestingly enough, they couldn't give specifics. It was all generalities and lots of negative feelings. They would say things like "He does good work", or "It's not a question of his technical ability", and then they'd say something like "He doesn't fit in with the team", or "I don't know, I can't put my finger on it". It was as if the team had developed some type of mass dislike for this person but no one could really put their finger on the reasons why. And it wasn't because these were bad people. They were good people, generally friendly but something had happened somewhere during the game's development that no one could really recall. And yet, I'd say 70% of the team had a real emotional reaction to this one person.

At the start of the next project, a large number of the team members emailed the producer and said that if this same designer was going to be on the team again, they would all quit.

The story, however, does have a better ending. After considerable thought, a recommendation was made to bring in a trained mediator from outside. Even though it was one person against many during the process, the designer joined the same team on the next project and never a negative word was heard again. In fact, things turned around so well that at the end of the project nothing but positive things were being said.

I guess one could say that this is a situation where the postmortem process worked. We found a problem and initiated a solution to solve that problem. But how much time and energy was wasted during the previous game's development because of misunderstandings that grew into hardened, negative emotions over time? What if this kind of thing had been discovered early and solved early?

The previous game's postmortem issues do not apply to the next game. How many true sequels do we all work on? Unless we are lucky enough to work on a franchise product, I'd guess more often than not that the next project we work on will be quite different. Much of the information we gathered during the postmortem process is pretty much useless and non-applicable to what is coming down the proverbial pipe.

Even when we work on a sequel, things change. The AI or sound engine gets rewritten due to unforeseen limitations, pipelines get reworked, proprietary tools are scrapped due to architectural dead ends, processes get revised, critical team members leave and new ones are hired, art gets changed, and so on.

Feedback Loops

Typically game development at the beginning of pre-production sets its sights on a final product by way of a variety of documents that focus on game design, technology design, sound design and art design. This type of planning is basically linear in approach illustrated in Figure 1.  However, what really happens is anything but linear as demonstrated by the Figure 2.

The problem with the postmortem process in game development is that it does not recognize nor permit a feedback loop as the process of development continues. It comes too late in the process to be of much benefit to the current game. Even if previous postmortems were consulted at the beginning of preproduction, the current game is, and will be, different. Developers will face a different set of issues than the previous postmortem offered solutions for.

Compounding the problem is that game teams are generally getting larger, and will no doubt continue to do so. It is now commonplace to have team comprising 50 or more people, and large teams make it difficult for management to continuously receive and process feedback as production progresses.

The currently used postmortem process is fraught with inefficiency and process shortcomings that are no longer applicable to current game development. It is time for a change.

Some Positives About Postmortems

Certainly there must be some positive aspects of postmortem process since it has been utilized for so long. I'm amazed that developers actually take the time at the end of a project to capture at least something about what went right on the game and what went wrong. The fact that there is at least some documentation is in itself a positive, especially in light of the fact that developers work incredibly hard to come up with games that hopefully stretch all aspects of the game industry.

Positives to the postmortem process do exist, as I heard from the audience in my session at GDC. At least some of the developers are asked their opinions, both positives and challenges are requested of respondents, and documentation exists for those that want to read it. For those of us who didn't work on the game, it's fun to read what went wrong on a big-budget project. Compilations have appeared in book form on the topic. I'm sure you can think of a few more positive aspects. Yet the overwhelming message is that the negatives of postmortems far outweigh the positives and it's time to come up with something better.

Introducing Critical Stage Analysis

Let me introduce Critical Stage Analysis (CSA). CSA is a tool that effectively improves your current game by examining its progress at critical stages throughout its development cycle. It replaces the postmortem process.

The beauty of the CSA process is its simplicity. I've found that the most powerful tools are the most simple and straightforward ones, about which people typically say "Hey, I could have thought of that!" The CSA process does not disappoint in this area.

The process is easy. At each milestone (or "critical stage" - usually monthly) ask everyone on your team to respond in writing to three simple questions.

Incorporate a rating scale of 1 to 5 (1 being the most important, 5 the least) for respondents to rank their five answers to each question, in order of importance. It doesn't matter at this stage whether the team leads, project manager or producer feel any of these items are more or less important than the respondents. What's important at this stage is to get an accurate snapshot of what everyone on your team feels are the important items for each question and their order of importance. Of course, if someone submits more than five answers, this should we welcomed. But requesting five issues reinforces that you want important information.

Communicate to the team that you want their honest and accurate feedback, but also let them know that they should be respectful of any and everyone on the team. What you don't want is people complaining for complaining's sake, nor do you want to start a mud-slinging session. You are primarily looking for solutions to issues in their early stages before they become major problems. I find it good to lead with the positive aspects of the project, since celebrating successes leads to many good things, including raising individual and team morale.

Project managers are the best equipped to gather this information from a psychological perspective, because often they do not manage the personnel (teams) directly and are therefore considered a neutral party. It also fits in well with the other aspects of their job description.

This questionnaire (a sample of which is included in Figure 3, below) should be made available to the team no more than three days after a milestone. I've found that if you make this a regular part of your post-milestone art, code, design, sound and lead meetings, the process of gathering the information will be quick and painless. It can usually be done within 10-15 minutes in such meetings. I also recommend having every option available to respondents -- let them submit their responses via paper form, email, or online, and give them the option to submit their answers anonymously.

Once the project manager compiles all of the information into one document, it should be emailed out to the rest of the team leads and producer, and discussed in a team-lead meeting no later than two days after the information has been gathered. The most important points should be discussed, solutions determined, ownership assigned (to either team leads and/or team members) and then presented to the entire team for discussion. All of this should be completed within a week of the milestone that has just past.

During the team meeting, the producer or project manager should go over all of the feedback in order of importance, solutions discussed and ownership assigned to solve the issues within a specific timeline. Follow up is important, and team leads should be assigned responsibility to make sure that the issues are addressed.

At the next general team meeting, the status of previous issues that were dealt with should be brought up first. If there is a problem that can't be solved, then it should be honestly brought forward during the general team meetings and an explanation given. It is important not to hide any issues -- honesty among the team leads when dealing with the toughest issues will go a long way towards increasing team morale, even if the problem couldn't be solved for whatever reason. It is important to publicly acknowledge the issues and to demonstrate that you are at least trying to solve them even if you can't. Issues that are buried will just come back to bite you at a time when it's far too late to do anything about them. (no solutions are available)

During the general team meeting, lead with the positives first by celebrating those things that went well! This is also an opportunity to hail those on your team who went the extra mile or came up with innovative solutions during the past milestone.

Once discussed in the general team meeting, the compiled and unedited CSA should be emailed to all team members and made available in a central repository along with past CSAs. This could be located in the team's Perforce branch or on an internal team web site. You could also have a company CSA website where all teams deposit their monthly CSAs so everyone can learn from each other.

From a management perspective, what's most important is to create a positive atmosphere that will reinforce to everyone on the team that you take their feedback seriously. It may take two or three CSA sessions to do this initially, but if the management team does this correctly you will see magical things start to happen: everyone will fill out the CSA forms happily; everyone will begin to feel part of the process, and most important, part of the project. Their emotional involvement will increase, creating a positive feedback loop that will enhance the potential of the game.

The whole process, including having the team respond, compiling the responses, conducting the team-lead meeting and the general team meeting only takes about two to four hours, total. The most affected person from a time perspective is the project manager, but this is a small price to pay for the wealth of timely information and the benefits it bestows.

So, how do you measure success utilizing the CSA process? You should see fewer issues raised repeatedly, and fewer CSA responses of high importance over the course of your game's development.

The CSA process is simple in design, lightweight, very powerful with many positive attributes that contribute to making better games in a highly positive team atmosphere.

CSA - A Perfect System?

One respondent during my CSA presentation at the GDC commented that I had presented the process as being perfect. Well, I'm flattered but I'm sure there are ways we can improve on it. Already many of my GDC roundtable participants have emailed and said they are using the CSA process as a regular part of their game development with positive effects.

Please email me and let me know how we can improve on the process and I'll share this in a follow-up article or another GDC presentation.