Author: by Paul Tozour

How do you best manage risk when creating a game? Using this article and the attached spreadsheet, you can better identify the problem areas in your game and get a sense of whether any decisions you are making actually make business sense.

Risk in Game Development

Ours is a tumultuous industry, rife with unexpected project and studio failures.  There are countless stories of canceled projects, unanticipated multi-year delays, cost overruns, mass layoffs, and unexpected project quality problems.  Often, these surprises come from well-known and well-respected studios with a long and storied history of producing high-quality projects.

On the technology side, we sometimes select platforms and engines poorly, postpone bug fixing until it's too late, build undisciplined technical teams to with inadequate standards and poor coordination, and incur mountains of technical debt while underestimating the risk of avalanche.  On the creative side, we can hire poorly, fail to develop our creative talent or the teamwork they need to flourish, fail to develop a coherent creative vision, or underestimate the costs of major late-stage creative changes.  On the marketing side, we sometimes inadequately analyze our markets and misunderstand which features our customers truly care about. And we take enormous teamwork risks, far too often short-changing priceless talent and underestimating the risk of low morale, teamwork failures, and employee turnover.

On the surface, these all look like completely different problems.  But a closer look shows a common thread underlying many of these project failures: as an industry, we often fail to accurately gauge and manage risk.  We very often make decisions by instinct or guesswork, let our excitement for a project blind us to its faults, or bring too many biases from past experience onto new projects without doing the work to truly analyze and manage the risks inherent in each new project.

Why do we so often fail at assessing and managing risk, and what can we do to fix it?

In 2009, I enrolled in a Penn MSE program co-sponsored by the Wharton School of business, in part to learn better answers to these questions.  While there, I had an unexpected and priceless opportunity to learn Discovery-driven planning ("DDP") directly from its co-inventor, Ian MacMillan, and from Professor Ron Pierantozzi.  After graduating from the program, I used discovery-driven planning to plan the development of the iOS/Android strategy game City Conquest, as discussed in the recent postmortem.

DDP has proven very successful in countless organizations outside the game industry for planning and managing risky and innovative projects.  There is no magic bullet for managing risk, but the DDP planning methodology can help you identify which risks can and should be managed, and can go a long way toward building a risk management focus into an organization, beginning at the very earliest stages of planning.

Nearly all of the ideas in this article are adapted directly from examples by Professors MacMillan and Pierantozzi.  This article is intended only as a brief and simplified introduction to guide the reader through the DDP process.  For readers interested in a more thorough introduction to DDP, we recommend the seminal article from Harvard Business Review and the books Opportunity Engineering and Discovery-Driven Growth.

In this article, I will explain the discovery-driven planning process and illustrate its application to games with an extended freemium game example.  However, bear in mind that DDP can also be used in other many aspects of development, such as planning the development of new technologies such as tools and engines. 

How DDP is Different

Every new project is a learning experience, with countless surprises along the way.  A good planning methodology should attempt to capture that learning.  DDP places learning at the center of the planning process, with an emphasis on maximizing the amount of learning per dollar spent.  In order to achieve this, it introduces several key differences from standard planning methodologies.

First, traditional planning works in the forward direction, starting from a viable model of a project's likely costs and revenues to estimate the venture's profit margin.  Discovery-driven planning works in the opposite direction, asking you to state your required profits clearly up front, before you project any costs or revenues.

It should never be enough for us to simply plan a project, estimate our likely profits, and then green-light the project.  We should know in advance what we are trying to achieve and where we want to end up.  It should never be enough to know that a project will be profitable; we should start by asking how profitable we need the project to be.  Before you even begin planning, ask: what profit margins and ROI do we require in order to even make this investment acceptable in the first place?  What level of costs are we willing to bear to reach that goal?

By forcing you to state your required profits and ROI up front, DDP ensures that you clearly define a set of unbiased hurdles to test your project against.  If your project's costs and revenues can't clear those hurdles, that gives you a good justification for abandoning the planning exercise at that point and moving on to other, more promising ventures.

A second major difference of discovery-driven planning is that it focuses explicitly on learning.  The DDP process asks us to separate our knowledge -- the things we can state with absolute certainty -- from our assumptions.  For example, if we put in a spreadsheet that Apple will take 30 percent royalties from a product on the App Store two months from now, that's a well-known fact and we unlikely to change in the next two months, so it counts as knowledge.  On the other hand, factors such as project costs, future sales, development time, customer retention, and user demand for our product or any of our in-app purchases are all highly uncertain until they actually happen, and we can only estimate them within a certain range.  They count as assumptions.

All of our assumptions are ranges of possible values, not point estimates.  Although the initial stages of discovery-driven planning ask us to pick the most likely values for our assumptions, the later stages of planning require us to specify a full range of possible values for each.  As product development proceeds, we should be able to gradually reduce the range of uncertainty around all of our different variables.  The goal is to transmute our assumptions into knowledge until there are as few unknowns as possible by the end of the project -- or at least, until we can be confident that any variation inside the range of all of the unknown variables will not sink the project below our profitability hurdles.

By using ranges, we are acknowledging the unknowns, and then working to determine which assumptions are the most important using a sensitivity analysis.

DDP introduces the concept of the assumption-to-knowledge ratio -- the ratio of all of the collected uncertainty around our project against the things we can state with certainty.  DDP focuses on reducing the assumption-to-knowledge ratio throughout the project lifecycle as quickly and cheaply as possible, asking us to prioritize those efforts that will offer us the greatest reductions to the assumption-to-knowledge ratio at the lowest cost.

This allows us to build any number of sanity checks into the plan itself, so that if our assumptions turn out to be wrong at any point and the project no longer appears likely to meet our hurdles, we can either abandon it or attempt to adjust course with minimal financial loss.  We never want to be 90 percent of the way into a project before we realize we face intractable technology obstacles, that our external development partner has delivered unusable game assets, or that our expected customer base does not exist.

DDP asks us to attempt to nail down this uncertainty as early and as cheaply as possible, prioritizing those efforts that will reduce the assumption-to-knowledge ratio the most quickly at the lowest cost.

Finally, DDP also places a strong emphasis on redirection.   When we find that our plan no longer meets our initial financial hurdles, we're encouraged to attempt to modify the plan by scaling it up or down, finding ways to cut costs, trying different revenue models, or otherwise adjusting it in a way that will get it past our financial hurdles.

As a result of all of these factors, the DDP planning process grows beyond an exercise to simply estimate project scope and viability and becomes a tool to actively identify and manage risk.  It is an exercise in disciplined reasoning that forces us to clearly articulate what must be accomplished and how we plan to accomplish it, separate the knowns from the unknowns, and plan in a way that will expose the unknowns to the harsh light of reality as quickly and cheaply as possible. 

Stages of DDP

For the sake of this article, I will separate DDP into seven steps that match the seven parts of the attached DDP spreadsheet. This is slightly different from most presentations of discovery-driven planning, but the overall process is broadly similar.

At step 4, we compare the expected costs and revenues of our project developed in steps 2 and 3 against the required profitability from step 1. If the project's costs or potential profits don't satisfy our hurdles, we stop here, and attempt to re-plan. There is no point proceeding to steps 6-7 if we can't find a way to adjust our plan to meet our profitability hurdles.

If the project does measure up, we move on to sensitivity analysis and checkpointing:

Note that Checkpoints are not the same thing as production milestones. The Checkpoints should serve as our overall production goals and drive the definition and scheduling of our narrower production milestones. Checkpoints define your plan to learn. 

Sample Project

To illustrate discovery-driven planning, we'll build an example of a freemium mobile game to be released on iOS and Android.  We assume that this game is published by a major mobile publisher and is able to use that publisher's network to drive a significant number of new installs every month.  We will drive revenues entirely through in-app purchases.

The numbers used in these examples are guesses and are not intended to be accurate representations of what these values would be for any other project. 

Similarly, the choice of a freemium model is only for the sake of this example.  You could just as easily use DDP for a paid-up-front revenue model, a subscription-based model, an ad-driven model, or any other.

Let's go through the seven steps one by one and show how you would build the DDP for this project.  All of these steps are available in the separate "DDP Template for Games" spreadsheet.

1. Specify required profitability

We need to state our required operating income and operating margin up front.  Assume that our executive team tells us that the company expects a minimum of $1.3 million in profits and a 20% operating margin from this project.  This implies that we'll need to make at least $6.5 million in revenues and no more than $5.2 million in total costs.  These are our initial hurdles for the project.

Here and in the attached spreadsheet, we will use numbers in blue to represent requirements.  Numbers in grey cells represent calculations based on previously stated requirements or assumptions.  Numbers in light orange cells represent assumptions.

2. Estimate Revenues

In this step, we'll estimate our most likely revenues.  Assume that our marketing department estimates 75,000 new users per month with a simultaneous launch across multiple platforms and a major marketing effort assisted by our publisher's cross-game promotional network (again, these numbers are not to be taken literally). 

They also believe that we will retain 60 percent of new and existing users each month and lose the remaining 40 percent.  We assume the game will have an effective lifetime of 3 years.

Based on this, we project average Monthly Active Users (MAU) over the project's lifetime -- this is calculated in a separate page on our spreadsheet ("MAU").  Our marketing department also estimates that 15 percent of monthly active users will be active users on any given day.

Also note that we list a number with each assumption in the third column.  This is a unique identifier for each assumption that we will use to refer to it later, during the sensitivity analysis stage.

Again, I remind the reader that all figures presented here are essentially arbitrary and represent semi-educated guesses for the sake of this example. 

If you intend to use the attached spreadsheet, do not use these numbers; do your own research to determine the optimal values for each assumption.

Now, we'll estimate our conversion rate.  We assume that 3.5 percent of active users will convert to paid users, which allows us to combine this with our DAU and MAU calculations to estimate how many customers we expect to be purchasers every day and every month.

Next, we'll segment our paying customers into three categories: high-spending "whales," medium-spending "dolphins," and low-spending "minnows." 

Assume that we estimate that 10 percent of paying customers will be whales and 35 percent will be dolphins, leaving the remaining 55 percent as minnows. Then assume that our marketing department has estimated our likely ARPPU (average revenue per paying user) at $29, $5.50, and $0.90 for whales, dolphins, and minnows. 

This allows us to combine our ARPPU numbers with our customer base percentages and our previously calculated daily and monthly purchaser estimates to determine our average monthly revenue from each segment.

In the last step, we combine the monthly revenue from all three customer segments, then factor in the platform holder royalties to determine average monthly revenue and lifetime revenue. Note that we do not list the platform holder's revenue percentage as an assumption, since it is known exactly and not realistically likely to change.

Again, the selection of a freemium model for the revenue exercise is arbitrary.  The DDP could just as easily use a paid-up-front, ad-driven, or any other revenue model, and would need to use appropriate assumptions in each case.  The intention is only to show how the DDP process works in practice, not to present any kind of accurate approximation of the freemium model. 

3. Estimate Costs

Now we will estimate our project's costs. 

Using the previously calculated values for the number of new users per month (75,000) and the retention rate assumption (60 percent), we calculate that 45,000 users are converted from new to active users each month. 

Assume that our marketing department estimates user acquisition costs (UAC) of 60 cents per user. Using these, we can then determine our acquisition costs over the expected lifetime of the project.

We'll also estimate our development costs, using our best salary estimates broken down by discipline, combined with the expected work-months for each discipline to give us total salaries paid.  We'll then combine this with estimates of operational overhead and additional market research costs to estimate total product development costs.

Finally, we add these total operating costs to our total user acquisition costs to determine total project costs.

4. Develop a reverse income statement

Now that we know our required profits and operating margin hurdles and the estimates for our actual costs and revenues, we can compare them directly.

The reverse income statement acts as an essential "sanity check" in planning. The "cost cushion / deficit" and "profit cushion / deficit" entries in the table below directly compare our DDP estimates to our initial hurdles.  If either of these is zero, it means there's a serious problem and we should either attempt to adjust our plan to fix the problem or abandon the project entirely.

Plugging in the previously calculated numbers, we get the following:

As you can see from the table above, our total costs of $2.46 million are well within our allowable costs.  However, our expected profits of $1.329 million are just barely above our minimum profit hurdle of $1.3 million.  This indicates that the project in its current state is only marginally acceptable, and we should see if there's any way to adjust the plan to improve our profit cushion.  We will also need to be cautious moving forward to ensure that the project's outlook remains above our profitability hurdle as we learn more and narrow our assumptions.

In this case, we've satisfied both hurdles, so we can move on to the next step. 

5. Isolate key variables and assumptions

In this step, we isolate all of the assumption variables from the previous calculations -- that is, all of the variables with an orange background, which had assumption ID numbers listed next to them -- and list them together in one table. 

Since we have a full 27 assumptions, we'll show only the first 3 here for the sake of brevity:

For each assumption, we also now estimate upper and lower bound values, along with a distribution type (the cells in blue).  These allow us to specify the potential range of uncertainty around each assumption.  You should pick the widest possible range for each assumption that can reasonably be correct based on the best available data.

All of the variables in this example use a normal distribution.  However, depending on the optimization tool you decide to use, you could also use other distributions, such as a triangular distribution, which would allow you to skew the most likely value toward the upper or lower bound as appropriate.  For example, you might estimate conversion rate between 0-10%, but with lower values between 1-3% being much more likely than higher values, and an expected value of 2%.  In this case, a triangular distribution would allow you to simulate the range for the conversion rate so that it was appropriately skewed toward lower values.

6. Perform a sensitivity analysis on your assumptions

In this step, we run a simulation on the assumption ranges built in step 5 to build a sensitivity analysis indicating the effect of each individual assumption on net profits.   This requires the use of Oracle Crystal Ball or an equivalent simulation tool.

At the end of this step, we should be able to make a tornado graph such as the one below, indicating that some assumptions have a vastly greater effect on profitability than others.  These are the assumptions we should work to test as early as possible.  In this case, retention rate, conversion rate, and new users per month are the top three unknowns with the greatest effect on profitability, so we should work to narrow these down as quickly as possible.

7. Develop a list of Checkpoints

Finally, we use the sensitivity analysis developed in step 6 to create a list of Checkpoints.  Each Checkpoint should give us an opportunity to learn more about our assumptions and reduce our assumption-to-knowledge ratio.  If our learning process shows us that the project appears to be falling below our profitability hurdles, we have a choice of either terminating the project or attempting to re-plan.

As much as possible, we try to order our Checkpoints so that we test as many of the most critical assumptions as we can as early in the schedule as possible, and at the lowest cost.

In the chart below, we list 16 sample Checkpoints, along with the set of assumptions that each Checkpoint will test and the cost of completing that Checkpoint. 

Because our retention rate, conversion rate, and new users per month are so important, we schedule several focus group tests as Checkpoints -- in this case, at the first core gameplay prototype, the first full playable version of the game, and the completed full game.  We also determine the total of all the cost values to compare it against our previous estimate for total project costs.

Note that Checkpoints are not the same thing as project milestones.  They are broadly similar, but Checkpoints are more high-level than project milestones and are used to help drive and define the production milestones.

Our actual production milestones should include the ability to run several Checkpoints as part of a single milestone (for example, when marketing and engineering are involved in two separate but adjacent Checkpoints, we can run them in parallel inside the same single production milestone), or to split Checkpoints into multiple milestones (as with the "Finalize game development & testing" Checkpoint in the table above, which would likely be split into several production milestones). 

DDP Best Practices

There are several additional practices that can help you get the most out of discovery-driven planning:

Conclusions

The game industry is constantly changing, often with breathtaking speed.  In an industry that changes so quickly, the greatest danger comes when we stop learning and adapting.  One of the key benefits of discovery-driven planning is that it not only encourages but actively enforces learning at the very highest levels of the organization.

Although change is inevitable and not all risks can be managed or even anticipated, we can only benefit from improving our ability to understand, analyze, and manage what risks we can.

Discovery-driven planning is an immensely useful tool for managing uncertainty and inculcating a risk focus into existing organizations.  It forces managers to articulate what they don't know so that they can orient the project toward learning the unknowns and reducing overall project risk.  It also ensures that the planning process takes the big picture into account, cleanly separates the knowns from the unknowns, and allows the team to learn as much as possible, as quickly as possible, and at the lowest possible cost.

Once again, to download the DDP spreadsheet, please click here.

The author would like to thank Wharton School Professor Ron Pierantozzi for his contributions to this article.