Author: by Tyler York

If you're involved in the startup community or even just follow Hacker News, there's a pretty good chance that you've heard about "lean startups" or the "lean startup method." In his bestselling book, The Lean Startup, Eric Ries outlines a framework for small, innovative teams to more efficiently find product/market fit for new products. At its core is a focus on evaluating product design decisions based on user data gathered from scientific experiments. Eric argues that by making "validated learning" your key goal, you shortcut your time to building a wildly successful mass market product.

I recently finished Eric's book, and what struck me was how many of the same principles could be applied to game companies. Eric Ries pioneered the "Lean Startup" method while at a game company, IMVU, yet many people that I've talked to in the game industry feel like this model is more applicable to "startups" than game companies.

I disagree: the product design decisions of a startup mirror the game design decisions of a game company, and both startups and modern game companies use customer analytics to drive their decisions.

There is no reason why game devs shouldn't strive to make games more cheaply, and that can be accomplished by incorporating the Lean Startup method into their production process. I've spoken to many developers who are trying it right now.

So what would a "Lean Game Development" process look like? How can game developers adopt the Lean Startup model to be more appropriate for building a game?

Minimum Viable Game

In The Lean Startup, a minimum viable product is the bare minimum that you need to build in order to test the assumptions of your business. A minimum viable game is similar: you need to build the bare minimum game experience necessary to prove that people find your core game mechanic engaging. What this looks like is going to depend on the type of game that you're creating and what game mechanic you want to test.

So what goes into a minimum viable game?

In order to test the core gameplay mechanic of your game, you need to do less work and spend less money than you think. You just need to build out the first level of the game, which will provide the framework for the experiment and give you valuable feedback on your initial onboarding process. Building this first level should be done as cheaply and quickly as possible so that you can start learning immediately. Here's some tips for staying lean when building your MVG:

Use existing code. To quote Raph Koster, "there must be a 95 percent overlap in what the code does between all FPSes, but somehow FPS developers have giant programming teams."

There's no need for you to fall into the same trap. Use github, open source projects and other resources to cobble together the basic facets of your game quickly and for free. This also gives you valuable perspective on how the system will work together and what parts you need to code yourself later.

Use very basic art and sound. There are great resources out there for free game art, unit sprites, and game sounds that you can use until you get your own IP. You can also get enough art for your first level by doing a small contract with artists that you already know, or reaching out to new ones on artist communities or /r/gamedevclassifieds.

Only include the necessary features. If the feature you're building isn't required for the player to run the game and complete the experiment, then you don't need it. Don't build a save game system, variable graphics settings, or even a starting menu into your MVG. Each of these features wastes your time, increases the number of things that can break, and doesn't contribute to your core goal. Just put the level online and let players have at it.

Setting up a Game Experiment

When setting up your first game experiment, you should have a specific goal in mind. A typical goal could be to prove that players enjoy the core gameplay of the game, or that players can understand the game mechanics and adapt to harder enemies over time. From your goal, you should be able to determine your hypothesis, or what you expect to happen, and the metrics that you can use to measure the results.

Hypothesis. Your hypothesis is what you expect will happen when players play your game. A good hypothesis will include a question to be answered, a characterization of the user and why they would act in a way that would validate the hypothesis, and a specific, quantified expectation of what will happen. You should base the metrics you use on what you need to measure to validate your hypothesis.

At first, it may be hard to determine your initial hypothesis, so I find it helpful to think of the game itself as a funnel. You want players to start playing your game, pick up their first weapon, kill the enemies, etc. At each step of the funnel, there will be drop-off as people get lose interest, get lost, or skip a key interaction entirely.

To start, you should hypothesize that a certain amount of users will get to the end of the level that you've built. Over time, you will want to optimize certain interactions or milestones depending on your game.

Metrics. To establish your metrics, determine the key parameters that will define the success or failure of your experiment. Metrics should always be quantitative, because only hard data can give you the definitive answer that you need to make tough decisions.

In order to obtain these quantitative results, you will need to either build or incorporate some basic analytics into your game. There are numerous good game analytics platforms that you can use for free, including Playtomic, Apsalar, Claritics, and Mixpanel. Just make sure your metrics are functioning correctly before you start your experiment, or else your entire test might go to waste!

Running the Experiment

Find your testers. The advantages of building a MVG are that you can immediately see if players take to your gameplay idea. Player feedback is 10 times more valuable than internal testing because it ensures honest feedback, removes the chance of preconceived bias, and puts your product in the hands of your actual customers.

However, one of the points of feedback that I got while writing this article is that most players are no longer willing to accept half-baked games on mature gaming platforms such as Facebook. I would counter this concern with one of the key lessons that Eric learned from his time at IMVU: no matter how much your product is lacking, early adopters that love it will still use it. The early adopter community that you should target will differ by game type, but finding an early group of excited alpha testers is not as hard as it seems.

To find your first group of testers, you can always start with your friends and family. However, most people don't have enough interested friends and family to provide statistically significant data (your tests typically need at least 30 data points, aka people, to be statistically significant). To bridge the gap, you should seek out communities of players that are dedicated to the type of game that you're testing.

For instance, if you were looking for hardcore strategy gamers, you could look at the Strategy section of a large gaming site, a Strategy game-focused site, or the site of a Strategy game company that is similar to yours. You can also seek out communities that support indie game development, like /r/IndieGaming.

In my experience, a simple forum post by an indie developer asking for participants is usually well-received. The advantage of using this tactic to get your initial group of testers is that the players will already have a relationship with you when they try your game. This means they will be more likely to forgive your game's experimental quality and give you valuable feedback.

Another method for acquiring test players is to use paid traffic. The advantage of paid advertising is that you have the ability to turn on traffic whenever you need to test something. While this does require a crash course in internet marketing, with some basic targeting you can keep your costs in a reasonable boundary.

Certain platforms, such as Facebook, offer laser targeting options that let you sort by game type, age/gender and even crazy stuff like marital status. On mobile platforms, you would probably want to target your ads by publisher, so that your ad is only shown in other games of the same genre.

The disadvantage is that this doesn't give you nearly the same connection with your players as the community outreach method, but you can still collect feedback with a plug-and-play widget such as UserVoice, or simply by offering your email. As long as you communicate to your users that this is a test game and you're looking for feedback, you should still be able to get some sympathy from a paid player base.

Measure the results. The most important part of running an experiment is tracking your game's key metrics and evaluating the results. Before your experiment begins, make sure that you are specifically tracking metrics related to the hypothesis you are testing.

To make your results simple and actionable, it's best to have one key number to base your decision on. It could also be valuable to assure that all of your traffic in a test comes from a single source, such as a fresh URL that you only distributed to one forum, to cohort your results.

Then, give yourself a timeline for the test so you have a set date to stop collecting data and start building based on your new knowledge.

Once you have successfully completed an experiment and have a data set, you need to evaluate your results. While it might be tempting, resist the urge to "eyeball it" and declare a winner: even large discrepancies in your numbers may not be statistically significant due to small sample size.

Instead, use an A/B statistical significance test to confirm your results. Sheets like these automatically take your sample size into account and give you a conclusion based on your data. Let me give you a quick rundown of the terms so you understand the sheet:

Now, your goal is to communicate the results to your team. I find it best to summarize them in a small table to make it easier for the rest of the team to digest. With this table, you should display the results, whether your results validated your hypothesis, and what you learned from the results. Lastly, you should recommend the next steps that the experiment should take. Below is an example table, with details redacted, from a recent player marketing test that I ran for Betable:

With this framework, anyone can quickly absorb the results and what you've learned from the experiment. You can also recommend next steps quickly in bullet points below the table. For instance, this test showed us that players were interested in "win real money" marketing copy, but interest does not necessarily denote customers. A good next step would be to test players' willingness to sign up based on messaging by creating landing pages for each copy. As with a Lean Startup, your goal is to continuously learn and improve your product in tandem.

Applying Lean Game Development

Building a lean organization overnight isn't trivial, but with a little dedication to this model you can see it pay huge dividends. The most valuable aspect of this process is that it incorporates customer feedback into your game development process, so you know that you're building a game people will love before you even launch.

This can save you untold amounts of time and money when applied correctly, and I think the challenge is in the application. That's why I talked to Ben Taylor of Mechamagizmo on how this model would actually fit into game development in the real world. He told me that some game developers already apply many Lean Startup principles in two ways:

Horizontal slice. It turns out that this post largely describes the "horizontal slice" method of game development and testing. Building a horizontal slice means creating a low-fidelity version of your game with the goal of testing the core concept and game mechanics. Many game developers "widen the slice" over time by building new content or features for players to test. If the game is doing well in its low-fi version, the developer would then add graphics and polish to prepare the game for a mainstream audience before officially launching it.

Vertical slice. If a horizontal slice is the low-quality version of your entire game, then consider a vertical slice to be a fully polished version of a very small part of your game concept. The goal of a vertical slice is to prove mass market interest in a gameplay concept without going into the full production of the game. For this model, the overarching experiment is the success of the game itself versus the hypothesis set by its studio. Mechamagizmo created a vertical slice of an education game called Range that is a good example of this in action.

Conclusion

The goal of Lean Game Development is to apply scientific method to what is often considered an artform. By adding learning to your game development process, you can build better games more cheaply than if you had followed traditional production methods. And by incorporating customer feedback into your game design decisions, you should be able to test new game mechanics and validate your game's market potential before spending time and money to build it.

For more on the lean startup philosophy, I recommend Eric Ries's blog and book. To get a play-by-play manual of lean startup in action, check out Ash Maruya's blog and e-book as well. Keep in mind that while most Lean Startup content is focused on startups and consumer products, many of the same methods and ideas apply to games. It's up to you to take it from here and build a lean game company that uses speed and innovation to challenge the status quo. All I can say is "good luck, we're all counting on you."

[Image courtesy Open Game Art]