Indie Game Analytics 101 (part 2 – five mistakes)

This post if from the featured series on the Unity Analytics Playnomics blog (or on theDylanJones.com). It takes from lessons after launching Battle Group 2, which is a finalist at IGF China and was featured by Apple, Google and Humble Bundle.

There are plenty of pitfalls to trip into when first exploring analytics. But now that we’ve covered basic terminology in part 1, we can go over the best practices for analytic users who are starting out. Again, these analytic practices are vital for not only the business side of development, but for understanding what your players are experiencing from a creative angle.

Usually, the primary goal of game analytics is to measure your game’s retention, and improve it. Early on, you should know:

1.    Don’t pay for analytics

It’s no secret companies make money off your data. Thus, think of yourself as a prized customer that deserves the highest quality tools for that charge! Depending on your development platform there are dozens of options to choose from, we certainly don’t have the space to review each one here though.
 

Statistically, if you’re indie, you’re using Unity. Check out ‘Game Analytics’ for easy integration and some nifty features. (if you’ve done some analytics in the past, know that Game Analytics uses a “bucket style” approach to tracking and needs separate events for funnels which can be eye-brow raising when first adjusting to). If you’re familiar with Google Analytics from your website, they’re just jumping on the Unity bandwagon with an open beta. Stay tuned, I’ve poked around Unity’s official beta analytics tools, which are looking marvelous, you should jump in here.
 

I would recommend you hire an analyst for your current data before paying for data collection, but if you have a fetish for gorgeous graphs and decent predictive tools, Upsight (formally Kontagent and PlayHaven) and MixPanel are good starting options (which can charge between 500 bucks or 4 figures a month even if you 1/10^th sample users and data, and charge during development…).

2.   Only track metrics that lead to actionable insight

When starting out, you will drown in data. I always recommend that beginners only track events in places where there are no other factors to consider. For instance, you get hard (undeniable) insight from checking if a player started the tutorial and finished it. There are other factors that make tracking a player who looked at an item but then chose not to unlock it a much more complicated insight.

If you’re looking to add some fun to the mix, create a separate project for vanity metrics. This may be tracking total coins collected or even player maps created. “1 million+ ships sunk” is always a fun bullet point for marketing.

3.    Track your tutorial / on boarding way more

Getting players understanding and enjoying your game (improving retention!) may be why you did all the set up for analytics in the first place. The first experience a player has is one of the most important parts of your game and therefore should be monitored thoroughly to check for unforeseen hiccups. The best way to do this is by tracking mandatory events for the first 5 (or more) minutes of gameplay and at least the entire tutorial series. Think of each hoop you demand the player to jump through as small milestones (think abstract if you have a dynamic or procedural game) and track them. Clicking start, starting dialog, completing first action, getting to the next level, unlocking new item, etc.

For example, in Battle Group 2, we track dozens of mandatory interactions in a funnel. Here’s an older snapshot of a week’s worth of Android players dropping off through a funnel in the Unity Analytics Beta:

After all this, any decent analytic service will allow you to put these events in a funnel, as seen above. Once you test and confirm your analytics setup, you will be surprised how easy it is to find huge drops between steps where players perhaps don’t understand or had to wait for a long load or were too challenged. The fewer actions between steps, the more accurate your guess of the issue will be. Also, always start these from app launch, never app initial load, in order to better see if the initial game load is bouncing players. Taking action to resolve these issues will lead to players having a much more enjoyable game experience and higher retention rates (plus better monetization if that’s your angle).

4.    Don’t keep A/B testing the same experiment

A/B testing allows you to try two different approaches to one problem and see what players react more positively to. Would it be best (for either gameplay or spending) for a player to fight the first boss with an extra health pack? You can separate these tracks of players and see who keeps playing the longest and/or spends more. A/B testing is also used for less intrinsic questions, like which button color for my store page gets the most clicks? However, be wary of iterating on the same test, A/B testing has a rapidly approaching diminishing return. Without jumping into statistical significance, when you start to only see improvements of around 5 percent, stop and move onto a new one.

5. Don’t obsess over industry benchmarks

Sometimes I get in trouble from suits during conference talks when I advise not to read benchmarks. There’s little reason to compare yourself to Supercell or other veterans unless you’re going for funding or have external motivations. Millions of dollars get pumped into re-attaining users for LTV, it’s simply not constructive to consider them when sizing up the health of your game. Not to mention many beginning folks who read them might not understand how easy it is to game statistics and inflate them for investors, let alone positive results bias, which then gets averaged for indies to compare themselves to. Maybe ask a friend with a similar game of genre or size, but just do your best. Set your own personal goals and aim for improving on them.

Bonus: Don’t stop believin’ (or iterating)

Creating a proper analytics setup is far from easy. Things will be in the wrong spot or simply broken. Tell your teammates or write a blog about what you found challenging or hard to decipher and what to fix in the future.

Whether you’re worried about how many tokens players are pumping into your machine, checking if your game economy is better than the great depression or just making sure players can get past your tutorial, analytics can help you create a better game.

Have some questions or comments? As always, feel free to post them here, reach out to me on twitter @tDJ or find me at theDylanJones.com. Best of luck!