Thought it might seem obvious, most game designers tend to ignore A/B testing. While it is a must if you are launching free-2-play game, since it allows to apply new changes almost painlessly. That's basically the following blog post is about.
For those who are unaware of what A/B testing is, I am putting this short explanation here. Different components of your game can be implemented in different ways. And we, as game designers, are constantly hesitating which way would be the most fitting and effective for a specific setting, audience, or an entire game. Or we are planning to implement new features and don’t know how it will affect our game. Therefore, we split the player to form different independent groups, so we could test different varieties of new component and see how it affects the game.
Conducting this kind of experiment on real players, we determined what solution works better based purely on KPI metrics.
First, you need to determine the list of areas you would like to test. If it is a new project with a lot of new theories to prove, you’ll probably need an expanded split system that would allow you to collect qualitative and representative data.
Areas to A/B-test:
Interfaces
Game Mechanics
Game Economy (balance)
Content Progression
New Feature of Content Effect
Each of these options is measured by different metrics and the metrics is what is used to determine the effectiveness of every change. Interfaces and Mechanics are measured by the engagement KPI: target actions (their sequence), session duration and quantity, number of target actions per DAU etc. Balance and Economy are measured by financial indicators, like deficit and surplus of resources in your game. Content Progression and Game Goals are measured by the retention-related metrics which include long-term retention, number of sessions and their durations, and by other custom metrics.
In other words, the analytical systems you use in your project should be able to analyze different splits based on different metrics and KPI sets. It is a common mistake to analyze all splits using the same dashboards with the basics metrics. Nevertheless, you should not replace those metrics - it is possible that a certain feature would improve engagement metrics but kill monetization at the same time.
To properly work with A/B-testing tool, you’ll need clear and comfortable to work with interface, and this is the task for your server developer.
When users install your game, they are divided into different cohorts, which are further used to conduct a cohort analysis. You can set any custom division setting, like 50/50, 33% of users per a single cohort etc. Also you can set the system to send, for example, 10% of new installs to a specific cohort, if you don’t want to risk critical mass of players by giving them the untested feature.
You might also want to set a targeting. It is possible to set how the split is assigned to a certain user. For example, a test might only affect iPad users, or female from the USA - then you’ll be able to split only specific users and don’t risk the rest.
During an install or the first session, new user is assigned with certain settings, configurations, and other parameters. From now on, the user lives with these parameters and sends correspondent data to your analytical server.
Since you want to have different split options, you will need to create tables of settings for each splitting area and a general management table.
Interface Split Table:
Mechanics Split Table:
Configurations Table:
General Split Table:
The tables that combines all previous tables in one:
It is important to determine the number of simultaneous tests based on your audience and capability - do not run more tests than you can analyze.
Make sure that your trackers are placed and the data is collected properly.
Ask your analysts (if you have one) to test splits and their settings on the internal audience before enabling them globally (on the production server).
Conduct A/A-test - the one that divides the audience in half - and look at the results. If the results are the same, it means your system works properly. If the results are different you should ask yourself, what went wrong.
Read more about:
BlogsYou May Also Like
