5 top tips for A/B Testing in free-to-play games

All free-to-play (F2P) game developers and marketers know that regular updates to games are essential in order to provide the best on-going experience for players. And the decision making process, though creative, should be data-driven. The impact of any changes you make to a game need to be measured, so you’re likely to use A/B testing, but your method can dramatically impact how meaningful your results are.

Working in conjunction with other analytics tools, such as segmentation and targeting, A/B Testing is used to test different variables within distinct groups of players in your game to see the positive or negative impact. This might include adjusting the game difficulty, trying out different in-app purchase offers, or changing the color of a button.

Perfecting your strategy can make all the difference when it comes to improving the vital indicators of a good game; retention, engagement and monetization. Here are 5 top tips to make your A/B testing strategies as effective as possible.

1. Decide where you need to improve the game 

The clearer your test, the better, so having a defined goal or hypothesis is important. Focusing on what needs to improved within the game is a good place to start.

You should be measuring events in the game and monitoring your key performance metrics on Dashboards. Track the First Time User Experience to see if your on-boarding is successfully introducing players into the game. Use funnels to look at how players are progressing through the game and to identify the churn points.

A typical A/B test set up on the deltaDNA game analytics platform

2. Size matters…


It’s essential you hit enough volume with your data to get a robust result. It helps to schedule a test for when your players are most likely to be playing. Use data from your performance dashboards to determine when this is.

If you’re not sure if your sample size will present you with accurate results, read our practical guide on how to test if your data statistically is significant.

3. Don’t draw conclusions too early


Patience! It may take a while for a real result to show in your test if, for example, you are are looking to increase the first payment. After you’ve determined the sample size needed, plan in advance the length of time you’ll need to run your test for to gather enough data, but only run it for as long as necessary.

4. Improve the experience for different segments 


Not all players are the same. Testing in real-time, and checking the impact on player behaviors live, is a long way from the days of “do you like mission 1 or 2 best?” With today’s powerful deep data capabilities, rolling out the best performing game to the entire player base is foolish. What works for one group might not work for another.

Focus on understanding which variants you need to present to suit different player behaviors. To do this, it’s essential that you identify different segments within your game and map the journey of each separately. You can then test out variables that are tailored to improve the experience of each segment and personalize the gaming experience.

5. Don’t stick with just one test


As in life, what you put in you get out! One test won’t give you all of the answers that you need. Expect to run several tests and collate the results to get the best idea of performance and effectiveness. However, remember to keep a control group so that you can benchmark.

So you’ve got A/B testing down, and you’ve made changes to your game that are bringing in good results. What now? The easiest option would be to think of your work as done, but you really need to keep on perfecting the game if you want a shot at success in the F2P marketplace. Make all the hard work worth while – keep an eye on your KPIs and keep on testing.

This post first appeared on the deltaDNA blog.