Closing the Loop: Validating Design Decisions Using Player Data for a Platformer

Quantitative measurement and feedback has been a main pillar for all disciplines of science for a long time and validation of the design assumptions in every product design is an essential part of the whole design process in many areas. In game development, art and programming are two main aspects that can be measured and analyzed very quickly during development, however, the domain of game design is usually much harder to study quantitatively and a lot of times the validity of many design decisions remain a mystery for the developers and they can suffice to a few qualitative feedbacks such as critic scores or unorganized user feedback to validate the various design decisions.

This has changed dramatically in the recent years with the availability of different data gathering and analysis techniques available to developers. 

We recently released Shadow Blade on the AppStore, a fast paced precision action platformer with combat elements. Like any other game project, finding the proper design decisions to provide the necessary fun and engaging experience for the players was a big goal for us. In this article I would like to elaborate the steps we took to validate a few of our design decisions related to the design of the levels and the level progression as it relates to game difficulty and player retention. The current game has 31 levels in the main campaign.

You can watch this gameplay video in case you would like to get a feel of the gameplay of Shadow Blade: 

We used a data gathering and analysis tool called Game Analytics (gameanalytics.com) which can be used for free. After integrating its plugin in your game code, you can submit your game design specific events to its online servers and use its online tools to view various reports or download the raw data for further analysis.

The opportunities for data gathering from the game and doing analysis are infinite, for the existing study, we focused on analyzing the player death and level start events in levels in order to find answers to the following questions :

1 - How does the game difficulty pace look like inside every level?

The flow and pacing of Shadow Blade inside every level is mostly linked to the player death. Dying will spawn you from the previous checkpoint and give you another shot at passing the platforming challenges or the enemies fighting against you. Your goal is to reach the end point as fast as you can while finding the hidden level secrets and collecting all the orbs scattered in the level. We needed a way to find out where exactly in the level are players dying and we used heat maps for this purpose. The heatmaps showed the frequency of player deaths laid out in the level geometry so we could easily find the toughest locations of each level. The following pictures demonstrate the player death data overlaid on top of the level geometry inside the Unity3D editor.

Part of a level in Shadow Blade

Same part of level with the Heatmap data 

Investigating the frequency of player deaths in the level revealed points with values much higher than other level locations. We expect certain points with higher death rates in the levels that correspond to higher challenges in the level to be normal but a point with a huge delta in death frequency is obviously problematic.

Design Modification: The challenge for locations with very high player death frequency or very low frequency can be balanced in order to provide a smoother difficulty and challenge pacing for the level.

2 - How does the difficulty pace look like in between different levels?

The flow inside every level is important, but the flow between the levels and the level progression in a platforming game are at least as important as what happens inside every level. We had an initial assumption for each level difficulty but our assumption needed to be validated by actual player data gathered from play session. 

Two main parameters useful for finding an answer to the above question were "total level start" and "total player deaths" for every level. We used the following formula:

Level Difficulty = Total Player Deaths in Level / Total Level Start Events

The above value is of course not the only factor defining level difficulty but it can act as a relative value to compare different levels with and study the flow between the levels.

Computing the above parameter for level difficulty resulted in the values below:

Difficulty Factor for Game Levels

The game levels are divided into three chapters and the ideal level difficulty should have started from a low value and rather linearly inclined towards the last level in the chapter. The next level (from the next chapter) can have the same pattern but with a little less difficulty from the last level in the previous chapter since some change in pace is nice to have and also there is a learning curve for new the mechanics and features in the new chapter. 

However, the above chart was clearly not ideal for level pacing and we needed to tweak the individual levels in order to balance and provide a better flow between game levels.

Design Modification: The challenges in some levels needed to be balanced in order for the whole level progression chart to follow a standard increase in challenge (A saw shape).

3 - Is there a level that causes the players to leave the game and not return?

Player retention is a very important goal especially for mobile games. Our goal was to see if there are any specific levels that cause the players to leave the game. We had to analyze the game data from play sessions and find out the unique number of players for every level and see what the player drop rate is in between levels, in other words, which levels cause the gamers to quit and not proceed to the next level. 

The following charts show the number of unique players that played every level and the user drop rate for every level.

Unique Players Per Level

Relative Player Drop Rate for Every Level

The above charts show that there are clearly some levels with high drop rates. These levels are causing players to leave the game much more than others. Investigating level 14 which appears to be one such level from the charts above and comparing it to the initial level difficulty chart reveals that this level has a sudden high difficulty ratio and is an obstacle towards proper player progression.

Design Modification: The levels that had a high drop rate needed to be tweaked and made a little bit easier to enable the players to bear the time needed to enhance their skills and pass the challenge.

Summary

Overall we feel that doing quantitative analysis on actual player data brings infinite opportunities for refining the game design decisions and our small study so far has barely touched the tip of the iceberg. Embracing the opportunity provided by new technology to measure actual game play data is helping the discipline of game design to get more scientific. We are looking forward to continue such analysis in the near future and hope that sharing such information with the development community can benefit all of us in creating fun and engaging experiences for the players. It will be quite educational to see such data from other games and be able to compare the different charts between games for further analysis and knowledge gathering.

Aknowledgement

This study was enabled with the help of my great colleagues at Dead Mage, specifically Hojjat Jafary and Aryan Bina. 

Inspirational Reference: The "Lean Startup" book by Eric Ries has been quite inspirational for us to think deeply about validating design decisions.The book is about the processes in startup companies in general but maps very well to game development.