Telemetry-Supported Game Design

Game telemetry is being used both during development and post release. One of the most exciting applications of this work is the use of game telemetry to support the game design process. Game telemetry analysis can help a designer answer the following questions:

Using game telemetry provides an alternative to conventional playtesting. Rather than bringing in individual participants for playtesting, collecting telemetry enables your entire user base to become playtesters. The advantage of this approach is that data can be collected at an enormous scale, from players in their natural environment playing with friends, across long periods of time. The main disadvantage of this approach is that telemetry provides only quantitative data about players. It enables a designer to identify what players are doing in the game, but not understand why players exhibit specific behavior. 

There is a general workflow for using telemetry to inform game design. It is an iterative process consisting of the following steps:

In order to utilize telemetry to inform design, it is necessary to identify which game data needs to be recorded. One common metric recorded is session information, such as the starting and ending time of a gameplay session. This metric enables a designer to track retention rates for players, which can identify how many players continue to play the game after a day, week, and month. In order to answer all of the questions listed above, it is necessary to record game specific data as well. These metrics can measure usage of different game modes and features, as well as track which content is presented to players. It can be useful to track both aggregate and individual usage of features, in order to further refine queries about the data. Another source of data is events that occur in the game, such as play-by-play summaries in a football game.

A variety of different approaches can be used to analyze data collected from players. Game metrics such as feature usage enable a designer to determine which aspects of the game are being explored by players, and retention metrics enable a designer to find out how long players are engaged with the game. These data points can be used to provide recommendations for modifying the design. However, identifying correlations between the design and results found in the data is a non-trivial task, because there is no qualitative feedback from players. One approach to overcome this limitation is to apply predictive modeling techniques, such as machine learning, in order to identify patterns and correlations in the data.

Telemetry-supported game design requires an iterative design process, in which new versions of the game are deployed to the user base. For web-based games this process is straightforward. For console games, this process becomes difficult, but franchise titles, DLC, patches, and sequels provide opportunities for updating a design based on results found from telemetry.  

While working as a technical analyst at Electronic Arts I used telemetry to track player behavior in Madden NFL 11. The work resulted in the following recommendations:

The main outcome of this analysis is guidelines for playcalling conventions and the matchmaking systems based on feedback from telemetry. A description of the predictive modeling process is available in this paper and the application to other games is discussed in this paper.