Adding Telemetry to Infinite Mario

I originally posted this on my personal blog in September 2010, but it is no longer available since Posterous shut down.

BioWare announced findings for Mass Effect 2 from a massive telemetry project in which millions of game sessions were analyzed. Some of the interesting findings include popularities of the different classes in the game, such as the fact that the soldier class was played more than all others combined. One of the possible applications of collecting large amounts of data is to incorporate results from real-world player data into future games. This post provides a tutorial for adding telemetry to your own games. While the presented approach is a Java application, it should be applicable to any language capable of an http post.

Since the announcement of the IEEE Mario level generation competition, I've been exploring procedural content generation for Mario levels. The goal of the competition is to maximize player enjoyment by adapting to specific player styles. The competition uses a heavily modified version of the Infinite Mario engine developed by Notch. My variation of the game with dynamic difficulty adjustment is available to play here (arrows to move, 's' to jump, 'a' to run). After releasing this game mod I asked:

To answer these questions, I've added telemetry hooks into the game. This process can be simplistic for games capable of posting to http and as long as you have a server capable of running PHP scripts. On the server end, you can use a PHP script to append telemetry data to a file, as shown here:

 < ?php
    $message = $_GET['message'];
    if ($message != '') {
        $dataFile = fopen("telemetry.log", "a");
        fwrite($dataFile, "$message\n");
        fflush($dataFile);
        fclose($dataFile);
    }
?>

This code checks for a message parameter, and then appends it to a file. In a real-world deployment, additional security measures are needed, but this tutorial is focusing on simplicity. Next, we need a way of posting telemetry data from the game engine. In Java, we can use the following code to post to our PHP script:

URL url = new URL("telemetry.php?message=" + message);
URLConnection con = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
while (in.readLine() != null) {}
in.close();

This code passes the content in the message variable to the PHP script using a HTTP post. The result is a method for posting game data to our server for analysis. I used this minimal approach to collect gameplay data from over 100k gameplay sessions. The figure below shows a histogram of frequencies of difficulty levels generated by my level generator. Each player starts out at level 50, which I have excluded from this figure. Then as a player progresses through the game, the difficulty adapts to the player's skill. The results of my initial data mining experiment suggest that the initial difficulty level is too high and should be set to a value in the 35-40 range.

Now that we see that the game starts out too difficult I investigated the second question, why are players dying so frequently? Since we now have telemetry data, we can determine the cause:

It appears that most players are not completing levels due to deaths caused by enemies. If my design intent was to create levels that are difficult to traverse due to gap challenges, then I have failed. At this point in the process, I can take results from my data analysis and apply them to future game design experiences.

I hope this post helps out any game designers interested in studying their users' behaviors. This has only been an intro, and there are many more issues to address, such as scaling up and extracting more complex patterns of player behavior. Further reading on this topic includes my workshop paper on player modeling and the recent Game Analytics book.