Measuring the benefit of deep learning for the games industry

There has been much exposure in the popular press of deep learning systems defeating humans in games such as StarCraft II and Dota 2. However, the goal with these competitions was to prove the capabilities of neural networks for tasks that were considered too complex for machines only a few years ago. The question remains: what use is this technology to the games industry and how can the potential benefits be assessed?

We should be both excited and critical of the opportunities presented to us by deep learning, because the cost and processing burden of such technologies can be greater than the value gained from their successful implementation. We can check the promise of proposed advantages by measuring and comparing them with solutions from other domains. The tools to enable this are lacking, so I developed my own and released them for free on github: https://github.com/shanemcandrewai/Speech-to-Facial-Landmarks

Before addressing the potential added value of these tools, we will examine some industry trends and associated challenges.

As games move increasingly to multiplayer and cloud platforms, the network becomes a bottleneck for performance. The network latency or ‘ping’ is affected by the physical separation between the player and the game server. eSports competitors based in remote locations know well this problem where even a 50 ms difference can impede serious gamers. Bandwidth is generally increasing over the years but this will never completely remove the problem of network latency.

Riot Games, League of Legends : Nexus Blitz mode

Consider a simple example of what happens when a player fires a projectile at a target. This action will be communicated from the client device (PC, console, mobile, etc) to the game server. The server will, in turn, transmit this information to other client devices. However there may be a very short time before the projectile strikes the target. How can the client display this movement in a fluent manner when the network lag is significant in comparison with the speed of the projectile?

Valve, Counter-Strike: Global Offensive

There is one possible solution for this that does not use deep learning, and it is called ‘dead reckoning’. This is a very simple method which has been used by sailors for centuries to calculate their location in the absence of more exact methods. If the client knows the time when the projectile was fired, it can simply multiply the speed with the time since firing to estimate the current location. But what happens if another player moves in front of the projectile during the split second between updates from the server? The client is then forced to update its estimation which appears as a glitch or erratic motion from the point of view of the player.

Alternatively, a neural network could be trained with many recorded gameplay sessions. It may learn, for example, that when a projectile is fired at a blind corner where another player is hidden, there is a certain probability the other player will move in front of it. This could reduce the chance of a client making a faulty prediction and hence lower the number of glitches perceived by the players.

There are two problems with this deep learning ‘solution’. Such deep learning techniques require time to train these networks, and this can take weeks or even months. However, this need not be a big problem because it can be done offline to avoid impacting the game’s performance. 

What is less well known is that the inference or prediction function of the network also takes a significant amount of time in comparison with the speed of gameplay. In my experience of using deep neural networks to extract facial landmarks from video, the rate of prediction can be as low at 2 frames per second. When deep learning is employed to make improvements, we must be aware of possible trade-offs. For the given example, since both video games and neural networks require heavy calculations, processes compete for the same resources which could introduce another bottleneck.

We must examine if the use of such deep learning solutions, that solve one problem, are not then creating another. It is important to measure the different approaches before deciding which to implement. 

During the course of my Masters in Game Technology at the Breda University of Applied Sciences (BUas), I developed tools to achieve this. These tools have been uploaded to github where they are available free of charge. The original use case was replicating a model which predicts facial landmarks from audio speech, but they could be used for comparing the accuracy of any solution (deep learning or otherwise)

In conclusion, deep learning is just one of many tools available to solve game developer’s and player’s problems. For applications such as image recognition, they have proven to be more accurate than humans, but only for very narrowly defined problems. The key challenge is to determine how these success stories can be leveraged in the game industry, especially how to measure them, and where they would be of the greatest benefit.

Thanks to Prof. Mata Haggis-Burridge for his assistance with this article