The current generation of games is pushing the boundaries of what’s possible with today’s technology, both in terms of art and gameplay. Game engines such as Unity and Unreal do a fantastic job of rendering game worlds for the player, yet they are also doing all the work under the hood to make the game itself happen — running physics, AI, pathfinding, and game logic.

For a single player game, the engine will be doing all this work on the player’s device. The concept of the game loop hasn’t changed much since its invention decades ago.

Multiplayer games aren’t very different. The game loop is split between client and server, with the client dealing with inputs and rendering, and the server performing the world simulation work.

But there is a limit to what can be achieved building games this way. You can only fit so much physics and AI on a single server, and this inevitably leads to compromises in terms of the size of the world, the number of players, or the complexity of the physics or the AI.

This, in turn, imposes limits on the creative vision. When game logic, physics and AI are already competing for a share of the available resources, there’s not much room left to experiment with truly innovative ideas.This is a problem we’ve been doing a lot of thinking about.

The Entity-Component-System pattern

Most games follow the Entity-Component-System (ECS) pattern to some degree. Every “thing” in the game world is represented by entities. These can be characters, trees, parts of cars or entire spaceships, depending on your game.

Entities are defined by their components, which group the properties that describe their state. Components can be shared between different entities, which leads to a modular design. For example, both characters and trees can share a Physical component that defines their position and rotation, whereas an Inventory component will be used only by characters.

These entities and components are brought to life by systems, typically threads that deal with the different aspects of the game. The physics system updates the Physical component, the AI system will observe the world and make decisions, the game logic system will make the game world behave in the way its designers intended, and so on.

The limitation of this approach comes from the fact that all these systems run on a single server with limited capacity; each system is competing for cycles with every other system.

But what if each system could be a distributed system?

This is the same epiphany Google had in the early 2000s: it is cheaper, simpler and infinitely more scalable and reliable to develop applications that run in clusters of commodity hardware instead of running on monolithic mainframes.

Of course, building distributed systems is hard. Insanely hard, even; you need to debug race conditions, write netcode, think about concurrency and consistency, try to understand Paxos, give up and implement Raft, build fault tolerance into your design, have a strategy to deal with network latency and packet loss, put metrics and monitoring systems in place, and a myriad other issues which are not directly related to the fun and joy of making games.

It seems like a hopeless situation. The solution exists in theory, but in practise it seems impossibly complex for game developers, who are already stretched to their limits with everything involved in building a game!

A new approach: introducing workers

Fortunately, it is possible to formulate this problem in an elegant way, which preserves the simplicity of the ECS pattern but makes it possible for a platform to abstract away all of the unpleasantness of building distributed systems.

The key innovation is introducing the concept of workers. We take each system in ECS and replace it with a distributed system; each distributed system is built out of many workers, each of them being a program that can simulate some components of a subset of the entities in the world. The most obvious general- purpose examples of workers are game engines,  logic workers or even game clients (desktop, mobile or even VR headset).

We call this the Entity-Component-Worker architecture:

The most immediate examples of what workers can be are general-purpose game logic workers, and a physics engines. Perhaps surprisingly, a game client is also a worker that runs on the player’s machine, but that isn’t treated in any special way otherwise.

It’s also possible to use workers for specialised simulation that doesn’t fit the game logic workers. These could include anything from accurate weather simulation to flocking behaviours.

Using this new architecture, the size of your game world is no longer limited by, for example, how much the physics engine on a single server can handle, because you run hundreds of instances of the physics engine to simulate different parts of the world.

All of this is invisible to the developer. In the ECW architecture, a developer would implement their game in a way very similar to writing a single-player game with the ECS pattern, and by specifying the kind of worker responsible for updating each component (for example, “Physical should be simulated by a physics worker”), the game world can scale up to millions of entities across thousands of workers running on hundreds of servers.

Conclusion

We envision the next generation of games will be built on a set of workers that cooperate to simulate a game world far bigger and far more complex than it is possible with a traditional architecture or in a single server. 

This is not purely theoretical; games such as Worlds Adrift by UK developer Bossa Studios are being developed with Unity and SpatialOS, our implementation of the Entity-Component-Worker architecture. It has also been applied to large-scale simulation outside of gaming, such as this simulation of the entire backbone of the internet created for cyber-securty research.