High Performance Networking with a ‘Web’ of Peer-Peer Connections

Introduction
Godkin is an in-development multiplayer co-op combat RPG, with a maximum of 4-5 players per session, and with high numbers of moving entities (4-5 players each with several AI-controlled followers; several villagers; and potentially 100s of monsters and 10s of defensive towers, missiles and spells active at a time). In order to handle the high amount of network traffic, I have developed an infrastructure involving, at its core, a direct peer-peer connection between each pair of players in a game session.

Peer-Peer Direct Connections
The main motivations behind a peer-peer approach are (i) low latency, and (ii) low server costs. Using peer-peer means that a hosted server is not needed apart from initial handshaking and NAT punchthrough. With 5-10 movement packets per second, per entity, per connected client, the amount of network traffic (and potential server CPU load) can grow alarmingly.

In my 'web of direct connections' approach, the player who starts up a game session listens for incoming connections. The second player to join connects to the 1st player and then itself listens for incoming connections. This procedure continues: each player who joins connects to each existing player's connections, and then opens its own connection for use by subsequent players. This model means that each player can send and receive data directly with each other player (a traditional client-server setup would of course mean data between players would have to be relayed via the server, i.e. 2 hops rather than 1). Latency is minimised.

In Godkin, we're using WebRTC data channels for both unreliable data (i.e. entity movement updates) and reliable data (i.e. game/entity states and event data).

Client-Server Fallback
The trickiest thing about peer-peer networking on the public internet is the fact that each client will be sitting behind a router which will probably not allow unsolicited incoming traffic, and in any case without manual configuration will not know where to route that traffic to, in its private network. The solution is NAT punchthrough which involves both clients connecting to a publicly-addressable server, and then (via data from the server) negotiating connections directly to each other through the same port that the other client just opened to the server. It's a somewhat messy process and a small fraction of routers pretty much refuse to do it. Therefore, a fallback is needed whereby clients who have failed to achieve peer-peer connection will relay data to each other via a traditional server. In Godkin, we use UDP (for unreliable data) and Websockets (for reliable data) as a fallback. The exception is the game's public hub where there can be larger numbers of clients, with players joining/leaving at a high rate: here, we use server-relaying all the time.

Scoping
The idea behind scoping is that certain data is less relevant to a player if it related to a game entity that is far away from their camera. The main data that can be culled through scoping is position updates (which are also overwhelmingly the most frequently sent types of data). Since Godkin is a 2D topdown/isometric game, it's easy to decide whether a particular world position is visible to other players or not, assuming you know where their camera is located. If a packet is defined as 'scopeable', the sending client will typically only send it to other clients for whom it is in-scope. Every 25th packet is sent to everyone, regardless of scope. The exception is data related to Player characters, which is sent without scoping, since everyone needs to accurately know where everyone else's camera is.

Distributed Control
One further system in Godkin which I have implemented for efficiency reasons is distributed control, by which I pass control of AI entities between clients based on whoever is closest to them. This means that both CPU and network loads are balanced between clients, rather than the game having a heavy reliance on the power and connectivity of the 1st player. In fact, distributed control is also a requirement in order to support scoping: we need to make sure that each client is controlling interactions between entities that are 'in scope' for it, i.e. it knows their positions accurately. Since Godkin is a co-op game, we're not overly concerned about security (although, I have implemented some anti-cheating measures).

Unity3D Implementation
Godkin is being developed using the Unity3D engine, and I was initially developing the networking infrastructure using the excellent Bolt plug-in. Last year, however, the company Photon bought Bolt, and implemented a punitive per-seat cost which is the same as the per-seat cost of their client-server solution, despite the fact that server overhead in a peer-peer situation is a tiny fraction of that in a client-server situation. Unhappy with this, I explored other possibilities and eventually picked a really nice, bare-bones WebRTC plug-in. One of the benefits of moving to a simpler, low-level solution was also that I was able to fairly easily construct the 'web of connections' approach as described above, rather than treating the 1st player as a server (which is what Bolt does).