Rapid Iteration of Persistent Multiplayer Games

(Reposted from my game development blog)

This is a collection of thoughts on efficiently iterating persistent multiplayer games, based on my experience over the last couple years developing Antihero. The hilarious irony here is that my original development estimate for Antihero was 18 months; it’s now been two years and the game is still not done, so what do I know about making a multiplayer game quickly anyway. (At least I’m not dispensing advice on accurate scheduling of indie games.)

For the purposes of the post, I’m using “persistent multiplayer” to refer to networked multiplayer games that involve a client and a server, and that span multiple play sessions and therefore require persistent state management: asynchronous multiplayer games, Facebook-style “checkin” games, MMOs, etc. (Antihero is an async multiplayer game, which is arguably the simplest, architecture-wise, of these three.)

In game development, there’s always friction between having an idea and implementing that idea. Persistent multiplayer games compound the problem with more sources of friction in the form of an additional application (the server), and two additional data layers (the network and the database). Rapid iteration involves removing friction from the process; for Antihero’s early development, that meant removing the server and network from the equation as much as possible.

The primary forms of friction I’m concerned with here are serialization, persistence, and the authoritative server.(1)

Serialization

Game state in a non-networked game needs to be represented in one or two locations: in memory and - if the game supports saving - on disk. Networked games usually also involve a server and a database, potentially with their own data formats.

When you’re iterating a game, your data is constantly in flux: in a single-player game, if you want to add a mana property to your Hero, you just add a field on your Hero class, and you’re done. In a multiplayer game, you need to modify your client’s Hero class, modify your server’s Hero class, perhaps modify a Hero table in your database, and make sure that changes to Hero.mana are propagated everywhere.

Choosing a wire format - the format that your client and server use to send data to each other - is one of the first architectural decisions you make when developing a multiplayer game, and the choice has big ramifications on your ability to quickly iterate the game’s data structure. When I’m prototyping a multiplayer game, what I really care about is:

Nothing else - serialization speed, data size, security, etc - matters: iteration efficiency trumps all other concerns. (I’ve worked on multiplayer games where there was an insistence on doing things “properly”, security-wise, up front. This bogs down the pace of iteration now for an insignificant potential payoff later.)

For Antihero, I chose Google’s Protobuf as my wire format. It satisfies my three prerequisites, and has Python and ActionScript implementations. (And, happily, it’s also fast and efficient.) Protobuf relies on code generation to work - you write your data structures in a simple description language and then a parser spits out class files in the language(s) you’re using. (Some devs are allergic to code generation, because they believe it increases maintenance costs and code complexity. They are wrong.)

If you’re not down with Protobuf, there are lots of reasonable-looking alternatives out there (Thrift, Avro, etc). All the cool kids write their own serialization format! But you, dear indie, almost certainly should not; instead, just take somebody else’s and get on with your game.

Persistence

Alongside wire format, you’ll also probably choose a persistence layer early on. “Persistence layer” usually means “relational database of some sort.”

But a database is another source of friction. It imposes unnatural additional syntax on your server (in the sense that manipulating data in SQL is very different from manipulating a runtime object); it slows down refactoring, because database schema changes are a pain and similarly unnatural; and it’s another frigging service to set up and futz with.

I avoid the database for as long as I can, and Protobuf actually helps here as well, because it’s trivial to save protobuf objects to disk. In the early days of Antihero’s development, I created protobuf_table.py, a simple Python class that the server used to manage a collection of protobuf objects (one for each match in progress), save them to disk when they changed, and load them all back up on startup. With protobuf_table, Antihero had many happy months of development where the client, server, and pseudo-“database” were all using the same data format. (And even now that the game has a proper database for account management and whatnot, its “active_games” table primarily consists of just a “data” column that stores the games’ protobuf data directly as a binary blob.)

Authority

I don’t worry about doing the right thing, with respect to security, during prototyping (and, possibly, beyond). The “right thing” is to assume that the client is in the hands of an enemy that’s tampering with it, and therefore to have the server validate everything. But this is a massively premature optimization (especially when the client and server are written in different languages, necessitating that game logic be written twice).

One game I worked on, in a past job, was still far from being fun - let alone shipped - when it was decided that server validation was necessary to make the game more hack-resistant. This took several programmer-weeks, slowing the game’s development without, of course, making it any more fun. And when the game launched, it still wasn’t fun; it never found traction and was written off as a failure and shut down shortly afterwards. Its low quality was what truly made it hack-resistant: it never had enough players to be a tempting target to exploit in the first place. All that time spent making the game less exploitable was simply wasted.

The Antihero server, whenever possible, acts as a dumb message-passing mechanism. Data comes in from a client, gets stored, and then naively shipped back out to other interested clients; aside from making sure that an out-of-date client isn’t sending broken data, no validation is performed. And when the game ships, the server probably won’t do a whole lot more; since the game will probably fail anyway, I won’t spend time on server validation unless and until I get lucky and the game finds a player base.

A Client-Centric Philosophy

My approach to multiplayer development can be boiled down to “spend as much time as possible working on the client.” The client is the game, as far as the player is concerned - everything else is an implementation detail. With Antihero, I’ll often go for weeks without touching the server except to recompile its protobuf files.

A solid server is only a necessity for a finished popular game; prototypes are by definition not finished, and the brutally hit-driven nature of the marketplace ensures that most games will never be popular. Having to shore up a janky server for a launched game that’s found traction is the very definition of “a nice problem to have.” Server procrastination is a virtue - much better to spend your precious development time increasing a game’s chances at popularity by making it interesting.

(If you’re interested in running your own thieves’ guild, you can read more about Antihero, my in-development game, or follow me on Twitter.)