Creating scalable backends for games using open source Orleans framework

Project Orleans is an open source framework for development of distributed applications. Orleans can be used to develop anything from complete game servers to scalable backend services like leaderboards/analytics/ account/….2 Halo titles used it to implement all of their backend services so you can do so as well. The next Halo game is doing that as well.

What is the difference with usual architectures?

Orleans uses the actor based programming model which got popularized in industry by Erlang. By popular I mean many people finally realized the power of actors for implementing logic for distributed applications and tried to use the available tools to do the job. Erlang was the only battle tested available solution at the time. I’m not going to teach history here but Erlang did choose to use the actor model and message passing (instead of shared memory which everyone else is using) more than 30 years before Orleans, Akka and other frameworks. So let’s define the actor model and see how it is different and then see why Orleans is a better fit for many applications compared to other frameworks and languages.

Actor model

In this programming model each entity in a system is an actor and actors can only communicate by messages. Each actor has a mailbox which receives messages from other actors and then reads them and execute the corresponding action. Actors are kinda like objects so in your typical game each character becomes an actor with different actions like damage, upgrade and move and other actors might request it to move or upgrade to a higher level.

So you might ask what is the difference? The difference is that each actor is completely isolated from the rest of the world. It only can read messages sent to its mailbox and write messages and send them to mailboxes of other actors. No actor can modify or even check the state of any other actor and global state in the form we know it is none existant. Of course you can have a global actor which holds global state but no other actor has no access to the data other than sending requests for data read/writes by means of messages.

This model is perfect for parallelism since you can run multiple actors at the same time without any locks or any other synchronization methods and be sure that they’ll never have many of the problems of our normal multi-threaded programs because they don't share memory with each other. Also actors are very lightweight compared to threads since changing between them doesn’t require any operating system work and no context switches and they don’t require many things which threads require to run beside each other. Programs usually share state and memory but actor based programs only can share by message passing so two actors can run on different machines and then contact each other by messages and share stuff easily. Actually from point of view of an actor it doesn’t matter where the other actor is most of the times. This model allows us to write distributed parallel programs much easier. Also if an actor fails or throws an unhandled exception all others actors happily continue their lives (unlike normal multi threaded programs) so it is fault tolerant as well.

Orleans compared to Erlang and other classic actor frameworks

Excuse me for the terminology because new phenomenon doesn’t have accepted terminologies. When you code in Erlang or Akka or other actor frameworks you are responsible for creating and destroying actors. In some systems like Erlang’s VM the location of actors is transparent to your program (i.e. it doesn’t matter from user code’s point of view where the actor is) but in some systems distributing actors over multiple machines is harder.

In Orleans actors virtually always exist, I mean you don’t create and destroy them , instead each actor has an ID and when you request its reference and send it a message the runtime creates it for you and after some time which the actor is not used, the runtime will destroy it until later on. This is like the concept of virtual memory which you have virtual memory available to you but some of it is sometimes on RAM and sometimes on Hard Disk. The hard disk part is not usable until the memory manager brings it to the RAM. Actors in Orleans are like that as well. It makes it much more productive and easier for developers to write actor based programs since you don’t need to handle many failure cases and the runtime will do it for you.

As an additional bonus the runtime gets the opportunity of load balancing the system by moving actors from busy systems to unloaded ones. Orleans can do this because an actor’s physical address is stored in a table and is not based on consistent hashing and similar algorithms which doesn’t allow the actor to move its physical location.

If you manage your actors life time yourself you can hand tune the system for specific scenarios if you are a distributed systems expert but the productivity gains and in most cases performance gains of Orleans are really big. Don't take my word for it, try Erlang or even Akka.net at the same time and compare them with Orleans. If you are an Erlang developer, Here you don't need most of the services which OTP provides, no superwisers, no trees of processes ...

Writing a simple Orleans based service

Orleans has a good amount of documentation and there is a good amount of material online about actors and actor based designs and everything else. Here in this first post my goal is to create a very simple program (as simple as hello world) by Orleans and connect by unity to it. Later on I might post more complex stuff if there is interest.

I’ve implemented a simple guess the number game with two requests. Users can create a game and can guess after creating. Games are single user for ease of development. In Orleans actors are called Grains, Each Grain is defined first by an interface which defines what messages the grain can receive and an implementation which implements the interface and everything else required. All interfaces have to derive from IGrainWithXKey where X is GUID, long or string or …. And all messages should be asynchronous and return a Task.

Below is the code for our Grain’s interface and implementation. Our Grain’s type is GuessTheNumber and we can create multiple instances of it by requesting grains of this type but with different GrainIDs, IDS are System.GUID values here but you can use strings and long values as well.

The implementation code

The interface

So users send a create message by minimum and maximum numbers and then the grain chooses a random number and stores it in its private field. No grain has access to private/public fields of other grains, they only can call each other’s methods which are defined in the interface. Then Orleans will send a message (instead of an actual method call) to the other actor with the arguments serialized and returns the results back if any.

As you can see Orleans uses task parallel library like await async pattern. It is mandatory that all messages only call none blocking code. Orleans gives each actor a turn to execute and it must execute fast and give its turn to the other actors. You can read more about these stuff on Orleans’s documentation. It optimizes serialization and many other stuff so don’t worry about that.

Now it is not a good idea to let Our games to contact Orleans directly because an Orleans client can call all actors. At least at the moment there is no way of hiding some actors and even if there was, from the point of view of the security guys it wasn’t a good idea to put Orleans online accessible by public. Instead we write a simple ASP.NET based web API project which acts as our client for Orleans (It’s a part of the server from the game’s point of view). We have two API points, one for creating the game and one for playing which they call Orleans code as client and return the result back as JSON. Then unity can call these API endpoints to play.

Another option is to put a high performance TCP socket server or SignalR in front of Orleans which I’m not going to discuss at the moment. So there are machines (or a single machine) running Orleans servers called silos and other programs can connect to grains and call their messages as clients. Usually this is the job of the part of the server which connects to our game clients which can be WebAPI or any other communication mechanism suitable for the task at hand. As a side note, the Web API project is created from a template, in controllers folder there is a guess.cs containing two API commands which call Orleans grains and return the result back. For more information go to asp.net/webapi

Unity Client

The client should simply call with WWW class the created API end points and parse the JSON values returned by the APIs. In fact any system capable of sending HTTP requests and parsing JSON can be used so Unity is not the only option. I left writing of the client as an exercise for the reader but feel free to send me a pull request on github if you wanted to include it in the project.

Error handling and other things

The code is not doing any error handling to be very very simple. We don’t handle anything from timeouts, wrong values and … Also many Orleans details are not discussed both theoretically and practically but I will do if there is interest. Did not want to make this post even longer.

For example if you try to make a new game with the ID of a currently in play game, the new game will just replace the old one and we don’t check any state to see if the game is already being played or not. In our implementation of the web frontend we use GUID.NewGUID() method which is very unlikely to produce none unique values but still the possibility exists. We could add an enum to the Guess grain to check the current state of the game. Orleans has a much more complete tic-tac-toe game sample available on github in their repo.

Do you need motivation?

Orleans has lots of great features, It has support for automatically sending events from Grains to any observer, it has support for timers, reminders and reactive extensions based streams and can easily store actor states in a database. You don't need a cache anymore since the application is no longer stateless. It will scale by adding clusters to your already running system so if your game gets busy then you’ll be able to add a machine to your Orleans cluster and then boom, your game is performing better. It doesn’t happen if you design your system in a bad way but it can happen and you can scale almost linearly and the actor model makes it as easy as possible.

Orleans is being ported to Core CLR so will run on Mac and Linux as well and instead of ASP.NET on IIS you can use OWIN based ASP.NET webAPI so you don’t need to be on windows.

Note: Microsoft Service Fabric SDK is another actor framework based on Orleans, actually It’s a framework for writing distributed systems as well. It has another model based on distributed collections and the Orleans like model. Be careful since it is different from Orleans in many aspects like actor placement and data serialization and … but these are combinable. So you can use otehr parts of service fabric on top of Orleans if you want to.

You can easily run Orleans on Azure or on your own servers and Grains can easily store their state in databases whenever you want and you are not bound to any database. Bioware also developed an Orleans inspired java based system called orbit which is available online and gamasutra announced it.

Many game related companies used Erlang, Scala and other actor based solutions for solving their problems so the actor model is a really useful tool in your arsenal and Orleans is one of them with great tradeoffs.

Running the code

The solution can be downloaded from here and contains 4 projects, Interfaces contains are grain interface, Implementation contains implementation of the interface for our grain, SiloHost is the program that runs our Orleans based code on the server machine and web front end is the webAPI which connects to our Orleans server (silo program) as a client.

Restore the NuGet packages and then build the visual studio solution and then execute the Orleans silo created by the silohost project. Wait for it to initialize. Then run the WebAPI project from visual studio (If it is not , set the project as startup project). Now you can connect unity games to these, just enter the machine address in your unity game implementation, (your IIS express in visual studio should show this to you) and try to play. The free community edition of visual studio is more than enough.

How to design actors/grains

It can have its own post or multiple posts but as a note keep in mind that it is desired to divide the program to as many small actors as possible in order to make it scalable. You should try to avoid central points and global places for everything as much as algorithmically possible.

So instead of having an actor for a user’s inventory, game state and account and friend list you’ll have an item actor per item and an inventory actor referencing all of the items and then a player actor referencing inventory and account which itself references friend list which contains references to other players. In this way if 10 of your items want to receive 10 upgrade messages they can do it in parallel but if one actor was responsible for doing it then they should execute one after another. Also this makes it possible to move some actors to other machines if required or at least move them out of memory. Orleans can run even a million actors easily so don’t think of them as usual threads and processes. You should try to make everything as parallel and none sequential as possible.

More resources

Other than the Orleans homepage linked above at the start of the post you can watch some great videos in the Microsoft Research page for the project.

http://research.microsoft.com/en-us/projects/orleans/

Acknowledgements

I should thanks the whole community in the gitter chat room of project Orleans which helped me to learn Orleans and specifically Sergey Bykov which reviewed this post. My workmates and friends at MuchDifferent which introduced me to Erlang and actor model before and the great posts and talks of Joe Armstrong which describe everything regarding actors and distributed systems beautifully.