Choosing the best networking middleware for your Unity based multiplayer or MMO game

It's my first post here but i've made games for years, from single player games to multiplayer ones and games using different techs, I've made multiple multiplayer game prototypes and have been a part of a big MMO and also before leaving the project i was making a MMO from scratch with some fellows, as a result of this i've done a lot of research in the filed and due to the lack of knowledge regarding multiplayer games and technologies became motivated to write this article. The article originally written months ago and is available in educational material section of www.nooparmy.com

When you want to choose a solution for performing a task in the computer world, there is not much difference in the method. There is no silver bullet or single technology that can do everything better than all others. The key is to research hard and choose or create your technology with lots of care to avoid later headaches. It’s an unchangeable fact that in current computing technology there is no black and white or good and bad ways to do stuff and anything has its own drawbacks. As a programmer or any other technical guy you might, one day or the next, hate this all but it's true and you have to either live with it or change it, and the first choice is easier. The key is to define your problem completely and correctly and then find your requirements. Then by examining different technologies and their advantages and disadvantages you can easily grab the suitable tool. The next time that you see people who are in holy wars to convince others that for example Linux is better than windows or C# and .NET platform is better than java, don't get into it. Any platform/technology has advantages which have associated costs and have disadvantages with reasons. No single tech could be prepared for all situations. The art is to strike a balance between them, focus on the goals and keep an eye on other solutions. If you think I am wrong, take a look at architectures of OSs and see for example why Windows developers choose a less secure approach and its advantages and disadvantages. Believe me, or not, it has its advantages.

So, there is no single networking middleware and game engine that suits all of your needs. You might argue that you can develop nearly all of what you want in Unity and Microsoft Office is simply the only great office suite software package in the world so there might be situations that one technology covers many things and covers them well enough that its not comparable to others by any means. You are right and sometimes some technologies are much more powerful than others and in most situations you’ll end up using them instead of some other technology. Also when you use something and gain experience with it, sometimes it’s better to use it for most of the projects because you are faster when working with it and you have knowledge regarding it. These are really great things when you are creating software. When you use it more you’ll learn more about it and will become a master of it after doing a couple of projects with it. So if you are making a 2D game and know there are tools which are more suitable for it than Unity but it’s five years that you’ve been using Unity and it’s fully possible to do it with Unity, it’s a good idea to use Unity to do the project instead of learning the new tech and investing in it. However, based on the amount of time and budget you might choose the other path. If you are reading this it means that you choose Unity as your game engine and you have your reasons. If you are not that sure after reading the above paragraph, answer some questions for yourself:

You might also want to ask some others. If you are sure about using Unity, which is a great game engine with fast development workflow for many kinds of projects and many many other advantages, then your next task is to choose your networking middleware.
To find the networking middleware for your game first you should know what type of game you want to develop and what your requirements are. I think there are 7 important questions that you should answer:

After you answer these questions and some sub questions, if you examine different technologies, it's not hard to choose your tool from this point.
Based on your answers the server might need different capabilities. Also, there are two other questions which relate to your own team and status and not the game so much:

Surely if possible, any company is safer using its own tech.

You should choose the middleware that makes your job easier and does much of the job. You should focus on your game and be able to start prototyping and iterating quickly. There is a rule that says your game will be better if you do more iterations and create more prototypes and do more playtests. I mean some middleware is just message relays and some handle databases as well. Some also handle transactions and monetization and some of them do not. There might be an engine for MMORPGs which only requires little logic and content and not any core components to be written by your team. You should also choose the middleware based on these factors.
I don't name any technologies to compare their features because they'll change over time and I don't know all of them and by writing their name this article would become vulnerable to time. At the time of writing (August 2012), the most important options which I know are: UnityPark suite, Photon, Roar Engine, electrotank, badumna and smartfox

 

Different game types in terms of networking and server side logic

Now I'll try to talk about games from the most simple turn based chess to the MMO (simple to complex). Each of them requires different features and one or another tech might be better suited to making them.

 

turn based regular games

If you are developing one of these chances are high that you can find ready servers which you just need to parse their messages on the client. Also, it's not that hard to code the server from scratch and those networking middleware that offer custom server side code can, by any means, handle what you are doing. It's preferable to choose those that handle users and databases and game saves for you too but, if you have no problems implementing them, it's not a problem to go with simpler solutions too.

 

more complex turn based games

If you are developing one of those regular social games which are totally turn based but need customization of the social mechanics or the strategy of the game then, again, there are good solutions exactly tailored for this need. You don't need Unity on the server to handle logic for these types of games most of the time. Some middleware provides panels to customize the logic and also provide the ability to script the logic with languages like lua and action script. Some more famous solutions also provide custom scripting using the .NET framework without having the overhead of Unity on the server. Again you should choose the middleware that makes your job as simple and fast as possible. Usage of complex engines on the server is possible for this group and the previous one but, based on the capabilities of your middleware and the scale that you require, it will have some overhead for your server processor and you'll need more powerful machines to handle the same amount of users. On the other hand, if time to market and flexibility is very important, using Unity on the server or using a specific platform makes your job much easier, and using some techniques the overhead is not that noticeable (50 players less in a server with 6000 players), then why not do it? After all, writing fast and great logic which really outperforms its competition is not an easy task and it's very possible that you start doing it from scratch in a really great platform like node.js or erlang but end up in something slower due to your developer skills and experience. Sometimes it's also a good idea to use multiple technologies and combine them to use for the features that they are best at.

 

normal multiplayer games (e.g. FPS, RPG, sports and ...)

In my definition, games that are complex and real time (i.e. with complex logic and collision detection and physics), like multiplayer sport, FPS and many co operation games, are in this group. The difference between these games and the previous group is that they are real time and collision detection and physics have a big role in their logic. They have worlds and objects in them and lots of dynamic stuff will happen. Worlds will change and objects should be created/destroyed in servers and clients. However, they usually will be played in small servers with 4 to 64 players and they cannot be considered MMOs.
These games are easier to develop if you have a full framework for handling all logics on the server. So, having Unity on the server is a big bonus. You cannot ever use exactly the same code for client and server for security reasons but it's always a great idea to develop both server and client scripts in one framework and language so having Unity on the server in these fairly complex games is important. You might be able to code all of your logic without Unity at first but then you might get into problems when you want to add features to your game. The middleware should not limit you; it should break the limitations. Think big and think out of the box at first to avoid later headaches. Chances are high that you'll need the collision detection and physics, AI and time management functions of the engine to implement the logic on the server easily so why you should try implementing them yourself when your team is smaller than what's needed.

 

massively multiplayer online games (MMOGs)

Usually those developers which want to create these types of games have lots of knowledge and money and manpower to research and can choose their systems easily and won't read our article but let's write about it. After all, there are newcomers who dream of making an MMO and, like us, some others are crazy enough to really do it. These days it's not as hard and expensive as it was even a year ago. These games surely need the game engine on the server. I mean if you don't use your game engine on the server so you should have a big team implementing collision detection, level management, AI, physics and lots of other libraries on the server. Most teams are not that big and we are not talking about Activision Blizzard so let's assume that we need the game engine. Many big MMOs use this approach. For example APB uses Unreal engine based servers.
These games require lots of other things too. They need load balancing to balance the load of processing and data flow on the server. They need flexible data models and database management. They usually have heavy loads on database and also they need to gracefully handle failure. All of a user's data should not go away even if the user's server goes out of service in middle of a game session. A fault tolerant, data redundant and scalable architecture of server software in a suitable hardware environment is required to operate a MMO.

 

middleware capabilities

Now let's examine all different capabilities that middleware can possibly have and then you can look at the problem from this side. See what features you need and which middleware provides them and choose yours. The features that seem more important to me are listed below and their descriptions follow however fully describing all of the different algorithms and their details is not possible in such a short article. It’s up to the reader to work from his/her knowledge on any of these; go and research more about them. Feel free to ask us your questions as well.

Reliable UDP and TCP

In fact, reliable UDP is essential for real-time games and games that handle multiple messages per second and HTTP and TCP are too slow for them. Also, the payload of UDP is small. The problem with normal UDP is that your message will not always be delivered and for many cases, like some of the important remote procedure calls (remote method invocations), you need to make sure data arrives. So, a software layer should provide a feature that sends UDP messages repeatedly until they are delivered. Most middleware provides this feature but be careful about it and check your middleware. If you are doing those async games or turn based ones, then it's not that important but for a FPS or sports games it's simply something that you'll miss if you don't have it. TCP also might be required to do http requests and other normal web stuff (if you don't have Unity on the server and other libraries are hard to use).

 

Custom data serialization

At first it might not seem that important to you but, to be able to manage the project, your middleware should provide you with a way to serialize all of your custom classes and structures and should support all of the fundamental types itself or provide the possibility of supporting them. Serialization might come in the form of writing functions, creating definition files in XML or another language but it should be there. Otherwise your code will become something that is impossible to maintain and changing data models will become even harder.

 

Specific features and ready to use scripts and utilities

Some middleware might provide specific features for the client or the server that are really great for your project. Imagine that you want to build a chess game followed by some other board games and card games and there is an engine with samples for these kind of games or think about middleware which removes some Unity's own limitations and has utility scripts for doing common tasks like in game chat and synchronization of physics based objects. These are useful for some projects, aren't they? If you want to do a FPS and one middleware has a good sample that you can build on top of and already has good scripts to synchronize character movement beautifully, then give it a go.

 

Load balancing

Simply balancing the load of network traffic and CPU and RAM usage between servers is one of the most important features of middleware for MMOs. For other game types it's not make or break however network load balancing is always required. At the time of writing none of the middleware that I know support moving Unity based processes from one machine to another but it's likely that it will be supported in near future. Some of them support moving GameObjects between server processes which is another approach to balance the load and sometimes it’s the best solution. Network based load balancing is even more important and is about dividing the traffic between servers by different techniques, from moving network objects from process to process to connecting new users to more relaxed machines. If you have lots of gameObjects and network replicas on one server then it should divide its load between itself and other servers or other cores on the same machine. Let's say you have 8 processes on a server and 3000 users and you should assign them to processes. Users come and go and in some designs some parts of the games are more CPU bound or network bound compared to other parts and these problems should be solved by someone. Sometimes you have a single universe and you want to divide it's regions between servers and you need good algorithms to do this. Some middleware might simply allow you to assign fixed regions to processes but more powerful solutions can do more to help by changing the sizes of regions at runtime based on the amount of load on them and number of GameObjects in them. One of your servers might become problematic and go out of service and others should be able to accept and handle the load that was on that machine before. Load balancing has different forms and books can be dedicated just to that. It is sometimes mixed with fault tolerance related stuff like the case that one machine goes out of service and others should handle its load. Documentation of MMO middleware and Googling the term will help you a lot to know more about the concept.

 

Network bandwidth optimization and network LOD and data prioritization

Both clients and servers have limited bandwidth and cannot send/receive an unlimited amount of data. Sometimes, like in mobile games, even the processing time of the messages is important for the developer so you can not send/receive anything that you want even when you have a good connection. On the other hand, sometimes some of user A's data are not relevant to user B, so why send them and make the server bills higher and user experience lower in terms of quality? Having the capabilities to prioritize data, sending them based on their importance and the current situation, and also assigning send rates based on logical factors, is a really important feature for any real-time multiplayer game. In different engines, you have different ways to implement network bandwidth optimization and LOD. In some middleware you should define all replicable data in a file and in some others its code oriented. In some situations you have full control and in some others you are not that free to do anything you want. These features can be implemented in different layers of the code. Some of it can be handled at the network layer, when routing traffic for servers and sending data to clients, and some of it is better to be handled at a little higher level near the game logic. In the end it's the networking code that should, based on networking logic and traffic or game's state, choose what to send, to whom and at what rate.

 

VOIP (voice over IP)

This feature is the ability to send the voice of users to each other. This is not a killer feature for most games and not having it cannot break the game in most situations but, if you have any plans to use it, it's really tricky. There are different companies providing different services for it. Based on your chosen platform, Unity web player or mobile platforms might not be supported and you should be really careful with this.

 

Database Libraries and systems

It's not an unknown fact that persistent worlds’ data should be saved in a storage system. It might be a RDBMS (relational database management system), a no sql storage, a new sql system or anything else based on the game's need or even a combination of some of them. It's really possible to do it in a good scale using RDBMS too but no sql is a great answer for the data storage requirements of today's games. EVE online for example uses Microsoft's SQL server and hero engine uses Oracle as their back ends but many other games use the concepts of no sql and big data these days. When EVE started development there was no sql buzz here and there and same is true for hero engine but they show that it's possible to do it in RDBMS. To know more about it you can read about CCP's tricks to make the database available and their scaling problems on the internet. If you don't want to be in those situations then you can use no sql databases. Your middleware might provide you some easy access to a serialization engine and storage system or might do it automatically for you based on some attributes in your classes and ... but it's possible too that an engine doesn't do anything about this and fully offloads the taskf. It might be good or bad for you, based on your database engineer's experience and needs, but an integrated system is always a good thing to have and the less code you write, the less bugs you'll have. If middleware does more, you can do less and write your game play. Some middleware can work with multiple databases and their data layer is open and in some others you have to use their choice. Most of the times, however, if licensing is not an issue, the engineers of those teams did a good amount of research and chose their back ends and libraries based on that. So, if you don't have real reasons, sticking to their choices is a wise move.
The number of different options and approaches is really high. There are many options other than famous sql and no sql databases. No sql databases have many different types themselves. New sql databases are also emerging and they are sql based databases which also allow access to low level storage systems when required. There are graph databases and memory based ones and SSD optimized ones. There are services which allow ACID properties on no sql none transactional databases. Do a good research if you know well about storage systems.

 

Unity based servers and their management

Middleware can help you in implementing the logic of the game. It might provide server side math/physics libraries, ready to use common code, or it can allow you to use your own lovely Unity stuff on the server too. In some situations it can have a little more overhead than pure implementations outside of Unity but in most cases (though not all card/board games) it's awesome to be able to use Unity servers. Even in card/board games it might be really handy to use your own infrastructure instead of a new one. After all Unity provides a lot of stuff other than collision detection. Unity provides GameObjects and events and Instantiations/Destroys in a familiar manner. It provides coroutines and Time management and ... Some middleware might allow doing the game logic in any language/engine you want but with less management, just doing the message relay itself. You should take a good look at your project and this feature of engines if you want to be able to finish your project.

 

Visual debugging of the server side game code

The ability to run the server side code of the game in a graphical environment which is rendering the game in one way or another is really useful for debugging. It’s important enough that when you are doing a complex game, it’s impossible to create the game without it. For simpler games, like turn based management games or chess, you might be able to easily write visualizers yourself. Printing values of variables and arrays is enough in most of those cases but if you are doing a RPG or a shooter the story is really different. Even doing simple visualizations, with capsules and gizmo like shapes, can help you a lot to find bugs and problems. If your middleware supports running server side game code in Unity then you are able to render the full game on server and at the end remove additional stuff like meshes and textures from it to reduce RAM usage.
This feature is useful in other situations as well: Do you want to record and broadcast the game? Do you want to do it in realtime like some Koreans do? If yes this can be a big help. Of course you can create a special client for this but if you can have it on the server it might be more than enough for you.

 

server instance management

Game instances can be created and destroyed as threads, processes or maybe other logical units based on your middleware and game logic. This ability is important to be able to manage game and server instances. Adding/removing server instances and machines shouldn't be hard and you should be notified when processes or game instances fail or anything else which is not good happens. You should not have to shut down the game to add a new computer to the server cluster.

 

fault tolerance and data redundancy

Any online game should have these features. your game should not stop working if one of the servers stops working and your user's valuable data should not be lost if one of your database servers crashes or anything else happens to it. Data redundancy and fault tolerance are really important in online games, especially the big ones. Your middleware's traffic routers and proxies should be able to respond correctly when one of your game servers dies and you should not have to deal with the whole logic yourself.

 

running in your platform of choice

You might want to run your servers on a cloud platform like EC2 or Azure. You might prefer Windows or Linux and lots of other choices. Middleware should be able to run on a good platform for doing MMOs and multiplayer games that you want/are able to operate. However, this factor should not be considered important enough to change your decision fully by itself but it’s a good thing to take note of.

 

common code

Your middleware might provide common MMORPG code or multiplayer shooter code and, if you are doing one of those games, the code might become useful and save your time for something else. Check for availability of reusable code that you can use in one way or another. Software reuse is one of the key factors of success in our industry. Systems like lobbies, inventories, shops and ... might be also really useful.

 

additional client features (avatar customization, day night cycle)...

Some features required for multiplayer games and MMOs like dynamic skies and day and night cycles or avatar customizations are missing from the main Unity package. They can be bought from the Asset Store or developed but some middleware might be available to provide these features for Unity themselves.

 

extensive documentation

What can I say about this? A software rated 4 out of 10 with complete documentation is better than a software rated 7-8 without good docs in my opinion. An unclear software with minimalistic approach to docs is good for doing university test projects but when it comes to the real world it will be super crucial to have a great understanding of your tools to create that awesome architecture which should last for some months if not decades. You should not have to guess anything and everything should be described to you. Those who have done real world projects before know the value of support and documentation when time to market is an important factor and when you understand that time is the most valuable thing you have.

 

Support and commUnity

Even if middleware has documentation as good as MSDN, still there are points that they did not cover fully there. Having good support and having a paid support option are really important. If emailing with the CEO of the company is impossible and they are not Microsoft/Adobe then you might going to have problems with them. Tech support should not answer you within an hour but 3 days is a reasonable time. You might think more than 2 is not ok or 5 days is ok as well. I just love 3.
The commUnity around middleware is an important factor. If there is always someone in commUnity forums who is answering you better and faster than the company’s support staff surely it’s a good point. Sometimes when new technologies come out, no matter how good they are you might be among the first users of them so it’s not an option always to have a good commUnity. Good commUnity is not a big commUnity, it is a commUnity that is helpful. Naturally in bigger communities you’ll have more intelligent people but also more stupid ones so size is not that important. Numbers of users in Unity answers is more than 10 times bigger than in 2008 (estimation) but the top users who answer really well and answer many questions has only grown from 5 to 15 and some of them have been replaced by others or spend less time there due to different things. The good thing is that more questions are being asked and chances are higher that you’ll find ready answers but when it comes to answering your own question, the situation is mainly the same as the days when we were 2000 people in Unity answers (personal opinion of course). Those who use multiple OSs understand this really well. The Linux commUnity might be much smaller than the Windows one but it doesn’t mean you’ll get Windows answers in milliseconds while Linux ones take days. But you might find 10 ready answers for something Windows related while finding 3 Linux related ones. Keep in mind that big commUnity is an advantage only if both products can compete technically.

 

live game tools (payment, analytics, CRM and others)

After you release the first version of your game, the process starts again for you as a developer of an online game and your game will require some support as well. You should solve people's problems and should hook to different services from Facebook social features to payment methods and analytics services. If your middleware provides them or has partnerships with them, it will save you lots of headaches.

 

supporting clients other than Unity

You might want to support clients other than Unity. A little web client for mobiles to do some auction or a webOS client or ... If possible, make sure that at least it's possible to do it. But if you'll not require it in the foreseeable future, don't choose worse middleware based on this. Be agile, don't over engineer and don't try to create some huge architecture because you're game might never see the light of day if you are an indie team. The main important thing is the current game and current design. You should be able to iterate fast and improve fast.

 

General considerations when choosing middleware

There are general considerations for choosing your middleware, tools and softwares to use, companies/persons to trust which we sometimes forget or don’t pay as much attention as required to them. When you are choosing middleware or a tool, you are effectively shaping your future. If the tool provider sells itself to a bigger company after gaining popularity or if they shut down services or stop development due to low amount of sales you’ll be alone in the cold. And this happens; the hero engine is now sold to another company called idea fabric; Big world is recently sold to wargaming.net; and instant action, the company behind torque game engine, is not available anymore. All products are available still but they have changed directions. I’m not saying it’s good or bad, I’m saying it happens and it will bring change if it happens. There are really bad situations as well. Many game engine providers are fully closed now or their development is fully stopped. In one way or another choosing the right company and tool is a talent, science or maybe an art form by itself :)
When you invest in such tools and middleware or services, whether it’s time or money, you are actually providing something to get something back. If the results are not that good, change early and don’t be afraid of it. Going the wrong path because you’ve spent 3 months on something has one and only one result: failure!
The good news is that, thanks to the world of the internet and the existence of search engines,social networking sites and technical communities, finding out more about companies and people is not that hard. If someone ran a healthy business for years so he/she will continue. Instant action is closed but garagegames is keeping torque alive. The development speed is not anything comparable to those glorious days but because they are healthy guys and have the opportUnity to continue their business, they are doing so. Search on companies and the people behind them, their roots and backgrounds and you’ll have lots of information. Talk to people who used their technologies and take a look at the company’s transparency and release dates and … If you love clear roadmaps, look if you have that or not and consider all important factors.
As the last point and maybe the most useful one that i want to bring up, I want to talk about decisions. I think the lifetime of any product/service/company is fully dependent of the smartness of the decisions behind it. I mean major decisions. When i saw Unity decided to have editor scripting while many others could but did not, when i saw them choose to target web in 2006 and when i saw them using PhysX while many others were using ODE/Bullet and when i saw them using Mono instead of Lua/Python, i said one thing “Smart people! They are going to stay around”. Take a look at company decisions. Major decisions are super important. If they did not choose a good approach at first and now are fast copying another competitor’s approach, chances are high that they’ll follow for the next coming years as well. Naturally as someone who is researching these you should have a good amount of awareness and knowledge yourself to avoid over excitement or falling in advertising traps. My father always told me “Educate yourself and study, try to be the best and you’ll have no problems and will be happy”. These are not his exact words (well we don’t speak English as a native language as you can understand while reading this) but he meant this and it’s a general and universal rule for everything. Don’t do guesswork when you don’t have to. Educate yourself and investigate everything carefully and hopefully you’ll end up in something good.

 

How to learn More?

You can use wikis and documentation of the available middleware for networking no matter if they are for Unity or not. They usually describe the concepts well. Also you can find many papers and articles about topics like load balancing and network LOD if you search the internet. Some topics like choosing a database are vast. In this era we have RDBMS, Graph databases, no sql databases and new sql databases and some other types and hybrid solutions.

 

I have some questions, where can I ask?

Contact any of the members of NoOp Army by anyway you want. The writer of this article, Ashkan can be contacted using skype and email. His skypeID is ashkan_gc and his email address is [email protected]