Making a Multiplayer XBLIG, Or: How I Learned to Stop Worrying and Love XBLIG

Hello, my name is James Petruzzi and I’m a co-founder of the independent game company Discord Games. Our first game Take Arms was released on XBLIG in August 2011 after two years of hard work, and was featured in the Indie Games Summer Uprising promotion. The game is a 2d multiplayer shooter or, as others have put it, a “2d Call of Duty”.

There were a lot of ups and downs during the development of Take Arms. It was our first released game, and we made a metric ton of stupid mistakes that you can read about in our post-mortem. Since the game was multiplayer-based, Kairi Vice approached me about writing a piece to clear up a lot of the misconceptions and confusion over multiplayer Xbox Live Indie Game development. This article is aimed at giving a general overview of that topic, as well as detailing the most common pitfalls developers will run into. There is a lot to cover, so I apologize ahead of time if I missed anything.

Take Arms

1. Network Architecture 101

All networking on Xbox Live Indie Games is Peer-to-Peer (P2P). If you are coming from the PC world where client-server is the norm, this will be totally foreign to you. In client-server, one person or server acts as the host and everyone else is a client and connects to them. Each client regularly sends their inputs (keyboard and mouse) to the server, the server processes that information and determines what happens, and then sends out an update to every client to keep them in sync. In this method, there is no argument of what happened since there is only one authority to decide. If I just shot my partner Tim in the face, that is what everyone will see.

In contrast, P2P relies on every player being their own server. Each Xbox takes the local player’s input, receives input from the other players, updates their version of the world, and then sends out an update to everyone else about the local player. As you can see, there are as many authorities as players with P2P. Maybe on my machine I shot my partner Tim in the face, but on his he saw me behind a wall shooting bricks in the face. In my perspective I would just be shooting him forever and he wouldn’t die, and in his perspective I’d just look like an idiot shooting a wall. Since his machine is the one that determines if he got hit, nothing happens. So which machine do you believe? Mine that is telling me Tim is impervious to bullets, or his that is saying I hate solid masonry?

As you can see, P2P is not the preferred method to networking by most of the modern gaming world. It is generally inaccurate, and is also very susceptible to manipulation. If I am the authority of my player, I am the only one able to decide if I die. If I pull my network plug, I am not sending packets to everyone telling them my health or registering their hits. In a client-server setup, all that logic is run in one place and is much more difficult to manipulate.


2. Membership Requirements

If you hear a developer moaning about developing a networked game for XBLIG, it is most likely going to be about the cost and overhead involved. It is surprisingly expensive and difficult for a platform designed for indie developers. When you are first getting started it is easy to test between you and a partner. You both probably already have an Xbox 360, Live Gold subscriptions, and App Hub memberships (Microsoft’s program that all developers must join for $99 a year).

Where you are going to run into trouble is when you want to test a full game session. In our case, we decided on a limit of eight players for Take Arms. The difficulties started to arise when we wanted to get some other members of the team and friends involved to reach the player limit. The first thing you will realize is that in order to deploy the game to an Xbox 360, you have to be an App Hub member. There is no way around this. You will even receive an exception message on the Windows build saying you are not an App Hub member if you try to search for games without one.

The second thing is that you are using Microsoft’s Live servers for matchmaking if you are playing over the internet. As you can imagine, Microsoft is not letting you use those for free. Every person that tests your game is going to need an Xbox Live Gold subscription in addition to the App Hub membership. This can be circumvented if you test locally over a LAN with System Link, but that gives you absolutely no idea of real-world performance. 

So there is a yearly cost of $160 per tester, granted they have their own Xbox 360. If you’re ingenuitive you may be thinking, “Why not use Windows instead?” Unfortunately, you still need both memberships, and it is even more time-consuming to get them playing due to licensing issues.

Take Arms

3. So what about Windows?

Surprisingly, there is no XNA networking support in Windows. Yes, you can develop your networked game in Windows and test it, but that is about it. There are two install packages for XNA; XNA Game Studio and XNA Redistributable. You will need XNA Game Studio if you are a developer working in Visual Studio. It integrates with Visual Studio making it easy to deploy your code to Xbox, and it includes the XNA framework to code against. On the other hand, the XNA Redistributable is what an end-user must install before playing your game. It contains the same XNA framework that came with XNA Game Studio sans one important piece: the networking library.

If you want your friend to help test in Windows, you will need to install the full development environment and the needed subscriptions mentioned before. For clarity, if I was hooking up a friend who doesn’t own an Xbox to test Take Arms here is what she needs:

What if you just want to make a multiplayer XNA game for Windows only? Well, you’re not going to be using the built-in networking. According to Microsoft’s license it is illegal for you to distribute the XNA Networking DLL with your game, and besides that, they would still need both memberships to play online. Your only options are either going with a 3rd party networking library like lidgren or writing your own.

A more general problem with P2P on PC is the overall terrible security. On Xbox, the game code resides on your encrypted hard drive and the network traffic is encrypted so it can’t be tampered with directly, but on PC it is like the wild west. People can modify their outgoing packets to always have full health, modify the IL to ignore damage altogether, or any other of a million possible hacks that would be difficult to detect. If you are doing a competitive game on PC, you should use client-server architecture.


4. Bandwidth

Be prepared to deal with bandwidth challenges if you are developing a multiplayer title. I do not have any experience with multiplayer outside Live, so I can’t really say if it is better in the PC world or not. I will just try to convey what our experience was and what steps we took to get Take Arms as smooth as it is now. Please keep in mind all traffic is routed through Live servers, so it does not travel directly from console to console. Voice communication is going to eat into your bandwidth as well, so expect some work ahead of you.

Going off the Net Rumble Starter Kit on App Hub, we used two different packets for players: a status update (state, health, ammo, etc.) and their controller inputs (which way the sticks are pointing, which buttons are pressed, etc.). With these two packets, you can somewhat reliably recreate the player’s actions on remote machines. Your first thought is going to be that you need to send these every single frame to make it smooth. While that will work well on your LAN, you will quickly find out it does not on Xbox Live.

In Shawn Hargreaves’ blog, he clearly lays out the fact that you need to stay under 16 kb/sec total bandwidth. This gives you 8kb going out, and 8kb coming in per second. There are two things you can do to improve bandwidth consumption: send less often and compress what you do send. We played around with the send rates for quite a while, and we ended up sending each player’s controller inputs every 15 frames (four times a second), and their status update every 30 frames (two times a second). This gave us something that was jerky, but stable. The other part of the equation is compression. We worked hard to pack as much info about the player into as little space as possible. For instance, a single byte stores how many grenades you have, how many clips you are carrying, and how much ammo is currently in your gun.

After reducing bandwidth, the next step is fixing the jerky motion. Your machine will update the remote player’s position when it gets their status packet, and apply their controller inputs to him as it receives them in the meantime. This may or may not be what is really happening on their end. When you receive their next status packet with their new position, your estimate via inputs will be off. This causes jerky corrections to happen that can be fixed with smoothing. Instead of instantly updating the remote player’s position to the new one, move them over time. If you keep track of how long it is taking to receive a packet from them, you can adjust how extreme the smoothing speed is. Check out the Prediction Sample on the App Hub for more details.

Take Arms

5. Pricing

During development we debated back and forth about the right price point. When we got close to release we had the option of 80 ($1), 240 ($3) or 400 Points ($5). We personally thought the game was worth 400 points, but knew it would sell more units if we lowered it. Making it 80 points did flash through our minds once or twice, but the 50MB limit, (we were at 49MB and did not want to raise the price if we added more content down the line) and the allure of roughly 3x the profit per sale pushed us towards 240 Points.

If you are making a multiplayer only game, your number one concern needs to be getting as many people online as possible. More people online means more value for your game. Your goal is to have friends recommending it to their friends to play with them online. While Take Arms sales started out decent enough in September during the promotion, they fell off a cliff in October. My first thought when seeing this was, “We need to drop the price immediately!”. I jumped onto App Hub to lower it, only to see the price was locked for 90 days from the date of release. 

Sales skyrocketed after we finally lowered the price in late November. We sold more copies in December than we did for the previous three months combined. Make no mistake, there is only one price point on XBLIG. You will only find misery if you go with the higher ones. If you have a multiplayer game, don’t even consider anything else. You need to build your community first and foremost. 


6. Patching

What is one thing that just about every multiplayer game has in common? They almost always have at least one major problem on release, and usually have a patch out within the first 48 hours. Networked games in general are very difficult to test due to their unpredictability, and I think in general most gamers understand that. Pushing out a patch on PC can be done as soon as the new code is tested. On XBLIG you wait. And wait. And wait!

A severe problem was uncovered on the day of release for Take Arms. Somewhere along development, a loop was commented out that iterated through all the packets in the buffer every frame. This in turn caused the game to only process one incoming packet per frame. With only a few people in the game, you couldn’t notice it at all. Once the game session was maxed out, however, the packets quickly started backing up. The game would slowly become unplayable, and it would eventually crash due to a buffer overflow. It took us about a day to figure out the issue and get some people together to test the fix. We put together the updated game package, uploaded it to App Hub for Peer Review, and then got a serious wake up call. 

We were well aware of the seven day lockout after failing a Peer Review. You don’t want your approval process to be clogged with untested games, so locking them out for a week helps to prevent abuse. That makes sense. However, what we did not know was the exact same lockout applies to approved games as well. So here we are, launch day with a fix for a broken game, but we can’t submit it to Peer Review for a week! On top of that is a 48 hour minimum Peer Review time. In total, a nine day wait to push out a simple fix.

Seeing that the game was totally broken, I did the only thing I thought was right. I pulled the game from the Marketplace in the middle of the Indie Games Summer Uprising promotion. We might have lost a lot of sales because of this (people playing the bots-only trial would not have known it was broken and still purchased), but I felt like it was the only course of action for us. I would hate to ever buy a game only to find out it is completely broken.

Take Arms

7. Trials, Bundles, & Re-release

Some people have asked why they cannot play online in the Take Arms trial. It is an integral part of the experience you are purchasing, so the question is definitely legit. The sad answer is that networking is not supported in any fashion while in Trial Mode on XBLIG. If your game is based solely around multiplayer then your only shot at giving them a taste of what it is like is to include bots. Also, keep in mind that after release there are very few people out there with copies of the game. This means that even with the purchased full version, they are probably going to be falling back on bots a lot of the time. In general, this is a huge issue with XBLIG as the market itself is small and self-contained, and the games do not tend to bring in outside people.

XBLIG Bundles are another thing we have been asked to participate in. Bundles are a point of difficulty on XBLIG for even single player games, but due to Xbox Live’s design they render multiplayer games near useless. The same goes for “re-releasing” your game as a new product with more maps or features. XBLIG uses Xbox Live for matchmaking, and as such each product has a unique identifier. This means when I go play a match of Take Arms, it is sending its unique ID to the Live servers to find matches. If you release your game in a bundle or re-release it as a separate game, you will not be able to connect to players that bought the original version due to different ids. Since your online community is already very small, you would only be hurting yourself by splintering it.


8. Leaderboards

Another commonly mentioned down-side to XBLIG is the lack of Leaderboards. I suppose Microsoft does not think the App Hub membership or XBLIG sales (Microsoft takes a 30% cut of each sale) cover the cost of storage and bandwidth, so it’s not made available to XBLIG developers at all. You are not allowed to talk to outside servers either, so how would you share high scores?

The only possible method is to use P2P networking in the background of your game. The idea is that you start playing your game as normal, and in the background the game does a search against the matchmaking servers to see if anyone is hosting a session. If it finds an open session it will connect to it, otherwise a new session is created. Once you are in a session with other machines, it talks to all the other Xbox’s in the session and syncs up the scores.

While it sounds great in theory, there are several pitfalls to this method. The first is that other people must be on at the same exact time you are to swap scores. This alone severely limits what data you’re going to get. Kris Steele told me that he left his Xbox on for weeks to act as a server for one of his games and the results were still poor. The second pitfall is that it’s incredibly hard to test. Given how difficult it is to test a regular multiplayer game, testing a background process becomes even trickier. Milkstone Studios are very prolific XBLIG developers and they even ran into issues on a recent release.

I received some flack for not including Leaderboards in our recently released title 48 Chambers, but I did so for good reason. If I cannot be reasonably sure that you are going to have a smooth experience, I am not going to risk it. After going through all the Take Arms problems, knowing how flakey the networking can be and how poor the turnout is, I decided it was best to not even bother with it.  

48 Chambers