Security in Online Games

Author: by Chris Kirmse

We are witnessing the biggest revolution in games since the introduction of home computers: online games, played via modem over the Internet. Many companies now insist that every game they develop must have an online component. The most ambitious online games are persistent worlds, which immerse hundreds of players in a single, shared environment. Traditionally, game programmers have faced well-known challenges, such as artificial intelligence, fast 3D graphics, and input devices. In the development of persistent worlds, there is an entirely new set of problems.

Consider the following:

It seems that the introduction of every new game is followed by the introduction of web sites dedicated to cheating the game. In one- or two-person games, cheating is a minor issue, since it only affects one or two people at a time. However, a single cheater in an online game can affect thousands of people and have lasting implications.

Of all the issues the online-game developer must be concerned with, security might initially seem like a trivial problem. However, it is central to keeping a system running. A design error in the communication scheme or graphics engine might lead to suboptimal performance, whereas a design error in the game's security is the first step to ruin. In a scenario where customers pay for continued access to the game, letting hackers run free is a sure way to lose a large part of a paying customer base.

Consider the following scenario in a persistent online game. Ben is an avid player of the game, and one day he shows it to his friend Alyssa, who is a computer science undergraduate student. Out of curiosity, Alyssa writes a program to examine all network traffic generated by the game. To further her knowledge, she extends the program to randomly modify pieces of some of the messages the client sends. Unbeknownst to her, the server contains a bug that accidentally incorporates some of this random data into the persistent world and corrupts its database. About a week later, without warning, the server crashes, and the system administrators are forced to rewind the database to the previous week's state. Outraged, thousands of customers cancel their accounts, and the game dies a violent death. Given the damage that a simply curious programmer could cause, imagine what a hacker bent on revenge might do to exploit the smallest security hole.

In this article, we identify areas where security has historically been a problem, and where a client/server system is typically weakest. Figure 1 shows several ways a hacker can attack a game system. We also suggest defensive mechanisms for the most common attacks, and we describe what can happen when hackers get through.

Figure 1

Real-World Adventures

Lest you think that security is just a theoretical problem, consider recent events in some popular games. It appears that Blizzard's Battle.net service contains some major security flaws at the outset. As any experienced DIABLO player on Battle.net will tell you, people have found ways to gather incredibly powerful equipment without earning it. Someone even went so far as to hack the game so that characters in a multiplayer game can be stolen right over the network. In QUAKE, hackers have created automated programs called bots that can automatically destroy opponents. These are free game networks; imagine the amount of effort hackers would exert to attack a pay-by-the-hour system, or if someone offered a Ferrari as a prize in an unregulated QUAKE tournament.

During the development of MERIDIAN 59, we received a shocking reminder of how dedicated hackers can be. At the start of the game, a player can customize a character by assigning numerical values to certain attributes. After the player selects the values, the numbers are sent to the server. Unfortunately, at one point we changed the selection method without changing the check in the server that ensured that the values were legal. Naturally, someone soon modified the client executable to send outrageously large attribute values, and they shared this hack with their friends. Soon we had godlike characters wandering around the world. Our first fix added the values together and checked that the total was under the legal limit. Within a week, someone had found that certain attribute values, when added together, caused the sum to overflow back into the legal range, allowing them to resume cheating. The lesson is that hackers can be extremely clever and persistent, so the lazy solution of security through obscurity is generally not enough.

It's not always the players who are to blame for security problems. MERIDIAN 59 has an administration mode, accessible only by game administrators, that gives complete control over the server. However, during our beta test, a careless administrator learning the system made some poorly considered modifications to the game world. The effects didn't show up until hours later, when suddenly characters' names started disappearing, and monsters began popping up in their inventories (they even attacked the players who were holding them!). We were forced to restore the game state from an hours-old backup. In a commercial system, this would have been quite disasterous. To avoid careless mistakes like this, you should be as restrictive as possible when handing out administrative powers.

Persistent World Architecture

Most persistent worlds share the same basic architectural features. To play the game, a user either installs client software from a CD-ROM or downloads it from the Internet. The client software contains code that communicates with the game server using a custom protocol designed for the particular game. Large static data, such as graphic files, sounds, music, and level layout are typically part of the initial installation onto the client.

The peer-to-peer approach used for most LAN-based games does not scale well to large-scale persistent worlds. When the game lasts longer than a single session, it becomes difficult to deal with players dropping in and out of the game arbitrarily. Performance also becomes a problem, since the amount of game state increases linearly with the number of players, while the amount of network bandwidth remains constant. A simple networked game, such as a shooter, has very little game state - perhaps as little as the coordinates of all the players and monsters, their weapons, and their ammo. This allows each game client to know the entire game state, and makes it possible to transfer to everyone else all the game states changes made by one client.

To solve the problems of a large-scale game, on the other hand, one or more game servers are set up in a central location to keep track of the game state. Each of the game clients communicates its changes exclusively to a game server, which communicates those changes only to the clients that need to receive them (Figure 2). For example, if one user moves his or her character, the server only needs to tell other clients in the user's vicinity. This reduces the bandwidth use of the game to an amount that will not overload users' modem-based connections.

Some virtual worlds last many years after their initial creation. This is one of the benefits of creating a persistent world instead of a transient game. To extend the life of a persistent world, developers often grow the game over time, by adding more areas to the world and new features to the game play. To support this, the client software must have a way to upgrade itself, preferably without any user intervention. This is one of the most important parts of the initial release of any persistent world, because although the world may initially be quite primitive, it has the potential to be improved dramatically over time.

Besides the game servers, there are likely other processes required to make the whole system work in a commercial environment. User account information might be stored in an external database, dynamic game updates might use FTP servers, and automatic mailings could require an SMTP server. These processes could run on the same machine as the game, but for performance reasons, they usually run on separate machines.

Security Precautions

There are two security goals in an online game:

Protecting sensitive information is mostly a matter of configuring the server complex correctly. Each machine connected with the game, such as any web, FTP, or database server, needs to be secured. All game servers should be behind a firewall that only allows data through the ports that the game needs to operate. Finally, the server complex should be located in a locked area, since physical access to the machines circumvents all other security precautions.

As you can tell from our previous example with MERIDIAN 59, some amount of internal security is also necessary to prevent employees outside the development staff from damaging the game or leaking information. In general, administrative powers should be given out strictly to those staff members who require those powers to do their jobs. Once you grant power, it is almost impossible to take it away, so be extremely sparing. Limit serious power, especially the ability to change account information, to a few trusted, on-site individuals.

Detecting Cheating

A good first step in foiling cheaters is to set up the system to detect cheating automatically. Record all player logins and logouts, as well as important game events such as player advancement. Keep track of key quantities in the game, such as the total amount of money and numbers of rare items in existence. If you review these statistics regularly, you can tell when there is a major problem, such as the appearance of multiple copies of items that are supposed to be unique, or the overnight doubling of the money supply.

Also, talk to your players occasionally. They probably have more experience actually playing the game than even the game designers do, and they are anxious to ensure that other players don't have an advantage over them. If they suffer at the hands of a cheater, they will complain to you loudly. On the other hand, since players by design have a limited understanding of how the system works, they tend to report far more incidences of cheating than actually occur. Consider creating an e-mail account specifically for players to report instances of cheating. Be aware that sifting through all the complaints can be a full-time job; we recommend waiting until a problem is reported at least several times before spending time to investigate it further.

Attack and Defense

Clearly, the consequences of a security hole in an online game are much more serious than in a standalone game. A malicious player might use a security hole to cheat or to compromise the integrity of the game. Letting hackers run free is a sure way to lose a large part of a paying customer base, so the game design should address this problem from the beginning. There are three places in the game that are vulnerable to attack: the data files, the client/server protocol, and the client executable itself.

A simple approach to securing data files on client machines is to encrypt them. Since there is a performance penalty at run time for decrypting files, however, only those files that contain important information need encryption. There is probably little or no harm in letting a player examine and modify music, sound, and graphic files, whereas access to levels or speech files might give someone a substantial advantage.

Encryption is not quite enough, though. Merely renaming or copying one data file over another might allow someone to fool the client into thinking that a player is in a location other than where the server places the player; this could have consequences ranging from odd player behavior on other people's machines to granting the player access to otherwise restricted areas. The solution is to have the server verify that the client is using the correct file data, not just the correct file name. When the client loads a file, it can perform a checksum of the file and send the checksum to the server (given that the file is encrypted, the checksum can be quite simple and inexpensive to calculate). If the server finds that the checksum doesn't match its expectations, it concludes that the player has cheated, and either logs this information or takes immediate corrective action. Note that it's not enough for the server to simply ask the client to deal with the problem, since a malicious user might block such a message.

In fact, there are many possible attacks on the client/server protocol. It's fairly easy for a hacker to use a packet sniffer, which is a program that displays all data transferred over the network (many such commercial programs are readily available). Armed with a sniffer, a hacker might try to reverse engineer the client/server protocol with an eye to changing packets as the client sends them. Encrypting the protocol effectively solves this particular problem, but there are others. Even when packets are encrypted, a hacker can capture an outgoing packet and resend it, possibly hundreds of times (an attack called "replaying packets"). If the packet is a request to fire a spaceship's lasers, replaying packets could give someone a significant advantage by making the ship's lasers fire rapidly.

A good way to thwart packet replay is to assign each packet a sequence number, which changes from one packet to the next. If the server receives a packet with the wrong sequence number, it knows that the sender is cheating. Of course, the sequence number should be more than just an integer that increments with each message; that scheme is far too easy for a hacker to replicate. Instead, the sequence number could be a mix of anything that acts as a state variable for the protocol. For example, it could be the total number of bytes that have been sent since the client started. See Figure 3 for a diagram of a typical packet.

Figure 3

Something else a cheater might do is use a packet sniffer simply to block certain messages from reaching the client or the server. Assume for argument's sake that the player's health value is stored on the client, and that messages that inflict damage are blocked from reaching the client. The player blocking damage messages would become invulnerable. By storing all important data on the server, it's possible to prevent message blocking from giving anyone an advantage. However, message blocking attacks can be subtle, and each protocol message needs to be examined to see what would happen if someone blocked it.

The client executable resides on the player's machine, and thus is vulnerable to tampering. Worse, the client possesses important information when it runs, including the contents of data files, which must be decrypted as they are loaded. A cheater might use a debugger to view memory while the client is running, or even modify the executable to disable other security checks. Unfortunately, there is little defense against such attacks. Any user sophisticated enough to modify the executable will probably be able to get around any attempts to prevent tampering. One way to contain the damage is to limit the client's knowledge of the game as much as possible. After all, a hacker can only learn as much as the client knows. By storing data on the server, a network roundtrip is required to retrieve it, which hurts performance and complicates the code. Data security is a balancing act between performance concerns and the danger that players may find a way to learn all of the information in the client.

Another possible security problem is the use of automated programs called bots to simulate user actions. Such programs are easy to write in a windowing environment, since external programs can send arbitrary window messages to the game client. A common use for a bot is to perform an action repeatedly, faster than a person ever could. To thwart bots, the client should require that a certain amount of time passes between two successive user actions. Depending on the particular operating system, it might be possible to do more to discourage bots by restricting message sources (for example, you can detect window messages sent by external programs under X Windows). On the other hand, some game designers might consider bots useful convenience features for certain purposes, and might even encourage their use. As an illustration, a player has written an add-on for MERIDIAN 59 that expands the number of available alias keys by simulating user input.

Dealing with Cheaters

What should the server do when it detects a security problem? One answer is to immediately disconnect or possibly suspend the offender's account. Suspension might be too severe - even one misdirected account suspension is a serious customer relations problem. On the other hand, simply disconnecting an account has the disadvantage of letting the hacker know that his or her attempt has failed. A more subtle approach is simply to log the security breach, and periodically review the security log. This approach discourages casual hacking at the cost of letting a few security violators through for a short time. It is best suited for less severe security breaches, where this cost is acceptable.

For more severe security problems, you should be relentless in removing cheaters from the system. Although the number of people with the technical knowledge to hack your system is probably small, there are many more who are willing to use other people's hacks. In a commercial system, the terms of service should disallow tampering with the system, so that you can disconnect serious cheaters instantly. It might be more difficult to keep cheaters out in a free system, because cheaters can just keep signing up.

When you find a bug or security hole that cheaters have exploited in a persistent world, you often have to write code to set things right again. In MERIDIAN 59, there were several instances where players found obscure ways of getting free money in the game. Each time, we had to write special-purpose code to find large stashes of money and reduce them down to reasonable sizes. Such code is error-prone, because there are many special cases, and because it is impossible to tell whether the money was actually obtained illegally. Be sure to test corrective code extensively. We actually set up special testing servers with external players to deal with problems like these.

A fact of life in the security business is that no defense is foolproof. Given enough time and money, an attacker can always defeat any security scheme. A hacker can disassemble and rewrite your entire client, dedicated hardware can break your encryption scheme, or a terrorist can drive a tank through the door to your server room. Your job is to make cheating prohibitively expensive, so that potential attackers have little desire to try. You have to judge for yourself how secure is secure enough, taking into account what's at stake if someone breaks through. Also, keep in mind the fact that the time you spend beefing up security is time that you could have spent improving the game in other ways.

Applying What We've Learned

Let's apply the principles above to a concrete example of a near real-time action game. All persistent data is stored on the server, and in most cases, the client sends a message to the server each time the player requests an action. The one exception is motion; to provide a realistic environment, the game must show the player moving as soon as a key is pressed. This one fact has important consequences for the game design and the protocol.

A hacker who finds out that the client is sending motion messages might try to modify the messages to make his character move more quickly. Encrypting the protocol should be enough to prevent this attack. However, the hacker might record outgoing motion packets, and then resend them later, achieving the same effect. Adding a sequence number to each message should stop this problem. For added security, the server can attempt to validate player moves. Simple checks, such as calculating the distance between two successive moves, can find the most serious cheats. Another alternative is to perform full collision detection to validate every incoming motion message. In many cases, collision detection is too expensive to perform on the server, but in some games it is possible. Where it is too expensive, the server could verify only a randomly selected sample of moves; this can even be done offline or by a separate process on another machine.

In a strategy game, it isn't as important for motion to take place in real time (consider a turn-based game like CIVILIZATION). In these cases, it might be acceptable to send every move to the server, which would verify the move and send back a reply if the move is legal. If the game is too slow when every move generates a network round trip, moves can be batched into larger groups and sent to the server for verification all at once. This is a good example of how security concerns can affect a game's performance.

Take It Seriously

Security is a serious concern in all online games, but especially in persistent worlds. A single security hole can make your customers disappear overnight. As more games move to the Internet, we'll hear about security more often, since players will break through the weak protection in most games. With proper planning and eternal vigilance, however, you can avoid becoming a casualty of the online revolution.

Andrew and Chris Kirmse are the inventors and primary developers of MERIDIAN 59, one of the first graphical Internet games. They have been writing computer games together since 1982. Andrew holds degrees in physics, mathematics, and computer science from MIT. He can be reached at [email protected]. Chris graduated from Virginia Polytechnic Institute in 1996 with a degree in computer science. His e-mail address is [email protected].