Why Making Multiplayer Games is Hard: Lag Compensating Weapons in MechWarrior Online

Network latency, or lag, is a reality that multiplayer games need to deal with. Messages transmitted over the internet take time to reach their destination. Depending on the route and the distance they travel, these messages can spend a relatively long time in transit. This can have a negative impact on gameplay experience, especially for quick paced client-server games like first person shooters. What seems to be a very simple experience, like shooting and hitting a target, is suddenly very difficult to make consistent and satisfying for players. Needless to say, making multiplayer games is hard and there are many problems developers need to solve. In this article, I will discuss what MechWarrior Online’s weapons system does to deal with lag.

One strategy for dealing with lag is to reduce the distance and thereby improve the route between players and the game server so that the lag is not as noticeable. This can be achieved by regionalizing servers and ensuring players only play on those servers that are closest to them. Or by ensuring that your game traffic takes routes through specific networks to optimally reach your servers. Unfortunately, this may not always be possible due to cost or can only be leveraged in a limited capacity; even then this will not necessarily work for players in rural areas or with limited internet connection quality. MechWarrior Online (MWO) is no different in this regard and uses regionalized dedicated servers to try and minimize latency for players. However, this often isn’t enough for most players and we still need to do more.

Client Side Prediction

Client side prediction is a phrase you may hear when people talk about client server multiplayer games. It is an important concept that makes it possible for multiplayer games to feel smooth and fluid, especially fast paced games. Without it, even a small amount of lag can be very noticeable and frustrating. The idea of client side prediction is simple: instead of waiting for the server to tell the client the result of an action, have the client predict the outcome of that action as though it happened immediately. For example, if the player wants to move forward, move the player immediately on the client and then send the move request to the server. As far as the player can tell, it appears as though they moved immediately even though the reality is that on the server they haven’t moved yet. If the game client’s predictions are always correct then the perception is that there is no lag. Of course predictions can’t always be correct and something needs to be done to address that problem, but that is beyond the scope of this article. MWO heavily depends on client side prediction. Both for its movement and weapon systems – when the player attempts to move their Mech or fire their weapons, the game will respond immediately.

Lag Compensation

Now that we have a client that responds immediately and appears to experience no lag, we have another problem. Although a player’s actions appear to take effect immediately, the world that they are responding to is out of date. Remember, we have only made it appear as though there is no lag, but information coming from the server is still delayed. That means the target a player sees on their client is not actually there on the server and even worse, once they shoot at that target and the firing request is received on the server, the target has moved even further from its original position. The time it takes to receive information from and then send a request to the server is referred to as round trip time or ping. Unless a player has zero latency, without any lag compensation, players would need to lead their shots ahead of a target in order to hit them; the higher the ping, the further ahead they would have to lead. As you can imagine this is very annoying for players. This is illustrated by the images below.

What the player sees when they are lagging

What the server sees when players are lagging

From the player’s perspective, it looks like they should be hitting their target dead on, but from the perspective of the server it looks like the player is completely missing. In order to deal with this problem we need to compensate for the lag of the player somehow.

At this point you may be wondering why even bother solving this problem. Since the client is predicting everything, can’t the client just tell the server they hit the target or not? The simple answer is yes, that is a possible solution; however, there is a very big caveat. The moment the game client is allowed to tell the server anything the potential for some form of cheating is created. On PC, a game client can trivially be modified and a smart cheater will modify the game client to send hit messages whenever they want, even when they aren’t near or can’t see targets. As such, this notion of client authority, especially in the case of dealing damage, is an incredibly risky proposition. If a multiplayer game is targeting a platform that provides some form of “protection” for the game client, like a console, then this may be a viable solution. But in the PC world, if you give the client any kind of control over the game’s state, it will almost certainly be exploited. There are many ways to combat cheating, but that is beyond the scope of this article. For our purposes, the safest solution is to leave the server in control and this is what MWO does.

In MWO there are two main categories of weapons and each needs to be lag compensated in a different way. These two categories are trace fire weapons – things like lasers and machine guns – and projectile weapons – things like missiles and ballistics. Trace fire weapons are much easier to deal with, so let’s look at them first.

Lag Compensating Trace Fire Weapons

Trace fire weapons fire instantly, impacting their target the moment they are fired. They do this by performing a ray cast or line trace against the current game world state to determine the impact point of the weapon. Therefore, what we need to lag compensate for is this line trace. If the server keeps track of the past positions and orientations of targets, the server could then use the player’s ping to determine exactly where the target was when the player shot. This is what the blue Mech in the image below represents. It is lined up with the position on the firing player’s screen above. MWO refers to the position of this blue Mech as the rewound position of the target. Now that the server knows where the target should be, it can move the target back to this spot, perform the line trace to determine if a hit occurs, then put the target back in its original spot. In this way, the server makes the prediction of the game client accurate.

Server view of rewound target position

In the case of MWO, it is not sufficient to only store past positions and orientations. Each component of the Mech can be damaged separately, and this is a very important gameplay mechanic, so the animation state is also relevant. This must be remembered and rewound on the server as well. You can see this in the image via the difference in the walk cycle between to the rewound Mech and the present Mech. Now that we can deal with trace fire weapons, let’s see how we can build on this solution for projectile weapons.

Lag Compensating Projectile Weapons

Projectile weapons work very differently from trace fire weapons. They fire a piece of geometry, like a slug or missile, into the world and this geometry sails through the air and deals damage once it impacts a target. A simple rewind will not work in this case. If we used the same approach as trace fire weapons and assumed the projectiles travelled at an infinite speed, the server might incorrectly detect the projectile struck a target immediately, even though the target could have potentially moved out of the way before it hit them. And even worse, players intentionally lead shots ahead of targets for projectile weapons since they are using their judgment to predict where the projectile will be by the time it reaches their target. In this case, a player with perfect aim, and a server with perfect rewinding will cause the player to always miss. We need a different approach.

Game client leading a shot for a projectile weapon

What we would like to do is fast forward the projectile on the server to catch up to the position where the client has predicted it will be. If we can make the prediction of the client correct, then we will have compensated for the lag on the game client. In order to do this, let’s break down the lifetime of the projectile into two parts.

By the time the server receives the request to fire the projectile, half of the player's round trip time will have passed. But remember, the server is supposed to be ahead of the client by one half of the round trip, so as far as the server is concerned the projectile must have existed for ping seconds in order to be consistent with everything else in the server’s world. During this time, the projectile may have already struck a target on the client, and the server may need to do something about that, but let’s not worry about that for now. Let’s call this first stage of the projectile’s life the rewind period. Everything after the first part of the projectile’s lifetime is the point at which the server’s simulation of the projectile and the client’s prediction of the projectile should be synchronized. It’s important to note that synchronized in this context does not mean that the projectile is exactly in the same spot on the client and the server. The client is expected to be behind the server by one half the round trip time. So, in this case, synchronized means that the projectile is in the same place relative to the rest of the objects in the world between both the client and server. Let’s call this stage of the projectile’s life the synchronized period. Dealing with the first part is a bit trickier, so let’s tackle the second part first.

Let’s assume the projectile has simple physics and just moves in a straight line at a constant velocity. In that case, by the time the request to fire the projectile arrives on the server, the projectile will have already travelled a distance on the game client equal to its velocity multiplied by the ping of the client. If the projectile has more complex physics like drag and gravity, it should still be possible to calculate the distance it has travelled using a more complex formula.

Overhead view of projectile trajectory on server

If we ignore the rewind period for now, the server can make the client’s prediction correct or synchronize with the client by simply spawning the projectile the distance it has travelled ahead from its normal spawning position. In other words, the server will not fire the projectile from the barrel of the weapon, but instead fire it ahead by distance units along its trajectory. If the projectile has more complex physics the server may also need to calculate the appropriate velocity and acceleration the projectile should have at that point along its trajectory. Assuming the projectile moves deterministically, it should now travel in a synchronized trajectory between the client and server. That's easy enough, so what do we do about the rewind period?

If we were to always spawn the projectile ahead by the rewind period distance, then it would be possible for projectiles to tunnel through targets or even static geometry. It’s easy enough to prevent tunneling through static geometry by performing a line trace. But what about moving targets? If there was a target inside the rewind period, then it’s possible the projectile has already hit a target according to the client’s prediction before the request even reaches the server. This is the real problem with projectiles – the server needs to be able to detect this situation and to make the client’s prediction a reality.

One way to do this is with an iterative solution that combines rewinding with some kind of fast forwarded simulation. When a projectile fire request is received on the server, it could rewind the target, spawn the projectile and tick the projectile and target forward over the rewind period. If a collision is detected between the target and projectile while ticking forward, we could deal damage to the target and avoid spawning the projectile at all. If no collision is detected, we can spawn the projectile into its synchronized period.

Unfortunately, this solution suffers from a couple different problems. The first is that fast moving projectiles will tunnel through targets on an individual tick without some type of continuous collision detection during each tick. The second is that this type of solution can be very expensive to run on the server. The number of iterations will be related to the length of the rewind period. So if players happen to have higher pings, each projectile shot is going to be more expensive to process. In MWO, there are many projectiles and they travel very fast, so for these reasons and others that I won’t discuss here, this solution doesn’t work well for us.

In order to come up with a non-iterative solution, let’s first make some assumptions to simply the problem. We assume the following about our projectiles: the projectile can be approximated as a point so that we can use line traces to approximate its trajectory, the trajectory of the rewind period can be approximated reasonably well with a straight line, and that projectiles have a constant velocity through the rewind period. In MWO, projectiles use ballistic physics, so they don’t always travel in straight lines, but they are very small relative to Mechs, they travel very fast and do not decelerate while travelling through the air, so these assumption work well for us. We also need to make a few more assumptions about the motion of targets. We assume that the motion of the target between its rewound position and current position on the server can be approximated relatively well using a straight line and that the velocity of the target through this period is also constant. These assumptions may be a bit harder to swallow for some fast paced games but luckily for MWO, Mechs turn, accelerate and decelerate slowly and smoothly, so these assumptions turn out to work well enough for us.

Using these assumptions, we can break the problem down into two parts. In the first part, we want to know if a collision with a target is even possible. If a collision with a target is possible, we then can do a calculation to determine where along the rewind period that collision would occur. In the second part, we use this information to do a normal trace fire rewind with a modified ping value to determine if the hit actually occurs.

To check for the possibility of a hit, we first determine the rewind position of the target based on the player’s current ping and the vector that represents the trajectory of the projectile through the rewind period. This is similar to what we do for a trace fire rewind. What we want to know is if these two objects intersect while travelling along their respective trajectories. The assumptions we made earlier allow us to transform this problem into a single line trace by making the velocity of the projectile relative to a stationary rewound target. This can be seen in the image below.

Using our assumptions, the server can transform the problem on the left to the problem on the right, which is much easier to solve

Solving this problem is equivalent to solving the tunneling problem I mentioned earlier. You can imagine that the projectile travels from its starting position to the end of its rewind period in a single tick – checking if a collision occurs during the motion of these objects between their start and end points is a continuous collision detection test. It is important to note that this check is not the same as a normal line trace. We intentionally ignore the rest of the world and only consider the rewound target and the adjusted trajectory – this is a very important detail. If we don’t ignore the rest of the world, we can erroneously get a result that tells us a collision is not possible when it actually should be. For example, if the player is standing next to a wall, we could end up with false negatives. This is illustrated below.

Possible situation where world geometry would produce an erroneous result for the adjusted trajectory test

Recall that this test only tells us if a collision is possible; it does not guarantee a collision actually occurs. There can be other moving objects or geometry in the way. So, we still want to do some kind of rewind test along the original projectile trajectory, similar to what we do for trace fire weapons. If the first part of the test passes, it gives us exactly the information we need to do this. It tells us that if a collision does occur, it occurs at some percentage along the path from the rewound position to the current position of the target. Since both the projectile and target are moving over the same time period, this percentage is exactly the percentage along the adjusted trajectory the intersection occurs – sometimes called the hit time. We can use the hit time to calculate an adjusted ping value that rewinds the target into this middle position.

Server projectile rewind states: Black line – original firing vector over rewind period. Blue line – adjusted firing vector for continuous collision detection test. Black Mech – the current position of target Mech when projectile was fired. Blue Mech – the rewound position of target Mech based on ping of player. Yellow Mech – the candidate position for final rewind check

This middle position is the rewound position we actually want to perform a trace fire rewind test against. We can perform a normal trace fire rewind check with the adjusted ping value to determine if a hit actually occurs. If a hit occurs, we deal the appropriate damage to the target and avoid spawning the projectile. If no hit occurs, we spawn the projectile into its synchronized period as described previously. In the case of the image above, we can see that in this particular situation a hit does occur.

Conclusion

Lag compensating weapons is just one example of some of the difficult challenges MWO and multiplayer games in general have to deal with and this article only begins to scratch the surface. We haven’t even talked about what to do for explosion effects, tracer effects, the inconsistencies rewinding can cause between lagging and non-lagging players, and how the server can do all this work efficiently for many targets. Aside from lag compensating weapons, there are still many other problems multiplayer games need to deal with including error in client predictions, cheating, bandwidth management, and managing server performance, to name a few. It is easy to take for granted the difficulty and effort that goes into creating what appears to be a simple experience for a multiplayer game. I hope this article sheds some light on how complicated developing a multiplayer game can be and that the next time you are playing your favorite online game, you will appreciate it even more than you already do.