Respawn says "single line" of Apex Legends code was behind elusive audio bug

Apex Legends developer Respawn found that a single line of code was the root cause of an elusive audio bug that arose following the launch of Season 16.

Breaking down the issue on the Apex subreddit, the dev team said they clocked the problem after recognizing that grenades would sometimes damage players without appearing to actually explode. It noted the issue didn't occur during Season 16 playtesting and also couldn't be replicated internally (at least, not initially).

At first, the team wondered if the system Apex uses to dispatch "stop" / "start" commands for various effects was to blame, and explained that because the game's servers simulate entities (such as weapons, abilities, and loot) they will emit sound and visual effects.

"Every server frame compiles an effects list of a maximum of 128 entries–any additional effects above the limit were getting dropped. This list is sent to any players who need the effects for the specific server frame," wrote the team.

"From there, the theory was that something may be flooding this engine limitation, requesting thousands of effects every second! But was this a systemic issue or could it be a single entity acting up? Every season update comprises thousands of changes to assets, code, script, and levels. Which meant finding a needle in a haystack."

Metrics would usually help a developer find the root cause of an issue at this stage, but Respawn said its telemetry data wasn't helpful since it "didn't indicate any flags or issues in the system."

"This left us with a complex issue that we knew was impacting our community, but was hard to reproduce despite detailed reports, had minimal leads internally, and there were no metrics to prove definitively that this limit was being hit at all," continued Respawn.

Cracking the code

After investigating the various limitations of the system, the team was able to reproduce the problem in artificial situations–such as by getting a squad of 50 legends to all fire their weapon simultaneously. That provided concrete evidence that FX would get dropped, but only during "completely unrealistic" test cases.

As the investigation progressed, Respawn said it kept a close eye on new reports spotlighting the issue, and found that it seemed to occur more frequently during "high-level play." With a fresh subset of games to scan, the team deployed a server update to add some metrics that would bring in more data about its server effect networking, enabling the studio to continue narrowing its focus. As that update was finalising, the eureka moment finally occurred.

"A single line of code was identified to be the root cause of the issue. Season 16's new weapon. The Nemesis has a particle effect that ramps up when heating up, but when it's uncharged we don’t need to waste resources playing it as the effects are concealed inside of the weapon. So, when the weapon gets updated we would simply stop this particle effect if the weapon had no charge," said Respawn.

"Every time the server simulates an input from the weapon's owner, this particular line of code is executed. Players send in their inputs for every single frame that is run on their client, and it’s the server’s job to simulate all of these inputs. This means that every single player with an uncharged Nemesis would create a 'stop particle' 'effect' on the server every frame, and this line of code was being called even when the weapon was holstered."

Respawn said that "immediately explained" why the issue was occurring more often at high-level play, because it directly correlated to the frame rate of each client that had a Nemesis. "14 clients with a Nemesis running at 180FPS would be enough to cause FX to begin being dropped," it added.

That also explains why the bug wasn't spotted during internal testing, because the builds used for testing might not had have enough holstered Nemesis in play, may have had a rarer correlation with missing FX, or might simply have been missing enough clients at that FPS.

The dev team pledged to look at improving its testing protocols in the future, but also pointed out that "a minute of players playing Apex is the equivalent of 10 testers playing the game for a year," which explains why rare bugs often slip through the net.

For more information on how Respawn identified and fixed the issue, check out the dev team's full explainer on Reddit.