Deprioritizing skill-based matchmaking turned Call of Duty into The Bad Place

Activision Blizzard has published a lengthy white paper that explains what happened when it dialed down skill-based matchmaking (SBMM) in Call of Duty. The TL:DR version? Almost everyone hated it.

The company sought to explore how SBMM impacts players in a variety of ways, which led to it running a 'Deprioritize Skill Test' in Call of Duty: Modern Warfare III earlier this year.

The experiment saw the studio leverage its A/B test framework to loosen constraints on skill in matchmaking. It ran the 2024 test in North America and established a treatment group comprising 50 percent of the population. The other half of players were left unaffected.

Returning player rates went down when skill-based matchmaking was loosened

Those who unknowingly experienced a reduction in SBMM weren't happy. After a month of running the test, Activision categorized the treatment population into 10 equally sized groups—sorted by skill. With SBMM deprioritized, returning player rates were down significantly for 90 percent of players. Only the 10 percent most skilled players came back in increased numbers, but on an aggregate level Activision Blizzard saw "meaningfully fewer players coming back to the game."

"This effect may appear small, but this change was observable within the duration of the test. This will compound over time, just like interest, and will have a meaningful impact on our player population. This is a concern for all players, including the top 10 percent, as if this pattern is allowed to continue, players will exit the game in increased numbers," explained the company.

"Eventually a top 10 percent player will become a top 20 percent player, and eventually a top 30 percent player, until only the very best players remain playing the game. Those original top players will become increasingly likely to not return to the game. Ultimately, this will result in a worse experience for all players, as there will be fewer and fewer players available to play with. Also, as noted above, this test only deprioritized skill in the matching rules. If it were completely removed we would expect to see the player population erode rapidly in the span of a few months."

Players impacted by the Deprioritized Skill Test were also more likely to jump ship mid-game, with Activision Blizzard explaining the quit rate "significantly" increased across 80 percent of players. Only the top 10 percent of players saw a meaningful decrease in quit rates, but Activision Blizzard noted that upside would be fleeting.

"As the accelerated departure of players in the lower skill brackets takes hold, top 10 percent players will eventually drift down the skill distribution (as originally top 10 percent players will make up a larger and larger portion of the player base)," it said. "As a result, we expect to see once top 10 percent players quit games at increasing rates as they become a 50th percentile player after much of the lower skill population has left the game."

Team Death match saw more blowouts

The number of 'Blowouts' in Team Death Match (TDM) also increased. Activision Blizzard explained a blowout is when a team in a lobby "wins with a score delta greater than 30," meaning they have absolutely battered the opposition. It said the number of blowouts increased for all Deprioritized Skill Test players in TDM lobbies, which means everyone was likely having less fun. "We see similar results in other game modes," it added.

Tracking other metrics such as Kills Per Minute (KPM) and Score Per Minute (SPM) across lobbies impacted by the experiment showed that lower skilled players generally performed worse across the board, while the top 10 percent were able to "dominate."

"Kills Per Minute is down significantly for the bottom 20-30 percent of players. The next 60 percent of players have no significant change, and the top 10 percent see significantly higher KPM," stated Activision Blizzard. "Like KPM, SPM follows a similar trend. The low-skill players perform worse, while the top 10 percent can dominate. As with KPM, we expect to see players shift to the left on this distribution over time, as low-skill players return to the game at lower rates.

"The use of killstreaks and increased KPM and SPM shows that the wider lobby skill percentile disparity is disproportionality leveraged by the top 10 percent of players. Unfortunately, this increased performance comes at the cost of much greater impact to the much larger 30 percent of the population toward the bottom of the skill distribution."

Weighing in more broadly, Activision Blizzard said its latest test and historical experiments shows a clear link between SBMM and player retention.

"We can see that loosening skill negatively impacts our ability to keep players interested in our game," it said. "In a test similar to the Deprioritized Skill Test discussed above, we were able to see a significant decrease in the number of players playing Call of Duty: Modern Warfare (2019) and an increase in the overall match quit rate, when treated with a looser skill matching. Subsequent attempts to protect only the bottom 25 percent of players and allow for looser matchmaking for the remaining 75 percent of players also had clear negative indicators of self-reported 'fun.'"

You can read the full white paper on the Activision Blizzard website.