Why Producers Matter (Statistically) More than Designers

Is it the company or the people on the team that determine the success of video games?  And, to the extent that it is the people that determine success, is it the designer or producer role that has the largest impact?  A recent research paper of mine attempts to answer these questions, by studying 854 computer games, 537 individual producers, 739 individual designers, and 395 companies between 1994 and 2006 (representing $4 billion in revenue), using a sophisticated econometric technique called Multiple Membership Cross-Classified Multilevel Modeling.  Controlling for a wide variety of factors, I found that the difference among individual producers accounted for over 22% of the financial success of the game, the differences among designers around 7%, and the rest of the game developer’s people and organization, about 21%.  These results, while surprising, turned out to be extremely robust over many types of analysis – the choice of producer is key to determining a game’s revenue, while the choice of designer matters very little.  Further, these skills are portable, even when producers move from one developer to another

For this analysis, I used the MobyGames database, combined with the work of a team of RAs, to get data on game credits and company success.  The list of games included everything from the top selling games (The Sims, Roller Coaster Tycoon, Starcraft) to smaller and more specialized projects (Immortal Cities, Two Thrones, Battlecruiser 3000AD). I used NPD Funworld’s revenue figures to track success.  (It may be unsurprising to people in the industry that I tried to use GameRankings and MetaCritic ratings as well, but they turned out to be bad statistical measures of game success, though they showed similar results – that the role of producers were at least twice that of designers).  I statistically controlled for the size of the team, the genre of the game, year of release, whether titles were based on licensed property, whether they were sequels, and the publisher of the game.  After accounting for these many differences, I looked at how much of the revenue differences between games could be explained by the designers and producers on the project, and how much by the company.

Compared to the impact of individual designers and producers, the importance of the game developer as a whole was surprisingly small, accounting for 21% of the difference between games.  Especially given that I do not include other important team members in my analysis (such as programmers and artists), it is unclear how significant top managers at game developers actually are in explaining performance, but they are, at most, on the same scale as the role played by just one individual within the product team.   Far from being interchangeable, the individuals involved uniquely contribute to the success or failure of a game. 

Additionally, the relative importance of producers over designers in explaining a game's success is surprising.  Even in an industry that rewards creative and innovative products, the difference between top and bottom designers seems to have much less impact than the difference between top and bottom producers. Rather than acting as cogs in the machine, dwarfed by the importance of top managers, the effect of producers on performance was actually larger than that of organizational factors, implying that differences between producers play an outsized rolee, even over the $4 billion in revenue generated by games in the sample.  Though this finding might seem surprising, it is supported by research on the role of project middle managers in the innovation process.

Recent research on the role of individuals and groups in industries as diverse as consulting and comic books supports a longer literature on project management that has demonstrated the complex interaction between individuals and teams in successful innovation.  The finding that producers have significantly more impact on firm performance than individual designers aligns with this tradition.  It suggests that high-performing designers alone are not enough to generate performance variation; rather, it is the role of individual producers to integrate and coordinate the innovative work of others.

It is worth noting that even though producers seem vitally important to the success of a game, they are not compensated in a way that acknowledges the vast difference between the best and worst producers.  Based on a survey conducted with the attendees of the Game Developers Conference in 2008, I found that individuals reported relatively weak connections between performance and reward in the game industry. Producers, for example, reported bonuses that accounted for less than 9% of their total salary on average, with only 10% of producers reporting either a share of royalties or profits. It may be the case that many producers are not truly aware their own significance, or at least cannot prove it, without the kind of detailed longitudinal data I used in my analysis.  Successes are claimed by many, while the blame for failures is often blurred, or, in the words of the old proverb, “victory has many fathers, defeat is an orphan.”

It is worth returning to the data one last time and noting that the difference in individual ability among producers, and, to a much lesser extent, designers, is one of the most important drivers of a game’s revenue.  A game that would normally generate $30M in revenue could see their sales change by as much as $7M, depending on the quality of the producers involved.  The difference between selecting either the top and bottom designers, however, accounts for a less significant $2M.  And, keeping the same producer and designers, but changing the developer, could result in a difference in revenue of about $6.5M.  You can read more detail in the full paper, People and Process, Suits and Innovators: Individuals and Firm Performance.

This does not mean that designers are not important, or that they do not play a vital role, but it suggests that getting the "best" versus the "worst" design talent has far less impact on the game than getting the best possible producer.

Some additional information (updated, again): Based on the comments I have received, a few additional points.  First off, this is basically the result of a statistical, economic analysis - the results surprised me as well, I was certainly expecting to find a bigger impact on revenue from designers than I did.  The results are not intended to show a lack of respect to designers, nor was the title of the post, and I have attempted to clarify the language in hopes that I do not cause offense.  It is merely the results that I found, and that I am reporting here.   The analysis shows that having the best versus worst designers on a team only changes project performance slightly, while having the best versus worst producers on a team has a tremendous impact on the performance of a game.  I can only offer theories about why this is the case, based on other academic work.  It might be the case that producers have a wider range of talent than designers (the best producers are really good, and the worst producers are really bad, while most designers fall somewhere in the middle), for example.  Or, as is the case with many project managers in creative industries, the producer's role in bridging the gap between all of the stakeholders during development might be far more important than is realized. Any ideas you have to explain the gap would also be appreciated.

As to the source of credits, I used Moby Games, with some spot-checking of the data to ensure accuracy (which seemed high).  I realize credits do not always reflect contributions, but unless there is a reason to suspect that there is severe bias that would cause either designer or producer credits to be particularly inaccurate over 12 years and over 850 games, this shouldn't have a large effect.  There is much more discussion of Moby Games and data collection in the comments below.