The New ESRB Rating Algorithm

The news media or whatever reporting on the ESRB's new "automated"/"machine" rating system is  confusing and probably inaccurate.  But I don't think that it is really their fault.  The press release by the ESRB is a bit cloudy, hyped, and open to mis-interpretation.

Having now done several years of empirical research on violence in video games, I think that I should comment on the new and old ESRB rating system as I understand it.  You can read about one of our studies in the Laboratory for Automation Psychology and Decision Processes.

The new process involves an online questionnaire developed by the ESRB. At the time of this writing, I have not seen the questionnaire, so I don't how long it is, what the questions are, the options, etc. 

I can only speculate about its contents and hope that the ESRB hired some really good professionals develop it using psychometric methods.  Whatever, some of the structural properties of the questionnaire can be inferred.  I should also note that I have more than 10 years of research experience with online questionnaires, 30 years with psychometric methods and statistics.

So as I understand it, the game developer assigns one of it's employees or perhaps a team familiar with the new game to fill out the ESRB questionnaire.  The answers to the questionnaire are run through a program an aggregates them and spits out a categorical rating (e.g., "E", "T", "M") and assignee descriptors (e.g., "Mild Cartoon Violence", "Strong Language", etc.). 

To know how this works, you've got to know the algorithm (formula) that they have developed and will be using. The formula, hopefully, was the result of analyzing a large sample of games with given ESRB ratings. They would have had individuals familiar with the games answer the questionnaire that they developed. Then they used a statistical clustering analysis, such as Discriminant Analysis, to approximate the best fitting formula.
 
Now here are the problems: (1) The ratings will be no accurate than previous ratings since the previous ratings were used to derive the formula. (2) Game developers can easily outwit or circumvent the system by figuring out how to fill out the questionnaire to get the rating that they want.

Ultimately, the ESRB has pretty much handed over the job of rating games to the developers and abdicated their responsibility in the name of efficiency and cost savings. But what else can they do with the number of games that are being produced every year.

The ESRB promises to check up on the ratings by having a human evaluate the game after the rating has been posted and the game released.  The result is that the backlog in the human rating process will no longer delay the game rating and hold up the release of the game.  Rather it will delay any corrective action.  Below is my diagram of the old and new timeline as I understand it.

Timeline of ESRB Process

 Developers will probably want to sample these testers for the answers to the ESRB's questionnaire. Thus, they can access the wisdom of the crowd rather than one person filling out the questionnaire. But there can be is a problem.  There is a strong correlation between type of gamer and acceptance of violence in a game. Heavy gamers tend to rate more violent acts as acceptable than do soccer moms and senators.  So it is crucial to have a representative range of people filling out the questionnaire.

A second advantage of the new system is that after releasing the game, feedback from the community can be used to refine the algorithm.  Essentially it can shift over time to accommodate the changing moral code of the society and perhaps even for specific societies and regions.  Thus, if different states were to enact different rules for rating video games, the ESRB could do it.

So we shall see just how this rating game plays out!