PC Gaming Master Race: Some Empirical Data

By now I'm sure most of you have heard the debate between players of FPS's at one point or another: 

"Dude, if you could play this with a keyboard and mouse, I'd still whoop you." 

...which is invariably followed by:

"Keyboard and mouse is infinitely superior to your dirty controller you mealy peasant."

The art of Saejin Oh, originally made popular on Kotaku

Much to the chagrin of PC gamers everywhere, their majestic throat beards swaying silver in the wind, there has as-yet been very little empirical study on any human interface considerations to add credence to their argument.  Companies like Microsoft and Sony are actively performing User Research on their games to make sure that users have the best experience possible, and this no-doubt extends to their controllers and other hardware peripherals.  

So... does the debate have merit?  Is there any difference between playing a functionally identical game on a console or a PC?  

Let's assume ceteris paribus for a moment - that is, let's assume that other than the control mechanism, a copy of, say, the newest Call of Duty is more or less identical on PC and console.

Before I incur the wrath of the PC gaming elite: I know this isn't exactly true.  It is, however, my strong personal opinion that if more than 80% of a population doesn't notice a difference between two things, the additional 20% probably isn't worth debating.  

I'll take it a step farther and introduce an old friend:    

Newton's Flaming Laser Sword is a philosophical razor that tells us, more or less, what cannot be settled through experiment isn't worth debating.  The time has come, then, for folks debating the differences between PC gaming and Console gaming to put up or shut up.

Unfortunately, there are a plethora of factors that go into making things a little "different" between console and PC versions.  If we're going to go ahead and conduct an experiment, we call these extraneous variables, and they're undesirable.  For this reason, we can't make any wild generalizations, no matter how much data we collect.  This is of course known as the scientific method.  

Without further ado... 

Some colleagues, students, and I created a game in Unity3D from an asset template.  In this game, the player needs to shoot and pop 25 innocuous blue balloons with an ugly yellow toy gun (I work at a university, after all).  

Before playing, players were asked to rate themselves on a scale from strongly disagree to strongly agree (a seven-point Likert Scale) in their evaluation of the following two statements: 

I am confident and competent with video games.

I am confident and competent with First Person Shooters.

By controlling experience level and previous playership (an extraneous variable), we can determine more readily that any potential difference between keyboards and controllers would be due to a difference in the hardware, and not due to difference in experience.  Players also played on the same computers, with the same keyboards and mice, the same Xbox360 controller, in the same room, at approximately the same time every day.  They were also drawn from a static population of 18-30 year-olds.  

Players were instructed on how the controls were mapped, then told to shoot all 25 targets.  The game automagically ended and stored data after the last target was shot.  

Players were randomly selected to use either a keyboard and mouse, an Xbox360 controller, or a third controller that we built - which is the topic of another post.  For the current discussion, we're only going to focus on the data points from the Xbox360 and keyboard and mouse groups.  

The following data were collected using telemetry: 

Runtime
Time Between Targets
Accuracy

Which can of course be compared against...

Overall Proficiency Level
FPS Proficiency Level
Controller vs. Keyboard and Mouse

So, first, let's get the SCIENCE out of the way with a little sanity check... 
Obviously, people who are better at games (e.g. rate themselves higher on the Likert Scale) should be more accurate, and have a lower runtime if the study is well designed, right?  Remember those back-checks your sixth grade math teacher made you do?  Wham.  Science.  

OK, this is good. Looks like as people rate themselves better at FPS's... they are more accurate, significantly, and they take less time, significantly.  For reference, p-value is a measure of how valuable a statistical data point is.  Generally speaking, if something's p-value is smaller than .05, it's considered to be a valid observation and we often reject the null hypothesis.  In this case, it makes sense that our observations found these things to be statistically significant.  

Now on to the juicy stuff: 

The first thing measured is accuracy, on average, between the two control methods. 
 

...Though it pains me to say it, the keyboard and mouse was significantly more accurate than the controller.  The correlation wasn't huge, but there is definitely a significant improvement.  

Here's another one: 

We see here the keyboard and mouse took significantly less time to complete the experiment.  I had to remove one outlier from this data, as someone using the Xbox controller took over 30 minutes to complete a 30 second task and really hosed the analysis.  

Again, on this performance metric, it appears that there is a significant correlation between keyboard and mouse use and improved runtime on a standardized test.  

While these two calculations appear to put the debate to bed, please note that my n is approximately 50, meaning even though these are insanely significant, external validity is somewhat limited.

There is also one other limitation: 
The correlation between FPS proficiency and use of a keyboard is also statistically significant at .01. What this means is that it's probably more likely already that people who play FPS's use a keyboard anyway, so to put this debate to bed, we're probably going to need to test again with different types of games.  
 

This is, for the sake of clarity and completeness, a summary of the average times it took each participant to shoot each target.  Keyboard is in blue, Xbox360 is in red.  (a delta-t chart).  This illustates that both the Xbox360 controller and the keyboard had approximately the same rate of change between targets, but the Xbox360 had more outliers.  That could also taint the data, given enough of them, though corrections reveal it to be unlikely in this case.  The odd spikes could also be endemic to the game design.  One of the later targets required subjects to turn around in order to locate and shoot it - this could explain the large jump toward the end.  

Conclusions?

Hurts me to say so, but it seems like there may be some validity to the PC Gaming Master Race argument at least in terms of FPS games.  I have no data to suggest that the other kajillion types of games are easier to navigate with a keyboard and mouse.  As with everything academic, further study is needed.

What do you think?