Why eSports Tournaments Make Great Venues for Research

The introduction of eSports to the "mainstream" is one of the many interesting developments of the tech revolution, and it is a particularly important one to youth.  It should not be news to anyone at this point that international viewership of eSports has been expanding rapidly for the past five years, and hit some pretty impressive numbers in 2015.  Big names are getting involved as well, with eSports recently getting a section on ESPN and the world eGames running alongside the Olympics.  As more money pours into the industry, viewership attracts advertisers, and more schools offer scholarships, we will see youth react the same way as they do to other competitive games rich in fame and glory: intense and focused practice driven by a dream.  

We at foundry10 have conducted a large amount of research on Dota 2, one of the most played (and viewed) eSports.  Starting in 2013, we have been attending the major Dota 2 annual tournament “The International” (TI) that the developer Valve Software puts on for players.  At these, we have been interviewing professionals, placing players in front of eye trackers, monitoring player growth, conducting on-site experiments, and gathering skill-related data to better understand the cognitive impact of skill development in Dota 2. Although we have been publishing our findings on skill acquisition and cognition one of the most surprising findings was just how novel and useful it is to run studies of gamers at large gaming tournaments. We believe that eSports and MOBAs specifically present interesting opportunities to study learning, skill growth and human interaction.

Just to get you pumped, here is a preview video we put together for the talk we gave at the conference from human-computer interaction, CHI 2016:

More on that study below.

Why eSports?

We feel an important first step is to share some of our results from our work with Dota.  We still have data to analyze, but over the past three TI tournaments we have learned a great deal that we think is of value to the larger scientific community investigating games.

Our work started in 2013 with a series of interviews with pro players.  We were really interested in hearing from the pros what they thought made them good at what they did and whether or not they thought those same skills would benefit them elsewhere.  Their responses really emphasized a few key things: the type of practice they do, where they focus their attention, and their ability to interact effectively within a team environment (a lot of games research focuses on single player games).  You can find a detailed write-up here.

The interviews drove us towards some interesting questions about how players develop skill, but we decided to hone in on visual attention specifically, and in 2014 we returned to recruit Dota fans to play a match in front of an eye tracker.  We noted differences in how experts scan the game-screen, assess information, automate processes to free up cognitive load, and think strategically and abstractly about the game.  The paper on this study is currently being published, so please keep an eye on our publications page and/or Twitter to get it when it is available.

The initial findings helped us to refine our questions even further and in 2015 we attended with a set of computers, a larger booth, and an experiment to run in the venue itself while the tournament was happening.  You can read the study itself here, but one of our biggest findings was just the sheer power of doing research at an event like TI, and by extension eSports tournaments in general, for learning from games.

As a new method of generating useful data on video games, competition, team play, and so much more, there simply is no better place than an eSports tournament.

Opportunity

During TI5, we had lines of up to three hours of avid, hardcore fans all waiting to participate in our study, which was amazing given the fact that there was professional Dota going on at the same time in a building next door.  Part of this, we are sure, was the exclusive Sven t-shirt (and later Dota 2 Plushies) that we offered as incentives, but there was a genuine interest among the event attendees to be a part of growing our understanding of competitive games and cognition.

This was a big eye-opener for us: by bringing the study itself to the venue, we were able to tap into a number of big opportunities and player skill levels that we would only have available at a major eSports event.

So many gamers

To start off, the sheer number of hardcore-caliber gamers was nearly unprecedented.  All-told, at the weeklong TI5 final rounds, we were able to run this study with just under 400 subjects.  These sorts of numbers of gamers are very hard to pull at nearly any other venue.  In fact, in most studies that involve gamers, the sample size is usually around 30 participants.

Furthermore, most studies classify gamers as people that play 5+ hours per week.  However, at TI5, we had many players who play at least that amount that every day.  In fact, the median lifetime total hours played of Dota 2 was around 2300 hours, which comes out to roughly 15 hours per week since launch (assuming consistency since July 2013).  This is an entirely different category of gamers, and one that research has struggled to tap into both in terms of finding them and getting them to go anywhere in large numbers.  Both of these problems are effectively solved by eSports competitions.

While still fairly homogenous, the sample we studied at TI5 was more diverse than we had expected.  Of our nearly 400 participants, 360 were male and 34 were female, and you can see our ethnographic breakdown below:

Breadth and Depth?!

While we are on the topic of the wide variety of gamers who come to these tournaments, they also run the gamut in terms of skill.  The ranking we used for skill was a particular Match Making Rating, or MMR, which takes into account a number of factors to ensure that you are ranked with players of similar skill.  Here the distribution of MMRs:

Look at that nice bell curve.  This distribution really shows just how great of an opportunity these tournaments present in terms of reaching a strong sample of the Dota 2 player population.  Being able to reach this level of skilled individuals really lends a lot of depth to the study.

The presence of professional players at these events also offers some interesting opportunities.  Participating in a study, even after an elimination, is a tough sell to pros during a tournament, so having some kind of agreement with the teams beforehand is advisable.  Even without the pros, though, we were able to still get a strong sampling across the skill band, and had some participants with comparable MMR to professionals.

International Appeal

One of the most interesting parts of eSports is the mass appeal to international markets.  All around the globe, people are tuning in to professional gaming.  Even four years ago, in 2012, eSports events were pulling in greater viewer numbers than the Rose Bowl.  The refinement of competitive games, accessibility of the internet, and the introduction of streaming have created a perfect storm for eSports to take off at an international level.  

This was very evident at all of the events we attended, but we were really able to capitalize on it when we brought our experiment directly to the venue in 2015.   Our other approaches used the tournaments to recruit players to come by the office at a later date.

Challenges

While eSports tournaments hold many opportunities for researchers, getting into the events and having a successful study is a sizable challenge.  During our work at TI6, we encountered a number of potential hangups and tested a few methods of getting around them.  

Getting in the door

The biggest challenge we come up against in looking to run studies at these events is simply getting a spot to occupy.  Tournament organizers are understandably wary of anything that may take up precious space in a venue.  External vendor areas can be costly to obtain, and space in the venue is generally limited at best.  So how do researchers get into these events with limited resources and connections into the venue?

Our work with Dota has luckily been supported by good connections with Valve Software, but other attempts by us and others to run studies at non-Valve tournaments have encountered these types of hurdles.  One option we are considering for getting around this looking into nearby, related events.  These can be run as accessories by the tournament organizers (a la Fireside Gathering during Blizzcon 2015), or as fully third-party, nearby gatherings like barcrafts.  It is possible that there is more space in these sorts of add-ons, and we are planning to explore these possibilities in future studies.

Obviously if you can get into a developer-sanctioned event of any kind, be it the tournament itself or some accessory, this would be the preferable place.  The tournaments often bring in thousands of players who will be out and about during the hours before and after play.  Events like Barcrafts are not just appealing to visitors, they also pull a lot of players who did not shell out the money for the tournament, so the opportunity continues to grow.

If you do wind up at a non-sanctioned accessory event it should go without saying that you will want to be within walking distance of the venue.  

Swag you can handle

Properly incentivizing gamers can be a difficult task, since not all of them are motivated by the same factors.  In a game-specific environment like a tournament, though, any sort of related swag is probably going to be a hit.  In many cases, this can function better than monetary rewards since a lot of potential swag has some sort of emotional pull for fans.

At TI6, the only place we offered a tangible incentive for participation, we worked with Valve and WeLoveFine to print one of the t-shirt designs that had been rejected for the event.  This became an “exclusive” shirt you could only get by doing the study.  Predictably, we ran out of shirts very quickly and had to find another solution, which came in the form of Dota 2 Plushies that were left over from the previous TI.  

These cute little dolls pulled in just as many people as the shirts, possibly more because they came in mystery boxes.  The take-away here is that game-specific stuff is the best incentive at an event like an eSport tournament.  

Conclusion

ESports is coming into its own, and is beginning to compete with other major forms of entertainment for the attention of viewers worldwide.  With this comes aspiration and a host of fans who want to participate in this culture.  As more people join this culture and pick these games up, whether professional hopefuls, casual fans, or somewhere in between, it will become increasingly important to learn more about the impacts competitive games have on the brain.  

Tournaments offer a solution for this study, and one that can advance our knowledge of how games affect the mind far quicker than most other methods that are currently in use.  

If you are a researcher, developer, or enthusiast who would like to know more about the work we do with games, what we have found, and what sorts of partnerships we offer, please check us out at foundry10.org or email me at [email protected].