The benefits and practice of harnessing players as a QA resource: lessons from open source software and EVE Online.

Brief introduction: establishing new QA practice?

Current QA practice tends to involve players predominantly in beta testing, with some productions getting players to test alpha builds as well. Players are seen as either sheer numbers to stress-test new game features, or as bug hunters of doubtful reliability. Nevertheless, far more beneficial applications of players' skills and knowledge can be found beyond simple toolkit approach. Their aim is to save developers' time, resources, and organizational effort, as well as to bestow significant competitive advantages.

Changing industrial landscape

The embracing of players' communities as a source of creativity and expertise, as well as acknowledging their skills in game-making, is gaining importance in the increasingly networked digital games market (Banks and Potts, 2010). Players are no longer simple consumers of games but also creators - becoming more vocal, better connected (both among themselves and to the firm) and better equipped to criticize, modify, and even make games. They are increasingly empowered to act upon their emotional attachment to the games, which should be seen by the industry as game development and testing resource (Burger-Helmchen and Cohendet, 2011).

Figure 1. EVE Online's spaceships and academic papers do battle - a metaphor for this article's roots in both industrial case study, and academic observations. Original image source: http://community.eveonline.com/news/dev-blogs/the-bloodbath-of-b-r5rb/

One of the most visible examples of co-creation of games is players’ involvement in quality assurance (QA). Studios embracing their players' inputs in QA are seen to benefit significantly from this process (although it is not free of risks or problems; Banks, 2009). Nevertheless, open source software development research and empirical data demonstrate, that there are many diverse ways of involving players in QA, and that this process can be deployed to compliment developers' role not only in final testing, but also during the iterative cycles of game development (for example embodied by SCRUM and AGILE practices). What is more, the research shows that players’ inputs to QA can be detrimental to studios (i.e. diminishing market response or slowing down product improvement), and thus the process of involving players must be conducted with the understanding of its hidden dynamics (O’Hern et al., 2011) and with an eye for studio’s market environment and competitive circumstances.

Hoyer et al. (2010) already demonstrated some of the advantages arising from embracing players' inputs in game development – which include not only reduction in organizational costs, but also strengthening the relationships with players and creating games better matching players’ needs. In the GDC 2014, we will explore how studios access those advantages, as well as what are the processes and mechanisms that they deploy (both internally as well as in their players relations management) to successfully integrate the resource ’crowd’ with their QA practices. Academic research also delves into techniques and methods of managing those without either causing disruptions to internal game development cycles, or antagonizing communities of players (for example in Malaby, 2009).

Practice of involving players in QA

The research was conducted over the period of ten months in 2013 and 2014, by studying the case of CCP, the makers of EVE Online. The framework of understanding players’ contributions to QA as being either idea- or solution-centric (proposed by O’Hern et al., 2011, stemming from observations of OSS development) was adopted to understand various types of players’ inputs to internal studio’s operations. Since there are numerous parallels between OSS development and empowered role of players in co-creation of games (MMO games in particular, characterized by strong fan following and good relations between the studio and players), this categorization can also be applied to the games industry. Three variables (Figure 2) influencing the success of the integration of players’ inputs with studio’s QA were identified: type of input (either idea-centric or solution-centric), type of channel (for instance public test server, volunteer bug hunters recruited from among players, game forums and in-game chat), as well as type of player (their skills in areas of game development, their motivations, knowledge of the game systems as well as overall engagement in the game). Concurrently, two constants were identified, which influence studio’s propensity to involve its players in QA and the ease at which it processes their inputs (constants because they result from wider business context, broader than any single QA project). Those constants are ‘organization’ (reflecting such studio characteristics as its philosophy, organizational structure, and project management techniques), and ‘game as product’ (reflecting the stage of game’s development, age, and role of players as determined by game design and genre). Those variables and constants together influence studio’s success with involving players in QA.

Figure 2. Table showing three intra-QA variables, as well as two extra-QA constants, together with comments as well as the actions that studios can take to improve their harnessing of players as a resource in QA.

As observed in the case of CCP, three actions can be taken by studios to enhance the harnessing of their players as a resource in QA. Those are to solicit right types of inputs when they are due via the right types of channels, separate idea- and solution-centric inputs of players into QA, as well as to incentivize players to engage in that type of QA activity, for which they have skills and motivation. The challenges to such an approach, as well as its benefits to the firm, will be discussed in the presentation of this research at GDC 2014.

Summary

All in all, it is vital to learn how game studios can accommodate this new and growing relationship and presence of players in game production, without causing disruptions to their normal development practices, or without compromising the control over their vision for a game or independence as a studio. Game developers who learn those skills first, will have a significant advantage over their competitors. This work sets out to assist in that endeavour - to inform industrial practice about characteristics, opportunities, as well as pitfalls of involving players in game development, and to demonstrate how to utilize those practices to their full potential.

Shameless plug: "Harnessing your players as a QA resource - benefits and practice" talk at GDC 2014

Based on extensive research, in this talk we will explore those advanced QA approaches and offer practical advice on how and why to implement them. We will also discuss their risks and benefits, and consider their outcomes for game development and marketing. Moreover, the session will provide practical advice on implementing innovative QA practices involving harnessing players' technical skills and creative ability. Risks, key opportunities and organizational challenges will be explored. By analyzing players' motivations for helping studios, as well as the dynamics of their communities, the talk will inform 'open QA' practices.

The session will be taking place on Monday 17th of March at 3.35 pm, in room 2004, West Hall. 

List of references

Banks, J., (2009). Co-creative expertise: Auran Games and Fury – a case study. Media International Australia: incorporating Culture and Policy, 130 (February), pp. 77-89. 

Banks, J., and Potts, J., (2010). Co-creating games: a co-evolutionary analysis. New Media and Society, 12(2), pp. 252-270. 

Burger-Helmchen, T., and Cohendet, P., (2011). User communities and social software in the video game industry. Long Range Planning, Vol. 44, pp. 317-343.

Hoyer, W.D., Chandy, R., Dorotic, M., Krafft, M., and Singh, S.S., (2010). Consumer Co-creation in New Product Development. Journal of Service Research, 13(3), pp. 283-296.

Malaby, T.M., (2009). Making Virtual Worlds: Linden Lab and Second Life. Ithaca and London: Cornell University Press. 

O’Hern, M.S., Rindfleisch, A., Antia, K.D., and Schweidel, D.A., (2011). The impact of user-generated content on product innovation. Available at SSRN: http://ssrn.com/abstract=1843250 or http://dx.doi.org/10.2139/ssrn.1843250 ;