Author: by Lennart Nacke

The quality of video game design ultimately depends on a developer's creative vision. However, evaluating how close the final game experience is to the original developer's intent is often hard. Games User Research (GUR) is an emerging field that aims to get the product closer to the developer's intention in terms of the player experience. GUR analyzes the interaction between players and games to get insights from players to improve game design (see here for a Gamasutra feature on what GUR is).

Games user researchers (GURs) use many different methods, tools and techniques to gather information about players and their gameplay experience. This information is used to support design decisions of game developers (see here for another Gamasutra feature on GUR methods).

When conducting user test sessions (either internally or through contractors), applying correct methods and thorough data analysis does not improve anything if the final GUR results are (1) not communicated well, (2) not convincing, or (3) not actionable for the development team.

User Research Reporting

In this feature we are focusing on a little-explored area of GUR: reporting. We will be looking at how user test (UT) findings are communicated among game development teams and how game development can benefit from storytelling and storyboarding techniques (which are common in web development).

We interviewed six game development professionals from midsize UK game design studios to explore storyboarding. We wanted to know how they communicate UT findings, what are the limitations in current approaches, and how these approaches can be improved.

For our developers, the main values of UT sessions are to see: (1) areas of frustration, (2) areas that are difficult to pass (blockers), (3) if the players are having fun, and (4) if players understand the game and are using all the game features. They mentioned that an ideal report would be a process to capture a massive UT data and report it in a way that it is easy to make sense of.

For example, in a previous title worked on by one of the developers (a racing game) they collected game metrics to generate a crash heatmap of each track. They added: "from heatmaps we could see the crashes, but we know they can lead to different experiences. Some of them lead to enjoyment and some lead to frustration; the heatmaps won't show this difference [...] [an ideal report] is somewhere between only seeing the heatmaps and talking to the actual players." They suggested: "for some issues you wouldn't feel a text report could put them in a right context and time line. For example, when interpreting from a text report, there is no way to see the change of pace and enjoyment."

Let's summarize the most important aspects of UT reports identified from these interviews:

The report summary is the section they all read and found most useful. All of our developers think a good UT report should show an at-a-glance-summary of a level.

Location of issues in each level is an important factor for prioritizing fixes for them. Our developers want to see where exactly the issues occurred. For example, they said the "UT report should show me where my good and worst parts are, this can help me to prioritize what to fix. It is more concerning if a negative experience is happening at the beginning of the game, because people can easily drop out there."

Trust is a vital matter for UT reports; if the development team does not trust the results the problems will stay. Our interviews suggested that the most convincing case is when the developers personally attend UT sessions and have a face-to-face conversation with players or watch the gameplay video. They also said "to trust a UT report, it should provide evidence of why something is wrong with the game; only stating the problems won't be enough."

Our developers want UT reports to enable them to make a comparison between the player's experience and the experience they intended to design for. They think it would be great if a report could show an accurate match to their intentional design. They suggested: "if using the UT report as a comparison tool, it has to distinguish between usability, user experience, and pace in the game." For example, the existence of a usability issue would not be intended, and it is not a way for pacing games.

Storytelling

Our examples have shown that UT reports can be very useful to game developers if done right. However, there are many purposes for employing storyboards to convey a narrative behind several aspects of game development, such as gameplay, art, animation, marketing, and the actual game narrative (Joe Gingras from Ubisoft Montreal gave a great talk at MIGS 2012 about finding a shared vision for developers using storyboarding in all these areas).

Before we revisit how storyboarding can improve our UT reports, we will discuss a little bit about the theory of storytelling, as there is a pattern behind great stories. To be entertaining for example, stories need to have the right dramatic cues and tap deep into an audience's collective psyche. (See fig. 1 for an example sketch of a story arc)

Fig 1: Story arc sketch.

On the other side, storytelling is at the heart of the human experience. Once we experience something great, we often cannot wait to tell someone about it. Stories help us shape our perceptions and consolidate our feelings. User experience researchers have leveraged the power of storytelling to drive observation-based and focus group research for improving websites and interface designs (see here and here for more).

For example, in scenario-based design (aka SBD -- see here for more) textual narrative descriptions of an imaginary situation are employed in a variety of ways to guide the development of an interactive system; in video game development the data gathered during UT sessions can make more persuasive stories (or scenarios).

In this feature, we focus on stories that have the goal of describing and communicating player experience aspects to the game development team.

Storytelling is one of the most natural and powerful ways to share information. As part of user experience, stories help designers to put the work in a real context and show design concepts or connect ideas. But more importantly stories help to keep users in the center of the design process. This is critical when developing for video games as the success of the final product directly depends on it being used (played and enjoyed) by users (players). Stories can be a way to keep players a center of game development process, even if they cannot always be part of development team.

Game development includes many disciplines, each with its own perspective, rhetoric, and formalities. By providing examples and a common vocabulary for everyone in a development team, storytelling can bridge these disciplines together and help in building a shared vision and interpretation (or a "visual consensus", as Ubisoft's Joe Gingras called it). We can use stories to gather, share and distribute information about players, tasks and goals (e.g., their motivation for playing a game). Stories can be a powerful tool in game development for encouraging collaboration and innovation of new design ideas across the whole design team (from programmers to publishers).

The limitations of stories -- if they are not data-supported -- can be that they are a personal and subjective account told from a consumer's perspective. Therefore, recording and assessing experiences can have fairly intangible results when they're simply subjective narrative accounts. However, stories can become useful when player stories generated based on data collected during UT sessions.

For example, these data may include player comments, observational notes, gameplay metrics, and biometrics. Analyzing large-scale or high-resolution player data can be daunting, and presenting results from these studies is often not straightforward. Using stories would facilitate understanding the human aspects in these data. These data-supported stories would help game developers to understand and interpret GUR reports better.

Biometric Storyboards

We are promoting the use of storytelling (or more specifically storyboarding) to point out UT findings and their impact on player experience. Storyboards could also visualize the impact of an improved design on players' feelings. For example, we could use storyboards to visualize positive and negative player emotions during gameplay as well as player engagement. Matching UT reports to these observations can provide a powerful overview of game levels and help uncovering game design weaknesses.

We also see storyboarding as a powerful tool for triangulating or combining different data sources, bringing together the power of quantitative data as well as the depth of insight gained from qualitative inquiry and observation. Our storyboarding approach aims to help games user researchers to visualize game design intentions, player experience reports, and physiological responses (biometrics).

We are currently experimenting with the integration of biometrics annotated by player-defined experience points in the game. We call these integrated storyboards that annotate physiological data: "Biometric Storyboards". They can help with visualizing meaningful relationships between design intentions by measuring a change in the player's physiological states (emotion) respective to game events.

The design of Biometric Storyboards went through a number of iterations based on feedback from game developers. In the first iteration we divided each level up by time; however we realized that time is not always meaningful for some games, and "beats" (or thematic areas) were considered more representative. The current iteration, (see fig. 2), is simple to read and understand as it couples behavior (the text along the bottom) with the associated player experience (the line graph).

Fig 2. Example screenshot of Biometric Storyboards prototype. Click for full image.

From these iterations we learnt that: (1) each level should be divided into thematic areas, as this would make the key sections easier to compare; it also could include the time it took the player to complete that area. (2) Green or red dots can be used to pinpoint the moments of positive or negative experience. They are key to providing context and establish cause and effect. (3) It is important to couple behavior (the text along the bottom) with the associated player experience to make the diagram easier to read. (4) The experience graph should go down (negative gradient) to indicate and isolate negative player experiences and to better represent the emotional change.

Storyboarding Player Experience

Visualizing player experience by storyboarding could make difficult-to-interpret GUR data more accessible to a wider game industry audience. Here we point out some key strength of storyboarding approaches for GUR:

Correlation between user research data and gameplay events: Storyboarding would be a powerful tool for tying the GUR findings together, since these findings contain data with a radically different formats (such as qualitative user comments and quantitative game metrics).

For example, in Biometric Storyboards, we use visualization of a player's actions, their biometrics responses, their post-gameplay comments, and gameplay events to help us to better understand and explore correlations between changes in player's feeling and the corresponding events or behaviors.

We saw these storyboards being used to examine behavior of several players during a single game event. Understanding how players are motivated to perform particular tasks in gameplay environments is vital information for game designers.

Comparison of players' behavior: Once we have created a series of these storyboards, we can compare the gameplay journeys of different players and use them to spot key trends in gameplay behavior. Further studies may show the player's background profiles and "psychographics" (motivations) can reflect a regular pattern of behavior and subsequent enjoyment in their corresponding storyboards. Future development of these storyboarding techniques would provide a tool that could be useful to compare gameplay experience in different settings.

Whole session overview: By visualizing the whole gameplay session, storyboarding would help to provide an efficient overview across all events, levels, and missions, enabling the developers to quickly scan for key elements in level design, player performance and player emotions.

Verifying the intended design decisions: We have seen these storyboards being used to compare how players actually felt during game event to what the designer had originally intended players to experience. Storyboards would suitably fit into the design process, enabling designers to verify the success of their game design environment, and judge whether their intended game experience matched the actual player experience.

Simplicity: Storyboards can be formed and iterated based on the demands of game developers to deliver visualizations that are simple, easy to understand and interpret with an immediately apparent benefit.

User-centered Design: GURs have a benefit from understanding of game development process and the relevant needs in the working environment to design visualizations that closely match the requirements and language of target users, and the subsequent level of detail necessary for the task.

Familiarity: Game developers and producers are familiar with storyboards, various data representation techniques and visualizations of game metrics. Similarity between these existing models help to support communication with and between developers and effectively increased the acceptance of new tools.

Support collaboration: Storyboards enable increased collaboration between games user researchers, game designers, other developers, and producers. We saw that producers and designers were able to more effectively discuss design strategy using these storyboards as evidence for player behavior.

Summary

Through storyboarding we can visualize and aggregate player data, this would help games user researchers and games development teams to achieve a shared view on critical game design events. Based on our experience, storyboards are not only a powerful tool to explain game design problems but also provide a way to discuss their solutions. They can help the whole team to visualize design problems, the potential solutions, and gameplay areas that need improvement.

Creating data-driven storyboards supports design arguments, so that game designers can see how players would experience their intended designs. These storyboards provide an analytical connection between players and game designers. With these storyboards we can provide engaging and actionable arguments explaining player experience issues to the games development team.

Game events (we call them "game beats") and emotions resulting from those events are at the heart of creating a great player experience. Biometric Storyboards allow us to visualize events in gameplay where player's actions or behaviors lead to change in their emotional states. We already ran several studies with Biometric Storyboards and hope to be able to present our Biometric Storyboards tool for game and player evaluation soon.

Acknowledgment: We would like to thank Mark Knowles, Jason Avent, Graham McAllister, and our interviewees for their contributions to this work. We would also like to thank Gareth R. White, Steve Bromley, and Kate Howland for their encouragement and constructive comments on earlier versions of this article.

More reading: W. Quesenbery and K. Brooks, "Storytelling for User Experience: Crafting Stories for Better Design, 1st edition," Storytelling for User Experience: Crafting Stories for Better Design, 1st edition, Apr. 2010.