Balancing Data and Design

[This article originally appeared on Game Design Aspect of the Month under the topic of Using Metrics.]

StoryForward NYC offered a fascinating look at the use of metrics in storytelling with the panel, "The Art of Data-Driven Storytelling: Creating Context in a Content-Driven World." Of course, I was fully aware that it was the other "storytelling," the one with marketing focus.  Still, the discussion was eerily similar to ones held at GDC about the importance of metrics in game design.

Metrics give a glimpse into the mind of the consumer and with A/B testing, designers can learn what functions better with an audience.  Often, after analyzing metrics, designers tweak the design.  Metrics is especially illuminating if it turns out if players are quitting the game after failing to complete the goal.  Was the learning curve too difficult?  Without metrics, the game designer basically has playtesting to fine-tune problem areas. Metrics delivers large quantities of data about actual users. Metrics tells you what players actually do, not what they say. An advantage to metrics-driven design is that the effects of changes to design are measured and the response time can be instantaneous.

I remember one story from GDC whereby a game designer tweaked the numbers to make the game harder, but because the metrics showed that the players were unable to advance with the new changes, it was immediately switched back.

I can see why metrics-driven design has appeal, but one problem may be that there's simply too much data collection and not enough knowledge to know which data is pertinent.  Or metrics may not capture the full picture, just like in that parable with the blind men and elephant. Data is only as good as your analysis.

In regards to storytelling, panelist Matt McGowan regarded data as a guidepost.  He said, "Start with what emotion you want to evoke, the story you want to tell, and then look at the data to see how you are going to do that."

That's the approach many game designers take as well. They already have a plan and metrics is just one tool in the toolbox. They don't need to base their entire design on metrics but metrics can inform their choices. In fact, designers can help analysts determine what to measure in a game that would be a good indicator of progress, or engagement, or learning, or whatever the analysts were after. 

Sande Chen is a writer and game designer whose work has spanned 10 years in the industry. Her credits include 1999 IGF winner Terminus, 2007 PC RPG of the Year The Witcher, and Wizard 101. She is one of the founding members of the IGDA Game Design SIG.