GDC Online 2011 Speaker Spotlight: Kabam's Hitchens On Metrics-Driven Design

Author: by Staff

As the first in a series of interviews with speakers from this October's GDC Online in Austin, Kabam's VP of product and platform services, Sheridan Hitchens, outlines the benefits of using metrics to guide design decisions for persistent online games. By analyzing data gathered from its existing games and players, Hitchens says that Kabam can quickly and objectively identify problems and opportunities that arise, allowing the studio to update and develop its games more efficiently. Prior to joining Kabam (developer of popular social games such as Kingdoms of Camelot and Edgeworld), Hitchens spent four years at casual game developer and publisher PlayFirst, where he oversaw the growth of the company's community, multiplayer, and microtranscation platforms. Before entering the game industry, he graduated from the University of Cambridge and earned an M.B.A. from the University of California, Berkeley - Walter A. Haas School of Business. Here, Hitchens provides some insight into Kabam's metrics-focused approach to game development, and offers a glimpse at the topics he will cover in his upcoming GDC Online session, "Data Driven: How Creating a Deeply Analytical Approach Drives Success." Why does Kabam embrace metrics so heavily? What benefits does this strategy offer? I think as much as anything it's a cultural norm that's driven from the top; our CEO, COO, CMO, and Chief Product Officer all are very analytical. When you make a pitch, or provide a recommendation, you're expected to have some level of data and analysis to back it up. But there's more at work than cultural or executive biases. In our view, metrics provide an objective view of performance across a range of functions. Carefully tracking a variety of metrics, enables us to establish measures of success, and alerts us to problems and opportunities quickly. Perhaps just as importantly, it reduces the amount of subjective, "I think/you think" debate that wastes time in meetings and in running a business. What sort of data do you look at when making internal decisions at the company? We tend to look at a wide range of data and not just the common acquisition, retention and monetization metrics from marketing and production. We'll spend time looking at customer satisfaction data, ticket volume, even tech ops data to overlay onto our common game metrics to understand problems and identify opportunities. We also conduct research to understand the continually shifting market; we survey our own players, survey the market, analyze market growth trends and so on. What tools does Kabam use to track this data? How are these tools helpful? Business Intelligence is something we believe is a core capability for us, and something we think drives competitive advantage. So, we made a decision last year that we needed to build our own tools, going it alone rather than using an off-the-shelf tool. Doing so allows us to iterate very quickly on new features, new metrics and data sources; we treat our BI tool like a full product, and we run sprints and push weekly. The tool we've built gives us a huge range of data on an hourly basis, it alerts us to issues, and it also allows us to benchmark our own games against each other. Just as importantly it provides a really easy interface for any of our producers and marketers to download data so they're able to do their own more detailed analysis more easily and make decisions more quickly that can impact the business. Are there any instances in which you find metrics to be misleading? How can you be certain the data will always drive you toward the right decision? I think there are lots of times when metrics can be misleading, but it's often because you're only looking at one metric and not seeing how it relates to other metrics. You'll see people express joy that they've driven down their cost per user by 30% which in the absence of any other data is good. Meanwhile they've failed to notice that new user monetization has dropped by 50% at the same time. Not good. I'm not sure you can ever be certain that data alone will drive you to the right decision; you try to blend data from multiple sources against your own past experience, business insights, strategy and instincts. Invariably this process will drive you towards a better decision than one made on gut alone. And when you're really unsure, our response is often, "Okay, we'll just A/B test it." How does a development studio remain creative when driven primarily by data? How do you innovate when looking primarily at feedback? This assumes that the two are mutually exclusive, and I'm not sure I entirely agree with that. It's not that creativity and analytic capability are not opposite ends of the same axis. Personally, I think as a society, that idea is driven by the fact that we try and pigeon-hole people as either good at arts or good at science, and I simply don't buy that. Creativity has an important place at Kabam, and we employ a variety of methods to inspire innovation and quality in our game designs, features and mechanics. After all, games are about fun. Our producers spend a lot of time thinking up wonderful new features, but they also spend time trying to understand the impact of those features, and then properly testing and measuring them once they're in the game. How will your GDC Online talk address social game metrics, and what do you hope attendees will take away from it? I'll obviously talk about some of the right ways to look at metrics, but a lot of that has been covered in other presentations. So a lot of my talk will be about how you go about building the capabilities, organizationally and culturally, to take a more analytical approach. In summary, I hope the attendees will see that building an analytically focused company is so much more than just looking at some metrics and pretty charts on a regular basis -- it takes organizational commitment and belief that properly used, an analytical approach can be a key building block for success in the evolving games industry. Additional Info In the weeks leading up to GDC Online, show organizers will continue to debut new interviews with some of the event's most notable speakers, in addition to new lectures and panels from the event's numerous tracks and Summits. Taking place Monday, October 10 through Thursday, October 13, 2011 at the Austin Convention Center in Austin, Texas, GDC Online continues as the leading worldwide event dedicated solely to discussing the development and business trends surrounding connected games -- including casual titles, MMOs, virtual worlds, and social networking games. For more information on GDC Online as the event takes shape, please visit the official GDC Online website, or subscribe to updates from the new GDC Online-specific news page via Twitter, Facebook, or RSS. GDC Online is owned and operated by UBM TechWeb, as is this website.