Interview with Dmitri Williams, CEO of NinjaMetrics

[Cross-posted at Motivate.Play.]

Amongst video game scholars, perhaps no one has been more prolific and had more impact than Dr. Dmitri Williams. Williams, a professor of communications at USC and CEO of game analytics firm NinjaMetrics, has fostered close ties with the game industry, allowing him to obtain and analyze huge data sets for his academic work. He's covered diverse issues in dozens of papers, from correcting stereotypes about gamers to detecting unwanted gold farmers. His collaborations with social and computer scientists as they pored over data from Everquest II resulted in the founding of the Virtual World Observatory (VWO). Some 80 scholarly papers on myriad topics have since been written and published by the observatory.

Now Williams is leveraging his experience and that of his colleagues into an analytics product for game companies. I’ve had the opportunity to watch the NinjaMetrics product develop over the past 18 months, and it’s an awesome example of how a deft, judicious combination of social and computer scientific theory and methods can lead to great analytics and business intelligence.

The following is an interview I conducted with Dmitri on November 13^th. We talked about the features of NinjaMetrics product, the unique team he has assembled at NinjaMetrics, and his long-term plans for the company. [Readers may also be interested in a previous interview with Williams, conducted by Motivate.Play.'s co-founder, Travis Ross.]

Isaac Knowles: I've seen the NinjaMetrics product presented twice in the past two years, and I have to say it's one of the most impressive tools for game analytics on the market. However, there are a lot of great companies in this space. What sets NinjaMetrics apart?

Dimitri Williams: There are a lot of really good companies and smart people working on these issues, so much so that what I call basic metrics have been completely commodified. If you want to know your k-factor or DAU or your retention rate, that’s not a tough thing to deliver, and it’s not super-valuable [to provide] as a result. There have always been free services like flurry, as well as analytics firms who will do it for little or nothing, and use it to sell advertising or to upsell other more advanced services.

The next level up is predictive analytics. NinjaMetrics has an automated predictive analytic engine with a machine learning system on the backend. It’s as high-end as you can get. It learns and tunes on its own, and it predicts churn, conversion, spending, [and other important KPIs]. It’s cutting edge. But it’s only a matter of time until other people have something like it. There’s nothing proprietary about the system and eventually others will catch up to us.

The thing we do that’s different from everyone else is social value. Using NinjaMetrics proprietary algorithms, we have the ability to measure the influence of people on each other in dollars. This is immensely important, because it accounts for between 10 and 40% of all players spending and decision-making. I’m eager to get our work out there.

IK: Talk more about social value. As I understand it, you are using information about a player's social influence on others in order to assign them a value that goes beyond their own spending.

DW: What we've done at NinjaMetrics is leverage information about social networks in games in order to measure the social value of players. We can show customers how much of a player's value is coming from their own personal spend, how much their spending is the result of the social influence of other players, and how much their social influence is affecting the spending of those other players. Because of this, we can obtain a player's asocial LTV [lifetime value], which measures how much the product or service, alone, is driving spending. If you add to that how much a person's spend is influenced by others, you get that individual’s personal spend. If you add to that how much that player influences other people to spend, you get their impact on the system, and that gives you the opportunity cost of keeping that player in the game.

We've built this social value into our analytics, so we can provide our customers with a suite of awesome tools for marketing, community management, and acquisition. You want people with high social values because they're going to drive spending. Our tool allows you to leverage that.

IK: How would I use this tool, for example, to better understand the impact of a player's decision to churn out of the game?

DW: If a person is, say, 90% likely to quit in the next week, you might wonder how much is really at stake. Now, you used to think that if the player's spend was  $75, then that was how valuable they were to your game. With our tool, you might find that - once you include social value - the person is worth closer to $125. So you might make a different decision about what to do about this person.

We jokingly call this a fruit-basket analysis: When do you send this guy who's about to quit a fruit basket? Sure, personal spend might not be so high, but if he goes, maybe his six buddies who spend $400 decide to go, too. Or, if he doesn't go, maybe he and his buddies would pay more if you gave him something to make him and his friends even happier. Maybe you give him a piece of content to share with those six people, and then maybe there’s a ripple effect out because he’s an influencer on those six people.

IK: Can players have a negative social value?

DW: Yes, social value can be negative and we counsel our customers to look at it in relation to asocial LTV. If the end user has an LTV of  $50 and a Social Value of -$25, then they are still a net positive. However, if they can be isolated from the other users without reducing their LTV, then they should be.

In rare cases there are end users with a negative social value so large that it outweighs their LTV, making them a net negative on the system. Our customers can get creative with this. Rather than, say, just banning them, they can sequester them, put them with each other (I call this "troll pooling"), give them a different experience, etc.

IK: What about recommending changes in a game? Do you offer that service to customers?

DW: We're not quite ready to do that yet [without A/B testing]. Within a year, we'll have enough customers that we can provide new and existing ones with recommendations for good promotions within their game's genre. In the longer term, we'll be able to use the patterns we've seen before to leverage our automated system to make those suggestions to customers. Everyone in our customer base is going to benefit from that because we’re going to be able to share insights and knowledge. That’s what we did in our former lives as faculty, where we explained and disseminated information.

IK: I understand that a lot of this work came from the Virtual World Observatory (VWO).

DW: The basic ideas of how to data mine virtual worlds and how to understand player psychology - these are things we’ve documented extensively over the years, and the VWO has over 80 publications written by 25 researchers from several universities. There’s a ton of knowledge that’s been built up about why players do what they do, how game economies work, why people role play, how their transportation patterns match the real world, and so on. But it was really the big data work that we started doing for government agencies that allowed us to incorporate social network analytics into them. That’s different.

IK: What government agencies did you work with?

DW: We did a project for IARPA [Intelligence Advanced Research Projects Activity]. They wanted us to build predictive models using data from Second Life and Everquest. To do that, we invented the field of combined social and computer science for gaming. There was no method; we had to invent it and figure out best practices. And that’s as much about figuring out how to make a team work as it is about the math, because you’ve got awesome tools in computer science, and awesome questions and theories in social science, but those are two things which are absolutely silo-ed off from each other in business and the academy.

IK: What was it like, trying to get those two paths to cross and coming up with new techniques between them?

DW: That was hard! We spent the better part of five years and a fair amount of money figuring out how to do that well, and we’ve documented that in our reports on that project.  The VWO team works that way and that DNA is very much a part of NinjaMetrics. It’s all about getting social science theories and questions pointed the right way, setting them off with big data algorithms, and iterating back and forth. You know, when you go to a PhD program in communication or psychology, you’re taught [statistical modeling techniques like] linear and logistic regression. They don’t teach you how to interpret decision trees models... but they should! Which is what we’re finding. So my team does those things now at Ninja Metrics.

IK: Did you have any issues or feel any pressure compromising the integrity of social science? Or were you able to seamlessly take your attitudes as scientists and integrate them into the NinjaMetrics product in order to make it successful?

DW: It's not a question of seamless integration, nor of bending. There’s a third way. That is to realize that - and this is really helpful with dealing with clients at our business – we’re just talking about a different set of questions. Those questions are driven by people’s personalities and types. Computer scientists are more left brain than right. Social scientists teeter over more to the right side. Their brains and their personalities and their cultures drive them to be interested in different types of questions. Neither is better than the other, and in fact if you can get them to work together you get chocolate plus peanut butter, right? You get two things that you wouldn’t think could go together, but can!

Now, how does this work at NinjaMetrics? The social scientists want know 'why?' But the computer scientists and engineers might not always care about 'why'; they just can't relate to the question most of the time. What an engineer wants to know is "How? I don't care about why. I want to do it more efficiently, faster, stronger, better." What I've done is tied the 'why' and the 'how' together to get 'what', and that's the NinjaMetrics product. For example, the main algorithm that our team uses for our social analytics was created by Nishith Pathak. Now, he’s so brilliant in this space it’s not even funny. But he’s not going to come up with that algorithm unless he was working with my team. I can get the team to focus on the product as the ultimate outcome, and that's where the left and the right brain get tied together.

IK: So is that your real value-added at NinjaMetrics?

DW: What I do is serve as a translator between arcane sets of knowledge for people in the real world who don’t care, and shouldn't care, how that knowledge got there. I can understand and speak the language of computer science with its data models, and I can understand the language of social science with its p-values and publications. I can walk in both of those worlds, and I understand ideas like 'actionability' and 'stakeholder groups', things which my academic colleagues might think of as irrelevant, or selling out. But these are just different cultures, and to be able to be a translator between those cultures is very valuable.

IK: As you know, some companies like King and Zynga are both famous and infamous for their use of analytics to drive design, customer segmentation, and pricing. The development community seems negative, or at best ambivalent, about this so-called "metrics-driven design", the argument being that applying analytics to games robs them of their soul, and/or incentivizes the creation of so-called "Skinner boxes". How do you respond to this concern as CEO of NinjaMetrics? What about as a games scholar? What about as a games player?

DW: My response to 100% data-driven design is negative, and that's as a CEO, as a professor, and as a player. Analytics are data points and rarely explain context. They give developers information so they can make a smart decision. If a developer somehow forgets about their users, their game, the context of play, etc., they're going to make bad decisions. This isn't an either-or moment. It's a "both". Decisions made without data are by definition uninformed. Decisions made without understanding are by definition uninformed as well.

IK: What's your long-term vision for NinjaMetrics? Will you stay exclusively in the video game analytics space, or will you expand beyond that?

DW: I love gaming; I understand it. We’re not here randomly. We’re here because we have passion for and experience with this medium. I think there are plenty of marketing and analytics dollars for us to acquire and I think we have a competitive enough product and it will do well and succeed. However, it’s a relatively small pie compared to banking, e-commerce, or gambling. If you look at the money from legalized gambling worldwide, it’s about ten times the revenue of the video game industry, and our tech is a pretty easy port from one to the other.

You’ve got to start somewhere and we’re starting in a place where we have passion, but ultimately our algorithm and our system is quite generic. I can take a campaign management tool to our system, strip out all the game developer tools that sit there, and then I can give anyone a very small, focused product that nobody else can offer, and I can do it pretty quickly at a low price. So, we will move out there. How we roll that out – whether it's through partnerships or productizing it or letting people hit our APIs... there’s a lot of things left to work out. But the potential is so big that we would be silly not to go explore it... once we’re good in the game analytics space.

[Cross-posted at Motivate.Play.]