GDC Europe: Playfish's Valadares on Intuition Versus Metrics: 'Make Your Own Decisions'

“Brains are horrible for predictability,” said London-based Playfish studio director Jeferson Valadares in his packed session on intuition versus metrics in social games on the first day of GDC Europe in Cologne. Most games in the traditional space are based much more on intuition and experience than they are on user data, but in the social game space, metrics have an importance that makes many developers feel uncomfortably tied to the whims of their users. Citing psychological research, Valadares points out that if you throw a coin 9 times and it turns up heads each time, people will generally presume that the chance it will be heads is 10 percent -- when in fact it’s at least 50 percent. Metrics can help take care of these sorts of decisions that shouldn’t be based on intuition, though there is some element of marketing to it. Restaurant menus, for instance, generally have an anchor price, he points out. It’s the most expensive thing on the menu, and it may be in a prominent place. But by comparison, it makes the second and third most expensive items seem much more reasonable. These metrics are good for optimizing, says Valadares, but that won’t take you to the next level. With metrics, “if you look at the space of the solution for the problem, you’ll reach the top,” he says. “What you might not know is that there may be a better top that you didn’t even consider.” That’s where intuition comes in. “If you want to do things that really innovate and differentiate from what’s happening, you really need guts,” he says. Valadares cited a quote by marketing guru Seth Godin, who wrote: “The analysis, based on past events, certainly seems sound. But your instincts are the only way you’re going to do something unsound.” By way of example, Valadares discussed engagement in Playfish’s Pet Society. When they introduced fishing, there was a big jump in users, but there were no metrics to support the idea that this would be successful – they had to trust the designer. But on the other hand, “some new features, like gardening, brought the game down,” he admitted. “It’s not going to get you good results every time, but it’s the only way to ever get great results.” Metrics For Metrics Valadares believes in intuition, but that it should be used carefully. His own rules for use of intuition are: • Experience is important. You need a solid basis of prior work in order to base your decisions on intuition. • Lean toward it for complex problems, not for details. If you’re deciding whether the border of your HUD should be one color or another, you test that with metrics. Don’t spend your time thinking about it, because there are more valuable things you can think about. • Lean toward it for long-term planning. “Even if you fail once but overall there’s better improvement…if you want to take a long-term view, data can be a bit deceiving at times,” he says. “Just because something happened in the past doesn’t mean it’s going to happen in the future.” • Back up your intuitive decisions with data whenever possible. Then try new things. Metrics are good at things that our brains suck at, he reminds us, like numeric choices. They’re great at optimizing details. Here are some good things about metrics, according to Valadares. • What gets measured gets done. Developers will actually act on things that have metrics, because it’s trackable and quantifiable. • If you have good metrics, it drives good behavior in the company. All discussion revolves around how to improve a part of the game that everyone can see is bad. • Good metrics help you make good decisions. Well-chosen statistics allow you to make very well-informed decisions about your userbase. • Metrics are good for highly-measurable environments, which lead to results very quickly (web environments, for example). Playfish, for instance, does postmortems for new features every week. “If you take three years to figure out if your game is good or not,” he prompts, “maybe you should’ve used metrics.” There are bad things about metrics too though, which closely mirror the first list: • What gets measured gets done -- if you don’t have metrics for something that’s really important, people may not do it. • Bad metrics drive bad behavior in the company, if you’re looking at the wrong things. “A lot of companies have incentives tied to certain metrics, which is fine if you do it right, but it can lead employees to cheat the system in order to increase their numbers,” he says. • Bad metrics help you make bad decisions. If you’re not tracking the right stuff, you can be making decisions in the completely wrong direction. • Metrics are bad for noisy environments with long feedback loops, he says. • Metrics can age quickly. “Three months go by and suddenly [what you’re tracking is] not the thing to watch anymore,” he cautions. “It’s time to start working on some other thing.” Best Practices How should you use metrics then? Valadares says you should align them with your goals, business or otherwise. Don’t just track whatever you can. “If it’s not going to help you get to the next level, there’s no point,” he says. Keep it simple -- chart a few good metrics, and don’t bloat your stats with unnecessary data. “It’s better to keep three to five very important ones that are simple to understand,” he says. At Playfish they had one particular variable, a measure of retention -- “every time I asked people what it was, it took three minutes to explain,” he said. “Now we have another variable which people get easily.“ If people can’t readily explain what the metrics actually chart, they can’t really be used efficiently. And you shouldn’t worry about perfection right off the bat. “It’s better to start with something, then improve,” he says. And then you should continue to think about those metrics and change them if they’re not right. “You really need both,” says Valadares. “There’s a time and a place for everything. And if you get good at using both at the same time, you’re going to have a big advantage.” By way of example, he discussed a new web designer that joined the studio and thought they should get rid of their banners, because in his experience, nobody clicks on those. But Playfish had metrics to prove that they were working. Intuition certainly would indicate that banners aren’t that useful, depending on the market you come from, but metrics can prove this quantifiably. “People tell you one thing and do another,” he reminds us, speaking specifically of the whims of the user. Say there’s a fire in the forums, and everyone seems to hate a new feature, but metrics prove that it added users and people like it. “Sometimes listening to them…it’s always important, but it might not be the best thing for a business,” he says. You’ve got to use both intuition and metrics at the right times, but don’t hide behind either one. “Many times I see people running away from making a decision because of what was done in the past,” he cautions. “Just because you did something in the past and it worked, doesn’t mean it’ll work in the future. It might work, but don’t use that as an excuse to take responsibility away.” As a final word, Valadares had this to say: “Make your own decisions, and don’t let metrics decide what you do.”