NetDevil Talks Merits, Precautions Of Early Focus Testing

When it comes to focus testing, the recent climate has favored evaluating the process closely to ensure it's done at the right time -- beginning focus testing too early can compromise a developer's vision and clutter the pipeline with too much feedback too early. But working on LEGO Universe, NetDevil's bucking the trend -- it's been focus testing the online world for two and a half years, using the same group of 19 kids and families from day one. LEGO Universe lead producer Ryan Seabury favors a broader term for the process -- "consumer testing." And as part of the wider-lens view, he says that who you have conducting the test process makes a big difference. "You need experts that know how to set up the research the right way -- people that know how to ask the right questions," he says. "Just bringing people in... isn't going to be useful. That's one big thing I think a lot of people mess up on." And it depends on the interpretation, too. Others, says Seabury, take the result of focus testing far too literally. "You have to read between the lines and trust your gut," he advises. "Early focus testing, if misinterpreted, can take you in a bad direction." It's an art and not a science, Seabury says. "You can save so much time and money, and so much effort if you just get a reaction to something. Put it in front of your players and it either resonates or it doesn't." Early testing is valuable as a "sanity check" and not an ultimate means to a conclusion, he continues. Developers should go into the process being well aware of what they're trying to measure, too. It's especially challenging for NetDevil focus testing with a group of children. "Kids are brutally honest, but often can't communicate exactly what they mean," Seabury says. "We found a good technique that we used a lot just because it's an online game -- we put kids together and let the moderator walk out of the room. I'm sure that technique could work with any kind of multiplayer experience." And NetDevil and LEGO also balanced out the testing experience by trying different audiences. The LEGO Universe Partners Program is a group of adult LEGO hobbyists that have also been contributing to the development process. "It's actually a vibrant community of thousands of adult fans who spend inordinate amounts of time building amazing things... those guys don't mess around. They're pretty purist," Seabury says. "We've been working with about 50 of them -- and now up to 68, and it's only going to keep expanding. We've brought them out to Denver twice... the first time, it was more about getting them up to speed on the concept and what was going on and hearing a lot of feedback and opening a dialogue, and the next time we brought them back out, they actually got hands-on with the tools and the technology." "Conceptually, we think of them as developers helping us build the game," Seabury adds, stressing the importance of community involvement in the early stages of a multiplayer online game in particular. "You never hear about a community being involved in an MMO besides coming in at the beta test at the end," he says. "Those people that can be the most influential... [are] heavily involved and tell us what their expectations are." If it seems that NetDevil is overcompensating a bit with the two and a half-year focus testing, perhaps they are -- Seabury says his team learned a hard lesson from having virtually no testing for canceled NCSoft-published vehicle-based MMO Auto Assault. "We wanted it really badly... it just never happened, so we did a really informal test." Close to the end of development, he says, about 20 friends and family of the developers came in to check out Auto Assault -- "Mostly, we saw that it was a pretty painful experience to start playing the game," he says. Coming out of that experience, Seabury says NetDevil was up front with LEGO about needing committed quality support -- and LEGO responded by providing its own consumer testing group to support LEGO Universe's development the same way it would with its toy products. "They take it seriously with their normal product anyway... so they were able to hook us up relatively quickly," Seabury says. Of course, not all studios might have the resources of a product-tested, professionally-moderated third-party focus group, but Seabury says anyone can benefit from testing early if their goals are clear and their instincts are good. "It's about asking the right question at the right time," he says.