Conversations From GDC Europe: Bill Fulton, Zeno Colaco, Harvey Smith

Editor's Note: This article is the second in a series of interviews and round-ups from the GDC Europe, which was held August 27-29 in London, UK. In this second dispatch, John McLean-Foreman interviews another three developers who spoke at the event: Bill Fulton, Zeno Colaco and Harvey Smith.

Bill Fulton (Microsoft)
Session Title: "Getting Data that Improves Games: A Case Study of Halo"
Bill Fulton was a founder of the User-Testing Group for Microsoft Games, which uses psychological research methods to get feedback that improves the usability and fun of games published by Microsoft. Since 1998, the group has tested 12,000+ gamers playing 80+ different games, including most Microsoft titles like Age of Kings, Combat Flight Sim, Halo and Rallisport Challenge. Prior to working at Microsoft, Bill did four years of post-graduate training in cognitive & quantitative psychology at the University of Washington, studying how people form judgments and make decisions, and how to meaningfully quantify theoretical ideas.

What does "usability" mean to you?

Usability work is essentially an attempt to see how consumers respond to the game prior to you [the developer] finishing the game, and buying the game and, , so you have a chance to revise prior to shipping. The basic ideas you want from the productivity world is that you want your version 1 to really feel like a version 2.

We use two major research methods to get information from people. The devil is completely in the details. It's all about how you get the right information from the right people at the right time. So, something comes online, and you have the time and schedule to actually fix it. You figure out who that segment of the game is being targeted towards. In the case of a tutorial, you would bring novices to the game, not experts of the genre, because you're not trying to teach experts. If, however, you were dealing with an advanced feature of the game, you'd bring in experts because they're the people that are going to find the advanced features and are going to want them to be fun.

Why would a development team need psychologists? Why not have the designers watching the game testers instead?

For the same reason that you wouldn't hire me to go and do your art for you. Doing research with humans is extremely tricky. Anyone who as ever filled out a questionnaire and seen a dumb question, or seen through to what the answer is desired to be, knows how easy and transparent it is to figure out what people are trying to get at. Often, if they know what you're trying to get at, they will either intentionally tell you what you want to hear (I'm sure that anyone who has been in a focus group has seen that), or sometimes they will purposefully tell you what you don't want to hear to antagonise you. You don't always know which is which. So, psychology has worked out a lot of methods on how you get information from people, how people tend to respond in research situations, how to know when they are suspicious or when they are going to give you incorrect answers and what you can do to minimize it. A risk is being run that people are going to behave in a way that doesn't reveal their true feelings and thoughts.

Some companies can't afford to set up a psychology department. Is there a book that developers can study so that they can get more accurate results from their own tests?

That's a very difficult one -- the books that I know of are designed for academics, not necessarily for lay people. I think that the best introduction to thinking about the effects of design, as opposed to game usability (for which there are not books yet), is Donald Norman's The Design of Everyday Things. Basically it's a book that looks at bad designs in common tools, like the telephone, light switches, and teapots. The people who designed them failed to think about the user's tasks and what the user's decisions are. My favorite example is doors. They're terrible. Half the time you don't know whether you should push or pull a door. You sometimes don't know which side the hinges are on when they give you the long bar across the middle. It's because the people designing those doors didn't think about how people would really use them. This book is more for planting the seed of thought, "How am I doing that with my design? How am I failing to remember the kind of people that I'm trying to suit?"

Another book on the subject is be Jakob Nielsen's Usability Engineering. He's considered one of the biggest figures in the field, but he's very academically oriented and very productivity oriented. I don't think that there are good texts for games specifically.

One of my colleagues wrote a chapter for a usability handbook, which is basically what we consider the opening shot to the game usability field. He's written a methodology of how and why it's different, how it's interesting and important, and you can access it on our website:

http://www.microsoft.com/playtest/Publications/User%20Centered%20Game%20Design.doc

We're trying to start the usability field into thinking about games.

Do you feel that usability studies will make a better game and increase its sales?

Do I believe it makes a better game? Absolutely. We can show people unable to do something in one case, like in the A-1 level [which was supposed represent enemy difficulty levels and read "A.I.", but testers thought it said A-1 and didn't know what that meant], and all people succeeding after the design is fixed. We have very good evidence of how our changes actually affect consumers more so than virtually any other discipline.

As far as tracking sales, that's the perpetual question. I don't have a good answer, but we are beginning to examine it now. We now have enough data and bandwidth to look at the relationship, but quite frankly I think that the correlation is going to be muddled at best. The only thing I can say is that I find it interesting that people ask us to justify usability because they view it as an additional cost. But how do you determine how much improvement your product will realize by adding an artist, or another tester? No discipline knows how adding one more person or doing something slightly differently is really going to affect the bottom line. You use your best intuition, you use your judgement and you say, "I believe that adds value."

After your speech I overheard someone saying that through usability you're not designing something new and creative, but are in fact designing games for the "lowest common denominator and idiots".

I think anyone who describes a consumer as an idiot should perhaps not be in the games publishing business. They should make games that they enjoy playing, and there's nothing wrong with that. Lots of people take up hobbies that they enjoy. But, if you're in the business of making games for other people, then one ought to attempt to find out what other people like. To me, it's very basic. Even if you throw away the profit motive, as an artist you want lots of people to understand and enjoy your work. Unless you believe they're too stupid to understand, say, modern art -- then therefore modern artists are making art for modern artists, and they don't care. As long as that's fine for them then that's great. If they don't make something that can appeal to someone like me who is uneducated, then I will be disinterested in their work. I assume that most people's egos would prefer a million people to love their game, to have said, "I spent 50 bucks and I'm proud to have done so because it was fun." as opposed to, "Oh… those are for other people."

Zeno Colaço (Sony Computer Entertainment Europe)
Session Title: "From Pitch to Publish: Getting the Deal"
In his current role as Vice President, Publisher & Developer Relations at SCEE, Colaço is responsible for the day-to-day management of SCEE's European licensee community that has over 250 active developers and 40 publishers on the PS2 alone. This includes licensee relations, product planning and the development of strategic partnerships. Colaço and his team are a focal point for emerging talent and new businesses looking to break into interactive games.

What was your talk about?

My talk tried to give some guidance to developers about what I think publishers want in the next big games, and tried to coach developers on how to pitch their title to publishers. Granted, that's slightly outside my general remit, being Vice President of Sony's third party division, [which] really looks after the platform edge.

How often do publishers try to look beyond the presentation to see what value is underneath?

We at Sony don't believe that we're necessarily the best ones to tell people how to write games. The talent and the creativity are the people themselves, and what they want to bring to the table. I think it is important and necessary to treat game development as a business. Whether it's the developers themselves employing someone who's got some exposure to that [part of the industry] to represent them, or even a third party, it's becoming more important. Tried and tested methods are there. It's about hooking up to one of those and trying to deliver it.

A lot of publishers are risk averse and as a result, they're putting the same game on the shelves again and again. Doesn't it makes sense to take risks at least part of the time?

I completely agree. As you probably saw in the panel, I felt it was something that needed to be said. Risk aversion is one thing, but "no new I.P.s" (Intellectual Properties) is going to lead to the death of great creativity. One thing that we do as a platform, and I'm sure publishers are looking at it as well, is that we want to try and help risky products. Something that pushes the genres on our platform, we tend to try and give a better voice to. We support many of these titles when they are coming out, sometimes even with a direct marketing contribution from the platform to the publisher involved.

I think developers should try to use and leverage the platform companies in the process of pitching. For instance, perhaps you're working on a title that is slightly different and is pushing a particular genre -- say, it's a music product. That is a smaller genre, but it might expand the platform's capabilities for some of the consumers. Clearly a music product is probably not going to do the same sales as a football product, but if that product were brought as a concept to my team, they would be happy to give feedback on how to maximize and broaden that genre. That meeting in turn could be used as leverage with a publisher who could get into early discussions with the platform company to say, "If we brought this to your platform, would you support us? Would you give us exposure in your catalogues, or maybe in your in-machine demo?" Or even, as I said, to request support for the marketing costs.

It's almost like incubating creativity in that way. But the developer has to [take the initiative] in the first place. The developer has to show some sort of originality to both platform [and publisher]. I think sometimes developers go straight to the publisher and miss out on the platform holder, and they then lose any leverage that the platform holder might have been able to give them.

Harvey Smith (Ion Storm Austin)
Session Title: "Systemic Level Design for Emergent Gameplay"
Harvey Smith is the project director on Ion Storm's DX2: Invisible War. He held the lead designer position on Deus Ex. Prior to that he worked as lead designer of FireTeam (an Internet squad game developed by Multitude, Inc.), and at Origin Systems, as a designer, associate producer and quality assurance tester on such games as System Shock, Cybermage, and the CD verson of Ultima VIII: Pagan.

What was your talk about at the GDCE 2002?

My talk was about what I call "systemic level design". After working on Deus Ex and seeing the contrast in the two different ways that we worked on levels (one was more systemic, one was more special case), it drove me to want to talk about when one was the right answer versus when the other one was the right answer.

How do you define "emergent gameplay"?

We talk a lot about game interactions on a class-to-class basis. When the player does one thing in the environment, and it causes an interaction between two other game elements that provide a second order of consequence, that maybe the development team didn't predict. [The game] behaves in a rational way that surprises the player. We call that emergent gameplay.

The example I used in my speech was an enemy unit that explodes when it dies, and a bunch of containers that the player could open with explosive resources or lock picking resources. Players figured out that if they lead that enemy unit over to those containers before delivering the deathblow, that the resulting explosion would open the container. We call that emergent gameplay.

In your session you mentioned that Deus Ex players created ladders using the removable proximity grenades and climbed off the map. How do you then make the player finish the level and go where they're supposed to?

Ultimately our game is still a combination of special case moments and systemic moments. We still say, "Get to the exit" or, "Go talk to this guy" or, "Go do X". How you do accomplish those goals is more or less up to you, however. We reward you for goal completion, not goal completion methodology.

Will designing levels like this give players an experience they've never had before?

The sense of possibility that players talk about when they play Grand Theft Auto 3 -- or any game that tries to make a really in-depth environment where the world is not just full of one-to-one interactions that were prescripted -- is a very powerful thing. It's empowering to the player. The gameplay itself becomes largely about exploring the possibility space. So yeah, we think it's pretty innovative. As game worlds get more complex, keeping this angle going will mean that every game experience will be richer and richer.