Manager In A Strange Land: Benchmarking, Part 1

I mentioned in my last column that learning from experience sucks. Most of learning from experience is learning from mistakes; learning from mistakes means you have to make the mistake at least once. It would be nice if we didn't have to make the mistake at all. But there's another problem with learning from mistakes: there are way too many mistakes you can make when developing games. I used to work at a game company where, every several months, the president would give us a speech that went something like this: "Well, we haven't been doing too good lately. We made a lot of mistakes, but I think we learned a lot from them, and we won't make those same mistakes again." No, we made all new mistakes.

What would be really nice is if we could prevent some of these mistakes from happening in the first place.

That's where benchmarking comes in. Benchmarking is looking at exemplary companies and adopting their best practices.

Sounds good so far. Here's where this article gets sketchy. I'm about to break every rule of good statistical analysis and still try to draw some conclusions. I've gone through almost every postmortem in Gamasutra, and listed the ones that got a score higher than 70 (Why 70? Because it's the mean.) on gamerankings.com, along with how long they took to develop, how many people were on the team, and their gamerankings score.

Some of the obvious problems with this approach are: scores on gamerankings are not a valid measure of game quality. (But these scores are a better measure than game sales.) Also, the numbers different people report for their team sizes use different methods: some report averages, some report total number of people who ever worked on the project, some report peaks. Also, the sample is not representative.

The problem is, we're starved for data, so I'm willing to take what I can get, in the hopes that some information that might be right is better than no information at all. So, like with all the articles I write, use it to stimulate your thinking, but don't take it as gospel.

For a while I succumbed to the temptation to divide the "buck" score by the "bang" score, but "buck" is nonlinear. Any formula I came up with to normalize "buck" could be adjusted to make almost any game on this list the highest bang-for-buck of them all. It's better to just stare at the data and try to get an intuitive feel for it.

So whose practices should we follow? That depends on what our situation is. For example, if I was a new developer with a small budget, I'd want to look at what people managed to accomplish with small budgets.

Suppose we arbitrarily choose 85 as the score we'd like to hit. The games that got 85 or higher, and took the fewest developer-months to make, are Unreal Tournament, Rogue Squadron II, Heretic II, and Rainbow Six.

And what do we learn from them?

The key thing is to look at advice from people in a similar situation to you, because the techniques of a small developer won't necessarily scale to a large one. Management techniques that work for a small team (let's get everyone in a room and hash it out) don't scale to a large one. (We don't even have a meeting room big enough for our team.) A developer who has four years might benefit greatly from writing their own engine from scratch. A developer who has two years might benefit greatly from a prototyping phase.

So suppose we wanted to make the best original game possible, and not spend a fortune? Here the winners are a totally different crowd: Thief, Deus Ex, and No One Lives Forever. What can we learn from these guys?

All of these suggestions sounded good up until the last one, huh? Like I said, use this technique to stimulate your thinking rather than override it.

______________________________________________________