Measurement Techniques For Game Designers

"One measurement is worth fifty expert opinions" - Howard Sutherland

Experts and expertise are hard to come by, and experts like us are often unreliable. There are often situations when there is no available expertise, or situations that no one has experienced before. Even worse, there are situations where experts disagree. In these situations, people generally turn to measurement.

Most trades consider measurement more important than expert opinion, or they largely abandoned complete reliance on expertise several hundred years ago. A casual look around you will reveal the extent that measurement is used in society. Food is sold in precise quantities; clothing comes in a series of formalized sizes. Politics and market research are dominated by polls and the analysis of measured data. Architects, engineers and nearly every other kind of designer use measurement and agreed units as one of the foundations of their work.

Measurement in any form must have been originally developed to determine absolute values in situations where there was no independent expertise, or where the experts disagreed. For instance, a measurement of area, using an agreed system of units, is an ideal way to resolve disputes over land. A formal measurement of volume or weight is a perfect way of ensuring fair trade. The use of measurements of qualities like force and mass has been standard in engineering projects for centuries. In short, the use of measurement, agreed systems of units and number are some the cornerstones of human civilization. The very first recorded human writing - Mesopotamian Cuneiform - was initially used to track measurements of traded goods like wheat and oil.

It’s amazing given the importance of measurement in so many trades and other aspects of our life that game design is in such a prehistoric, pre-measurement state. For whatever reason, game designers - working in a technological and logical field, hardly ever use measurement in their work. In fact, I’d wager quite a few people reading this article cannot even comprehend what we could measure and how it would help us. I’ve tried to use measurement as much as I can in my work, and have often found it invaluable.

In this article I’ll deal with the some of overall concepts behind these kinds of analytical processes, and I’ll detail some specific examples that will hopefully help your work right away. Finally I’ll try to examine how this kind of technique and philosophy could potentially mature to help us in the future.

Human Imperfections

As mentioned above, the key reason humans have developed measurement is because people tend disagree on interpretations of objective reality. Our senses are imperfect, having evolved for a simple hunter-gatherer’s existence, and our brains are similarly ill-prepared for minute, consistent and logical examination of things.

A simple example; in my hobby as an amateur racing bicycle builder, I often find myself trying to roughly determine the difference in weight between two components, a few hundred grams apart. Anyone who has tried to do this will agree that it’s almost impossible to judge without resorting to mechanical means, however crude. The mind and body is simply not equipped to make such a precise judgement, and as a result we often make glaring mistakes. This imperfection is what led to the development of a system of units of weight, and measuring equipment for said units – scales. Scales use two technologies - Firstly an agreed system of weights and measures, and secondly machinery to precisely and consistently determine the measurement of those units.
Game designers often find themselves in situations where they are as clueless as I am when comparing the weight of bicycle parts. Examples of this come up all the time when designers discuss games. An example:

GTA games can be pretty frustrating, right? Missions are often poorly balanced, requiring a trial-and-error playing technique, which is rendered even more frustrating by the fact that upon failing a mission, players are required to physically move the character across the map to a specific location to re-trigger the mission (I’m so very glad that the designers of Mercenaries fixed this with a "retry" UI option).

A couple of years ago I had a discussion with a designer about this aspect of GTA, and he made a statement to the effect: "when you fail a mission it takes about ten minutes to re-start it". Now I’m aware that people often exaggerate in these types of statements, so I asked him to try to guess how long it actually takes to get from the hospital or police station in Vice City to an average mission location (they are evenly spaced). He responded – "ok, exaggerations aside, maybe five minutes average".

One of my pet fascinations is the way people find it impossible to properly determine the passage of time, so I decided to go home and measure this value, rather than rely on the expert opinion of this professional game designer. My quick investigations at home came up with an approximate average value of 45 seconds to a minute for the task that was guessed at five minutes. That’s a serious inaccuracy, and one that I think anyone (me included) would be inclined to make.

So it’s clear that our brains are great at doing some things (coming up with creative ideas and ways to solve problems, team work, verbal communication), and not so great at others (measuring the passage of time in free-roaming gangster games, and the differences in weight between expensive bicycle parts). Nearly every other field resorts to measurement in situations where our expert judgement breaks down, or is found lacking. As game designers, we are professionals, and because precision, analysis, and logic are an important part of our work, why not measure?

Applications

In the face of any uncertainty as to what is actually possible to measure in a videogame, let’s look at some specific examples.

1. Time

Time and timing is vital to most games. Action games specifically often succeed or fail based on the precise timings of lower level player actions like melee attacks, weapon reloading, and character acceleration. I think it’s really important to measure your game and the games of competitors to try to find the “sweet spots” in these timings.

Some of you might have read an article I wrote for Develop a couple of years ago in which I claimed to have found the perfect timing for character jumps – at 0.7 seconds between hitting the button and the character landing the jump. I determined this figure by taking several games that were well received from a design point of view, recording the jump times with a stopwatch, and finding the average figure. As far as I know this discovery is still supported by the data, as now I often informally time jumps in new games.

There is huge room for research on this topic. I’m convinced that there are “sweet spot” timings for almost any action in a game. I hope that someday someone will accurately and comprehensively compile this kind of data. They would be doing a huge service to game designers all over the world. We’d no longer have to exercise our expert (or should that be inexpert) opinions because the hard data would be there to override our expertise. We’d have a manual, and the beginnings of a method.

2. Distance

Distance is a critical component, particularly in the 3D action/exploration games that designers love to make. The size of levels, the spacing between events, and the range of ranged and melee weapons are all critical to creating an entertaining experience.

The problem is that it is often impossible to do precise measurements of distance in competitor’s games. Obviously in your own project you can use the tape measure tool in your 3D package/level editor, or refer to data in a config file, but this isn’t possible in a retail game.

One thing you can do in these circumstances is use a stopwatch to measure distance by timing the duration of movements. For instance, it is possible to determine distances fairly precisely in first person shooter maps by measuring the time taken to move across a space, given the consistent and smooth movement of the in-game avatars. A few years ago I used this method as an amateur map maker for a Half-Life mod, Frontline Force. Looking only at the most popular maps, I measured the distances between spawn points and capture points. Spotting a pattern, I applied this "sweet spot" data to my own unique map with immediately positive results. The map "felt" good first time round.

The use of measurement gave me, an inexperienced Frontline Force mapper, an immediate advantage. I was able to concentrate on the creative side of map building and not worry about basic low-level characteristics like the sizes and the distances used in its composition.

3. Density

This is an area that really deserves a whole separate article. It’s really hard to measure density (meaning the density of in-game events) without developing a specific unique method.

You first of all need to define a new unit, because this is an area not covered by existing research. An analogy would be the development in of new terms and units like "Shot", "Scene", and "Mise en Scene" (the latter two borrowed and adapted from theater) which enabled academics to analyze film.

I recently outlined a technique and a proposed unit in an article for Develop Magazine. Put simply, my system involves recording gameplay using a video capture card, and microscopically analyzing that data in a video editing package like Adobe Premiere, dividing and subdividing the data with markers until you reach a level of subdivision which I call "Primary Elements." These elements are basically the absolute simplest, fastest, most minute player actions you can measure – actions like "select unit," "steer 30 degrees left” or "reload." Once you have these elements marked and recorded, the units defined, you can do qualitative and quantitative analysis of them, and of the recorded gameplay as a whole.

The advantage of this system is that measuring things "per Primary Element" as opposed to "per minute" or "per level" is universal across genres. You can try to find patterns emerging in games that on the surface bear no relation to each other. For instance, turn-based strategy games like Civilization and Dance Dance Revolution have very little in common using other measurement systems, but they both contain Primary Elements, so using this system they can be measured in almost the same way.

We may discover in years to come that there is a "sweet spot" of Primary Elements per "level/challenge/encounter," which we can directly apply to level/content design in almost genre. We might find that people get bored of playing games at a "sweet spot" of Primary Elements, irrespective of the actual playing time. I’ve long thought this could be one of the reasons why people play slower-paced games like Civilization for hours and hours longer than action games – we like to see minimum number of "actions" and "reactions" take place in the game before we get bored or mentally tired. It just so happens that this takes five hours in Civ and twenty minutes in FIFA Street.

4. Area

This is one type of measurement that is actually quite fun. Nearly everything we make as developers is eventually rationalized to 2D screen space, so measuring and examining the way your game uses area on the screen is really useful. The method I’ve used to do this involves using an artist’s chinagraph pencil to physically draw on the screen - marking the position and movement areas of in-game objects.

For example, I’ve specifically used this technique to mark and measure the extent of character movement in a third person game (meaning the total area of screen space that the character uses as it moves). First I marked the movement extent of the characters in various retail games, like Mario 64, GTA3, and Maximo. Then, with the marks still on the screen I looked at our in-development game. This method made it clear how our character movement and use of screen space differed from the conventions and standards used in other games, and it also helped us spot mistakes and odd glitches in the way the character and camera moved.

This is a really useful example of a method that insures us against the types of mistakes we make if we rely on solely expertise and personal opinion. Sometimes it’s hard for a designer to judge even the simplest of aspects of a game (like how much the character in the game moves on the screen). With a few simple marks and measurements you have incontrovertible data that has removed the need for opinion and guesswork.

Hopefully you’ve seen that I’ve scratching the surface of what’s possible to measure in a game, and the potential for applying these methods and others to your own work.

The Road Ahead

It’s obvious why measurement is used as a tool by humans, and why it has been so fundamental in the progress of mankind. People are imperfect, but fortunately our cultural developments have included techniques and machines to overcome these imperfections as much as we can. Human society and science are founded on concepts and ideas made possible partly by measurement and agreed systems of units. It is hard to imagine how hard life must have been for mankind before they learned to measure things. Except it couldn’t have been that hard, because that’s how most game designers work today!

Of course it’s easy to adhere to old habits and it’s easy to go along with the flow and follow the inertia of old working methods. Sometimes it’s also nice to feel like an expert and use gut judgement to make decisions. I do it all the time – it’s often the most efficient approach given the highly pressured and panic-stricken circumstances we find ourselves working in. It is however an imperfect and un-optimized way of working.
I feel there is a real need and a real opportunity for game designers to begin to use mature analytical processes that compliment and help us in our opinion-led, gut-reaction working methods. Using measurement and hard data, we can do several important things.

Firstly, we can examine phenomena that are impossible or very difficult to judge with our senses alone (like the timings of mission repetition in GTA, for example), overcoming the inaccuracies and assumptions that we habitually get wrong. Secondly, we can reinforce and qualify our correct expert opinions, by supporting those all-important gut feelings with proper data-gathering investigations (like my jump timing example). Finally, we can also use this data to help persuade others that we need to get certain things done. If you can show someone that you not only think something is true, but that you can ultimately prove with data that this is the case, then you will find yourself in a much stronger position when it comes to arguing for a related task to be carried out.

In an ideal future, measurement and analytical data-gathering techniques will give us a proper academic foundation (a formal, data-driven academia, not the horrible cultural-studies influenced rubbish currently pervading videogame studies). Designers will be able share data in repositories that are accessible to all, just like any other logical and analytical field. Field manuals, dictionaries, encyclopaedias and papers will be published on the subject. People will be employed full-time as academic game analysts and surveyors. Our craft and trade will move from a prehistoric, "feel" focused method, governed by the gut-feelings of disagreeing experts to a mature, modern state, where opinion and expertise are supported or disproved by something close to absolute truth.

References

- “Mind your language - Unlocking the secret formula of game design” Develop magazine, August 2002
- “Elementary game design” Develop magazine, October 2004