Planning and Directing Motion Capture for Games

Motion capture is a great tool for creating animation for certain types of games. Like any tool or piece of software, if you learn how to use it properly, it can make your life easier and produce great results. If you try to wing it, chances are you'll end up wasting time and money and may come away with nothing useful. Since motion capture happens to be a relatively expensive tool, it's worth spending a substantial amount of time up front planning your shoot and visualizing your end results. The first question to ask is whether motion capture is right for your project. It's most useful for a 3D game with tomes of character animation, assuming you have an appropriate budget and schedule. It depends on the game engine and the style of animation you are looking for. I'm not here to convince you that mo-cap is appropriate for every have or to convert people who believe it's "Satan's rotoscope". But by no means does motion capture make animators unnecessary; in fact, animators are critical to planning the shoot and then turning the data into something useful.

Let's assume that you and your team have already made the decision to use motion capture and are now faced with the job of organizing and producing that aspect of the project. I'm going to explain how to plan the shoot, run the mo-cap session and direct the talent. These guidelines apply to planning and directing any kind of motion capture production, but I'll be referring primarily to optical motion capture. And although I'm going to mention technical issues for your shoot, you will have to discuss the specifics with your team and the motion capture studio.

Now for the purposes of this article, you're a director. You may, be a producer, programmer or animator, but if you're the person in charge of the motion capture production, you've just become the Spielberg or Cameron of your project. I strongly recommend that your team assign one person as the director. This individual bears the responsibility for making the shoot a success and will have an overview of all project issues related to motion capture. The director will be coordinating information from the entire team, from the game designer to the marketing manager. You can hire a freelance director--like me-- but whoever's in charge has to have a thorough understanding of the project and enough time to plan it properly. Ideally, this person should be able to communicate clearly but diplomatically with the talent and with all members of the team.

Planning

Planning a motion capture shoot for a game is very different from planning a shoot for a film or any other linear end product. What's the difference? Your goal is to end up with hundreds of individual moves that connect perfectly to one another. If you've planned other kinds of game animation, you should have a good understanding of this. Of course, the actual production of the notion data is going to involve further considerations. Full-motion video sequences, since they are linear, should be handled separately from the in-game character moves.

Animation List and Flowchart

In order to begin planning your shoot, you need the game spec, including an animation list and flowchart. You'll be revising the animation list and flowchart back and forth with the team as you turn it into a shot list.

Let's say you are producing an action game called Superguy. I'm going to keep this example fairly simple.

Who are all of the characters in the game? This should be defined in the animation list along with the moves that have been planned so far.

Game characters in "Superguy":

Superguy's moves

You should have a separate flow chart for each character.

Look at your mo-cap flowchart and see if your character can easily transition from each move to any other. You'd do this no matter how the animation was being created, but with motion capture, you can't have an animator create the missing move without a re-shoot.

This animation list and flowchart would lead to many questions. For example:

The answers to these questions depend on the game engine and game design. Some animation blending tools will handle transitions, so you don't have to capture them. While it's generally safer to capture more data than you need, naturally you don't want to spend extra time and money shooting totally superfluous moves.

Review the list with the designer and producer to see whether additional moves are required to improve gameplay. Does the player's character have all moves necessary to confront or avoid danger and enemies? Can he/she reasonably travel through every area of the game's environment? While this is a design issue and therefore the responsibility of the designer and producer, it's up to you, the director, to avoid additional mo-cap shoots late in the production schedule.

You'll most likely need to add some transitions to any animation list. For this example, we'll say that your team has some blending tools, but you are going to change the following animations because you need realistic human motion for the transitions:

After you've reviewed and amended the animation lists and flowcharts for each character, you can start planning the shot list. You will also need an estimated frame count for each move. Work closely with the animation team that is going to reduce the motion data to the target size so you understand their procedures. No human being is going to be able to throw a punch in precisely fifteen frames, but you can determine a relative timing strategy.

Shot list

So far, an animation list for motion capture probably seems similar to that of any other type of character animation. Defining the shot list, however, is where you will account for how to shoot the moves in a motion capture studio.

It's helpful to use a database program, such as File Maker Pro, to organize the motion capture information. You can generate a shot list from the database, and later on you and the other team members will be able to produce customized lists needed for post production. You should have separate fields for character name, talent, move names, move descriptions, file names, frame counts, size of capture space, props and special set ups. Also, note whether it's a looping or transitional animation. Any move that repeats is a loop--standing, crouching, walking. Any move that has a defined start and finish is a transition--a punch, a fall, a special move. List the starting and ending positions for all transition moves.

Establish file-naming conventions with your team early in the process. This will allow you to name additional moves later in the process and make sorting the data alot easier.

For example:

SGSA001A: SG (Superguy) S (Standing) A (Attack) 001 (punch forward) A (version)

EVWT001A: EV (Evil Villainess) W (Walking) T (Transition) 001 (to run, left foot) A (version)

HBCL001A: HB (Henchman B) C (Crouching) L (loop) 001 (crouching) A (version)

Your preliminary shot list would have entries like this:

All key members should thoroughly review and agree upon the completed shot list. You must provide detailed written descriptions of each move, but on their own these can still be subject to interpretation. To clarify, create storyboards everyone can refer to. Videotape someone acting out the moves, even if it's no the actual talent for the shoot. You should also have an appendix to the shot list the agreed-upon scene and prop measurements, which correspond to the game environment.

Also include approved sketches of the game characters in costume. Anything that's flowing, like long hair or a coat or cape, is going to need special attention from both the animators and the studio personnel. You may be able to create a special motion capture costume or prop to track the motion of tricky costume elements. When you do a test shoot, you can see if it works properly.

Full Motion Video Sequences

The spec should provide information about all FMV's needed for the game. Usually, the motion capture for these sequences will be done in the last stages of production. You should have final voice-over scripts and detailed storyboards, including camera cuts, in order to create separate shot lists for FMV production.

The timing for these sequences will probably be closer to real time, and you will have to match any planned voice-over dialogue. You may be able to capture more detailed motion data for these sequences, because the whole thing will be pre-rendered and won't be competing for memory with other game elements. Depending on the type of motion capture studio you are using, you may also be able to capture the moves of two or more people at once. When casting your talent, remember that these scenes also tend to require more acting ability than the in-game moves.

Getting Ready for the Shoot

At what point in the development cycle should you start shooting? Of course, your spec, animation list and shot list must be completed and approved first. But don't let the schedule slip. As soon as you have the information for your shot list and your talent is available, do a test shoot and then star shooting, since months of post production will be needed to get the moves into the game. Roughly, in a 14-month development cycle, principal shooting should take place form months 4-8, with pickup shoots possible in later months.

Make a tentative reservation for studio time well in advance. Arrange for a test shoot as soon as it's feasible, preferably with your primary talent. Try to split up your principal motion capture production into at least two sessions spaced several weeks apart. Allow enough time for processing most of the data from the first shoot before the second one takes place, so you can learn from any mistakes and also re-shoot them.

Casting

You've figured out what moves you need to shoot. Now you need talent to perform them. Assuming you (and your marketing department) are not putting up the cash for celebrity talent, you have to hunt down some incredible "motion specialists" that can fluidly and accurately play your game characters.

For a non-sports game, stunt men and women are an excellent choice. (If you are planning to use a stunt coordinator, he or she should be able to bring in people for auditions.) Experienced stunt performers know how to throw a punch, hit their marks, fall on a mat and repeat the whole thing over and over exactly the same way. Martial artists and gymnasts are other possibilities for fighting games. Check your local academies. For sports talent, try local colleges and training center with a good reputation in your sport.

Hold auditions and record them on video (this will help you get all necessary signoffs on casting decisions later). Look for performers who move fluidly and have no unwanted idiosyncrasies in their motion (like a bad knee). Motion capture technology will very accurately reflect any unusual human movement, and this will seem magnified in a game where the same ten frames are repeated often. For example, if you motion capture someone with a bad knee, the character may appear to run more like a loping zombie than an all-star baseball player. You should also make sure the performer has roughly the same proportions as the game character. Height and frame are more important than weight--the actor's "skeleton" (i.e., the distance between joints) will be used to calculate the character's motion.

Another important casting consideration in the performer's attitude. Tell the prospective talent the bad news up front: body-clinging motion capture suit, unwieldy sensored props, repetitive moves, and the need to hit precise rest frames and marks. In exchange for these somewhat bizarre conditions, plan to pay your performers fairly and treat them likely royalty. Make sure you choose people who are not only enthusiastic but also intelligent enough to understand what they're getting into. They should also have flexible schedules and be available for pickup shoots later on. And of course, always have additional choices available as backup.

Remember that you can cast one person to play several characters. For our imaginary "Superguy" shoot, I'd probably cast three performers: a medium build man as "Superguy" and "Medium Henchman", a bigger guy as "Big Bad Boss" and "Large Henchman", and a medium-build woman as "Evil Villaness" and "Small Henchman". Also, find out from you technical team if motion from one performer can be applied to characters played primarily by other people.

Casting Celebrity Talent

One of the most common perceptions of motion capture hell is that of a shoot with a prima donna celebrity who could care less about your silly video game. After suffering through an agonizing and embarrassing capture session, the data turns out to be useless. Don't let this happen to you! The first step is making sure the performer is right for your project, even if he or she is a professional athlete that your marketing department is hot to sign up.

Ideally, you would give a short list of acceptable celebrity performers for your project to whoever is going to arrange the deal. You can't hold auditions of your potential talent, but you can study footage of them doing the kind of motion you need (e.g. basketball or fighting). If the talent is going to play more than one game character, try to choose someone who can perform generic moves in addition to his or her own distinctive style. (Would you want to end up with a basketball video game where every character--on every team--plays like Patrick Ewing?)

Before any contracts are signed, provide the talent's agent with a clear explanation of what the talent will have to do--the same "bad news" you would tell non-celebrity performers. If possible, send a copy of the shot list and a videotape of past mo-cap shoots, plus exciting video game footage showing the kind of result you're aiming for.

Scheduling

Before you can schedule your session properly, you should put the shot list in the order you plan to shoot in--logically and efficiently. Group the shots by a) talent; b) size of capture space to be set up; c) special set-ups (e.g. stunt rigs); and d) the logical progression of your moves. That is, establish your rest frame positions and loops before capturing moves that branch off from those positions.

Here is a sample schedule for Superguy shoot (main character):

Day 1: Small Capture Space

Day 2: Small Capture Space

Day 3: Large Capture Space

Talk to the motion capture studio manager about the studio's scheduling procedures. How many hours a day can you shoot? How much time does the crew need to prepare and wrap the studio? How long should you break for lunch, and who will order it? Is overtime a possibility? Usually, a good day's schedule consists of approximately six hours of capture time. It's better not to wear your talent out (at least not until the last day!), and this should allow enough time for the crew to prepare and wrap the studio.

Once you've got a plan, write up detailed schedule for all involved. It's also a good idea to write up "call sheets" for each day, indicating the time you plan to start and finish shooting as well as the talent, type of moves, props and special set-ups required.

Preparations For the Talent

I mentioned before that you should treat the talent--even the non-celebrities--like royalty. Make sure you have accounted for the talent's schedule, breaks needed, meal requirements, and transportation to and from the shoot. Assign the talent a "gopher" during the shoot day; remember the talent may feel imprisoned in a mo-cap suit. And you don't want him or her to try taking it off in the middle of the day. In other words, provide for (almost) their every need.

Why? sure, it will make them feel special and they'll like you better. (And believe me, that matters.) And being "spoiled" will also help ensure that the performer's full attention is on the shoot. But a more important benefit is that it will give you, the director, more control over the session. Fewer surprise requests and delays equals a more productive shoot.

Procedures During the Shoot

Depending on the studio and type of motion capture, it may not be easy to play back each take of the actual data or review the footage afterwards to select the best ones. To make life easier, set up a regular video camera as a "slate camera" to tape the session. This video camera should have time code synched to the tapes of data being recorded. Hold up a slate board for each take, noting the file name, move name and take number.

To document rest frame positions, take Polaroids and also trace outlines on acetate taped to a monitor showing the slate camera's output. Take notes and make preliminary selects on your shot list as you go.

Ideally, a few key members (lead programmer, animator, producer) are present during the shoot, schedule permitting. You'll want input from them as well as from the motion capture studio personnel during the shoot. However, have a meeting prior to the shoot to discuss how you and the team will communicate in the studio. Above all, establish how decisions will be made. You want to avoid arguing with other team members during the shoot about whether the move you just captured was good enough. This will waste time and make the talent quite uncomfortable. Make it clear that only one person--the director--will communicate directly with the talent on the set, explaining each shot and giving feedback. The director must also be in control of the session's progress: calling action and cut, allowing breaks, and deciding when to go on to the next move.

Directing the Talent

Your first job as a director is to put the talent at ease. He or she is going to feel a bit awkward in that skintight motion capture suit with sensors attached everywhere. Make small talk. Make jokes. Let the talent warm up and get used to the mo-cap suit and any props that will be used.

Treat the performer with respect--don't condescend and don't immediately assume her or she won't understand what you want done. Explain the importance of hitting rest frames and marks, but make sure the performer is relaxed and moving naturally. Explain the shot list and, if appropriate, ask for the talent's input. However, maintain your control as the director. Thank the performer for all suggestions, and do use the ones that will help. But emphasize that mo-cap and game requirements must take precedence over "authentic" sports, fighting and acting considerations.

Take notes on your shot list and mark down which takes you like best during the shoot. You will also have to watch for technical considerations: continuity, timing, rests frame positions, distances, contact points, and so on. But try not to get bogged down to the point where you are treating the performer like a robot. remember to keep an overall sense of the game's character and help the actor bring the role to life. Throw in some humorous idle moves or will improvisations if you can spare the studio time. The performer will probably move quite naturally, being relieved of hitting his or her marks. Often those last-minute ideas work well to add some spice to the game characters.

Directing celebrity talent is not much different than working with other performers. If the performer tries to pull a prima donna act, appeal to his or her ego. Say something like, "We knew this would be hard, that's why we needed you!" Keep the phone number to the celebrity's agent close at hand. (Don't threaten the performer. Let the agent do that.) You can't force a celebrity to give you a good performance, so try to establish a good rapport and talk about the great video game you are creating.

Special Set-ups and Stunts

If you are going to put performer at any risk of injury, you need a stunt coordinator. A good rule of thumb is that if there is going to be any kind of contact involved--between the performer and an object, another performer, or the floor--you should have an expert around to make sure things are done safely. Don't skimp on this. Not only could your performer get maimed and your company sued for millions of dollars, but your shoot will fall behind schedule.

Stunt coordinators are also needed to set up special rigs, like a flying harness. A ratchet is another interesting set-up, which propels the performer violently backwards in a harness to simulate getting blown away by whatever powerful weapons you've designed for your game.

You can also use rigs to simulate non-human motion. Put a performer on stilts or outfit them with extra motion capture arms and legs. Test an actor's ability by making them crawl around on their arms and knees imitating an alien monster that barks. You get the idea.

Wrapping the Shoot

After each shoot, you'll have to collect your notes and slate videotapes to review. After you've selected which of all your fantastic takes is the best for each move, provide the take number and time code in and out points to the motion capture processing team. After the data is process, keep in touch with the animators to make sure they understand what you had in mind. As the motion data is implemented in the game, continue to ask the team for feedback while you are still able to re-shoot anything missing.

When you finally wrap your very last re-shoot, have a blow-out party for your cast and crew. It's hard work, but your game characters are going to look great and be a blast to play. With any luck, you'll have had some fun in the process, too.

Melianthe Kines can be reached at [email protected].