Harnessing the power of motion control in video games

Motion control isn't a new option for video game developers. As an industry, we've been fiddling around with capturing player movement to allow for interaction with games for many years now.

Of course, there are been plenty of leaps and bounds in recent years, thanks to devices like Kinect, LeapMotion, PlayStation Move, et al. And with so many motion control devices coming in the near future, plenty of developers are giving the hardware some serious thought.

As part of Gamasutra's Advanced Input and Output week, we talked to a variety of developers who have plenty of experience in the field of motion control, to find out what they believe are the best (and worst) practices when bringing motion control into your game.

Douglas Wilson, Die Gute Fabrik

Chances are you've heard of Douglas Wilson as the designer of the widespread phenomenon that is Johann Sebastian Joust.

Wilson is a key figure in the motion control games movement, especially when it comes to encouraging the sorts of party games that you'd find at the various indie shows and get-togethers that are occurring more and more often around the world right now.

"I'm a little hesitant about recommending 'best practices' or other points that developers 'need' to keep in mind," he tells me. "One, designing motion control games is somewhat different for an indie than it is for, say, a big studio like Harmonix or Ubisoft. Two, I've found that some of the most memorable physical games break with traditional design wisdom in clever and deliberate ways."

Having said that, he believes that the best way for motion control games to evolve and get better is for studios to compare notes and share their experiences -- so he's happy to explain what has gone right for him.

"With my own physical games, I started finding my groove when I started thinking more seriously about playground games, folk games, and sports," he says.

"For example, Johann Sebastian Joust was heavily inspired by a silly (non-digital) slow-motion folk game some friends of friends made up. And our party game B.U.T.T.O.N. was partly inspired by the way some traditional Danish bar games mess with concepts of 'winning' and 'losing.' I see folk games as a kind of 'paper prototyping' of physical and motion control game design. Human beings have been playing physical and gestural games for centuries! There's a lot to learn."

He describes motion control games as the "slapstick comedy" of video games, as these experiences tend to focus on what happens physically in front of the screen, rather than on the screen.

"Marketing rhetoric would have us believe that next-gen physical games and so-called 'natural' interfaces will lead us to new heights of immersion, making us feel like we're actually in the fictional world of the game," he adds. "The reality, however, is that technologies like accelerometers and machine vision algorithms are quite limited. Both as a designer and as a player, I'm more interested in doing stranger, unexpected things with those technologies."

The classic mistake that developers can make when implementing motion control in their games, he says, is to treat the technology in a "binary" way -- for example, requiring that you waggle something to trigger a gameplay action on or off.

"The beauty of physical movement is that it's rich and complex," Wilson notes. "That is to say, physical movement is very 'analog.' Many of the best physical games figure out how to let players move in expressive ways, rather than prescribing exact gestures."

The other issues with tending towards exact gestures is that you are then forced to deal with false positives and negatives.

Says Wilson, "Gesture detection is hard! Consequently, many of the best physical games punt on that problem and embrace a far simpler solution, enlisting (or as I put it, 'deputizing') the players themselves to 'complete' the experience."

Take Wii Tennis for example -- "who cares if you can theoretically play by just flicking your wrist! It's far more fun to 'perform' playing tennis, to go through the full-bodied swinging motions. Wii Tennis does a good job coaxing players to act as such. In that sense, good motion control design is as much about the technology itself as it is selling players on the 'spirit' of the game, on the idea of performance."

So let's say a studio is planning to make a motion control-based game -- at what point should the dev know that a concept is working, and how much fiddling is required to get to that point?

"All of my most successful physical games were immediately fun, even from the very first playtest," answers the J.S. Joust dev. "By contrast, in some of my more failed motion control projects (especially those that used gesture detection), we spent months tweaking the input detection, hoping that one day we could make it fun."

"Of course, the successful games needed tweaking along the way, even after the initial playtests. But it's important to remember that game feel is also so heavily dependent on finding the right visuals, the right audio, and setting the right 'mood' and context."

He adds, "My other advice is to exhibit your game in public, as often as you can! At festivals, in galleries, in the park -- wherever. Watch enough people play your game and you'll start to get a good sense of whether or not it needs more work."

And what if a studio is thinking about adding motion control as more of a side-feature in a game, rather than as the main course?

"To be honest, you might be fighting a losing battle if motion control is viewed as a side-feature," Wilson muses. "All my favorite motion control games are built around the idea of physicality and expressive movement. I don't think it's impossible to employ motion control as a side-feature, but that sure sounds difficult..."

Pohung Chen, Leap Motion

Pohung Chen is Leap Motion's "Games Guy," building video games and other exciting concepts for the hand-and-finger sensor device.

With numerous notable titles now available for the Leap Motion, including motion-controlled versions of Cut the Rope and Fruit Ninja, plus titles like Double Fine's Dropchord and Flow Studio's Midnight, I asked Chen for his own take on what the best and worst practices for motion control are.

"Motion controls are pretty different from traditional methods of digital input," he says. "We are so used to binary actions: keys, buttons, mouse clicks, touch taps, etc., it can be difficult to think of good ways to design for motion control, which by nature is a very analog and continuous thing."

As a result, Chen says that most games do not translate well into motion controls -- when a developer tries to force motion controls onto existing games, it regularly ends up feeling clumsy and a poor experience for players.

"When building a motion control game, it's best to think about the core input interaction first," he continues. "Ask yourself: What is the player trying to do? What is the easiest and most natural way to use your hands, fingers, body, etc. to interact with your game? Make sure the core input interaction is good first before building game mechanics on top of it. If the core interaction sucks, the game will not be good. It'll be like trying to play StarCraft with just an Atari 2600 controller."

And conversely, trying to attach too many discrete actions to a motion control game is a common mistake that Chen sees. "Developers are used to the keyboard, which is really good at distinguishing between a lot of different discrete input (keys)," he says. "Unfortunately, it's very easy for motion control systems to mistake one action for another (if you're trying to discretize your input), so that ends up being a not very good experience for the user."

In terms of haptics technology, Chen notes that the tech available on a consumer level is still rather primitive, given that controller vibration is the only haptics option available to the majority of players.

"Valve is doing new haptics with their Steam Controllers, and there's some research on how to use ultrasonic waves for in-air haptic feedback, but consumer technology is still pretty far out from reaching out and feeling an object or having physical resistance," he notes.

"Because of this limitation, a lot of motion control mappings to real-life analogs with haptic feedback can feel really strange (this is why lightsaber duels won't feel quite right for a while). Since we don't have haptics, visual and audio feedback for what you're doing is incredibly important. This is anything from changing the color of a cursor to indicate a gesture, or lights/shadows to show 3D position of hands/fingers in a virtual scene."

Elsewhere, motion control developers need to keep in mind that connecting real-life movements and analogies up as closely as possible with video game motion controls can often prove very limiting.

"Despite all the generic first person shooter/person in a cockpit style games there are on the Oculus Rift, there's still stuff like Dumpy: Going Elephants (from DePaul University) and SoundSelf (Robin Arnott, Evan Balster)," Chen reasons. "With the Leap Motion Controller, you can use it to push physical stuff around, or you could map properties of the hand to MIDI output and hook it up to Ableton Live to expressively perform music -- the possibilities with new hardware form factors are endless, and I'm really excited to see what kind of creative stuff people come up with in the next couple of years with new hardware."

When it comes to deciding on whether a motion control concept works, Chen suggests that developers should begin by considering a wide range of potential control schemes for an idea, and then eliminate those that don't entirely work until you've left with the best choice.

"Once you find something that works well, and this is largely by feel, one thing that's super useful is to get someone new to try out what you've built," he says. "Don't explain anything, and watch what they try to do. Sometimes you'll get ideas for how to come up with better control schemes by just watching people struggle with what you’ve built."

And for new studios considering motion controls, Chen says that "the best experiences are designed from the ground up with motion control in mind."

"As for side-features or putting in motion controls after a game is already well into development, it is better to focus on doing a few things well than trying to do everything at once," he adds. "I think motion control is still in its infancy. As the technology gets better and more developers experiment with new and interesting models of interaction, we'll start to see some really cool stuff in this space."

Drew Skillman and Patrick Hackett, Double Fine

Double Fine has dabbled plenty in motion control games, with games like Happy Action Theater for the Kinect, and Dropchord for the Leap Motion -- and it's not stopping there.

Drew Skillman and Patrick Hackett head up the recently-formed "Future Tech" department at the studio, and have been playing around with the Oculus Rift, the Razer Hydra Motion controller, and more. The pair tells me that when starting a motion control project, developers should just start throwing out prototypes.

"It's critical to begin building prototypes as quickly as possible using whatever tools you can get your hands on," Skillman says. "That way you can see where the technology shines and where it falls down in minutes or hours, rather than days or weeks. We often use Processing, Unity, or our own internal 'Buddha' engine to get ideas off the ground as quickly as possible."

Rapid prototyping in this way allows studios to explore the capabilities and limitations of motion control hardware more quickly, and find novel ways to use it -- much like what Double Fine regularly does as part of its Amnesia Fortnights.

"Don't just test the controls internally with people on the team," Skillman warns. "Playtesting is always important for game design, but when dealing with new motion control technologies it's absolutely critical."

The main issue is that as you iterate on your initial motion control concept, you begin to tailor them to your own experience, rather than what players in general would enjoy.

"New players have none of that background and don't even know what's possible with the new technologies," he adds. "There's nothing more disheartening than working on a control scheme and convincing yourself it's awesome, only to realize you're using the tech in a way that is utterly lost on the average player."

So how does a developer decide whether the motion controls that they have pieced together are "good enough"? This was be incredibly tricky to pinpoint, admits the Double Fine man.

"Since they don't have the precision of a gamepad or touchscreen, you can't expect to get them to 100 percent reliability," he says. "But if they're not reasonably solid and consistent they will certainly ruin the experience of your game."

He continues, "The key, in our experience, is to aggressively iterate and re-work the control scheme until people essentially stop commenting on it. At that point you'll often notice players begin to have a natural interaction with the motion controls - one that doesn't feel forced or confusing.  We're always putting our games in front of new players, and when they start commenting on game mechanics instead of the motion controls, that's a pretty sure sign that you’re in a good place."

For example, on one of Double Fine's recent PlayStation 4 motion control projects, the team went through four or five massive redesigns of the control scheme over several months before it settled on the final control design.

"When we finally noticed our coworkers picking apart shadow artifacts and subtle game mechanics rather than motion controls, we knew we had found a natural and intuitive solution," notes Skillman.

I ask Skillman how a studio can decide whether motion control is a valuable addition to a game, rather than being thrown in for the sake of it.

"I'd consider the source of the push to include those motion controls," he answers. "If it's an idea that sprung up internally within the team and feels like it fits the game, then it'll probably work and be awesome. Often you'll find new hardware that complements the vision of the game - and in those cases even if it is just a side dish, it can still add a lot to the overall experience."

However, if the momentum to add motion controls to your game is coming from outside pressures, such as a publisher's marketing initiative, and the team isn't actually all that excited about adding it, then it can have a negative impact.

"At best it won't add much, and at worst it could seriously hurt the rest of the game," says Skillman. "However, given enough time and a flexible design, adding motion controls can lead to new experiences and sometimes entire new areas for developers to explore. That's certainly been true for us here at Double Fine, where we've been vocal about our love affair with exciting new technologies like the Leap, Kinect, DualShock 4, and Oculus."

He adds, "As an aside, if any hardware developers are reading this, hook us up! We want to make games for your hardware!"

Jason Alexander, Terminal Reality

Jason Alexander was the designer for the Jedi Mode in Kinect Star Wars, and as such, found that he had to compete with player expectations of fighting with lightsabers.

Although it didn't work out as well as the team was hoping, Alexander says that the base prototype had plenty of promise -- unfortunately, they leaned a little too heavily on "gestures = buttons."

"I think one of the biggest things to keep in mind with motion controls is that there is almost always an inherent degree of subjectivity that comes along with them," he notes.

"'Press the A button' is a predictable input, and pretty much everyone will 'Press the A button' in the exact same way," he continues. "But 'Lean forward quickly to dash'? How fast is 'quickly'? What counts as a 'lean'? Does that include a step forward? Is this a bend at the hips? How far do I have to bend? And so on."

Taking all of this into account early on -- and spending plenty of time researching and testing these ideas -- is key to victory, Alexander reckons.

He adds, "Getting this type of early feedback from outside sources is pretty comparable to any type of design ('what do you mean this game is too hard? I can beat it just fine!' says the designer of the level...), but it's even more important (and weirder) when motion controls are involved."

Alexander says that one of the best ways to approach a motion control game is to consider fun physical actions outside of a video game, and then attempt to transfer these into a video game context.

"For example, kicking isn't necessarily the highest priority attack in a Jedi's arsenal," he says, "but we found that it's one of the more inherently fun motions to do, therefore we wanted to get it into the game. And even as I play it now, it's often my favorite attack to use."

"With controller-based games, the actual physical thumb manipulation or hand movement with a controller, joystick, or mouse/keyboard are usually not the highest priorities," Alexander continues. "But the motions for a motion control game definitely are, in my opinion."

Alexander agrees with my other interviewees, stating that "'Gestures to replace buttons' are almost never the most fun method of motion controls, but they're relatively easy to implement and use as a fallback, so they tend to be pretty popular."

Unfortunately, this has led to the notion that motion control tech is inherently "laggy", Alexander muses -- when in fact it's simply the poor implementation of the tech that is causing the visual lag, rather than the tech itself.

"It may also help to avoid trying to pile on too many control inputs simultaneously," he adds. "For example, on Jedi there were a lot of motions that, by themselves, are responsive and fun to do. But when all 10 of them are active at once, that can lead to a lot of missed inputs, and the general response of 'these controls are busted and laggy.' Of course, the other extreme to worry about is whether a motion controlled game is too simplistic, so balancing that is definitely a tough goal."

With regards to Kinect Star Wars, some modes were notably more fun than others. I ask Alexander which elements appeared to appeal to players more.

"I think the modes that tend to allow more freedom in motions tend to be more well-received," he answers. "So the Rancor mode, while sometimes janky, still could offer some of the most silly fun. It's mostly 1:1, it's not something you do a ton of in games (become a giant monster, destroy buildings, throw and eat people), and the in-game response to all of your motions are pretty satisfying, since something tends to explode every time you walk."

Managing to get across this feeling of being a giant monster worked great as a motion game, he notes -- "my secret wish is that someone greenlights an Xbox One Pacific Rim game using Rancor mode as a base to start from," he laughs.

"When it comes to player expectations, it helps when you're working on a motion game that has no clear and obvious controller-based game to compare against," Alexander continues. "With Rancor again, although there have been other giant monster games, they're not super popular or mainstream, so one could say that people are more open to different ways to playing them."

Dance games, on the other hand, are a proven successful genre for motion controls, so Kinect Star Wars' Dance Mode was, unsurprisingly, better received.

"Pod controls were somewhat well-received since the motions for it are relatively subtle, and 1:1," he adds. "Most criticisms tended to come when you were called to do secondary actions that interfered with the primary one."

"Jedi, on the other hand, immediately brings to mind numerous controller-based third person melee action games like Ninja Gaiden, Devil May Cry, God of War, and even another Star Wars game, Force Unleashed)," Alexander reasons.

"And also, of course, the most popular characters and weapons in the movies. So from the beginning, players will likely be wondering in the back of their mind 'does this play as precisely as those?'. Combine that with the rather vocal 'Where's my Battlefront 3? Where's my 1:1 hardcore lightsaber simulation game?' crowd, it's definitely a tough thing to go up against. And unfortunately, if it doesn't work well, it's very easy for someone to then say 'this is why controllers are superior, and motion controls are lame.'"

Alexander admits that his team was still tweaking and adjusting the motion control for Kinect Star Wars quite late into development.

"Unfortunately, there is no existing design bible to point to for how to do a third person 100 percent motion controlled action game, so it was very easy to keep iterating much later than usual," he notes. "There are still things I would have personally loved to try as separate control options, but at some point, as everyone else knows... you have to stick with something good enough, and ship a game. Unfortunately, 'good enough' wasn't enough to get better review scores!"

For those studios considering introducing motion control into a game, Alexander says you should ask yourself one question: Could this action be done with a button press and still be just as fun?

"If yes, then motion controls are probably not needed," he reasons. "To be clear, there are fun games I think that support both traditional controllers and motion controls (Child of Eden comes to mind), but that's because the actual physical motion and response (pushing forward with your hand, seeing pretty lights and music synced to your actions) is still fun (more fun, in my opinion), even if it technically works fine on a controller."

Matt Boch, Harmonix

Harmonix's past and present output has been very much music-orientated, but the studio has also been a key participant in the motion control movement, thanks to its popular Dance Central series, and the uploading Fantasia: Music Evolved.

To those developers considering motion controls in their games, the company's Matt Boch says: "Ask yourself if motion control is truly bringing something new to your game."

"Controllers have a huge number of advantages over motion control," he continues. "They offer tactile and haptic feedback that isn't easily afforded with camera-based systems. Controllers take advantage of the highly developed fine motor skills that gamers have spent years, even decades, honing."

Notes Boch, "These skills are so over developed that many gamers forget they ever took effort and have since subsumed them into their corporeality: wearing an Oculus while playing with a controller is 'immersive'. Players' overdeveloped fine motor skills are a far more reliable than their underdeveloped sense of proprioception. Beyond that, if you look at a sensory homunculus or a motor homunculus you can clearly see how much 'bandwidth' the brain has dedicated to controlling hands & sensing hand and finger position.

And Boch is keen to reiterate what other devs said shouted loudly from the rooftops: "Gestures are not button presses, and they shouldn't stand in for button presses."

"They are complex, nuanced movements and you should consider what aspects of a unique gesture performance can or should be communicated back to the player and what aspects factor into the game system," he says. "There's a huge amount of information in even relatively simple gestures. The challenge is determining what information is relevant and how best to use it."

For those studios investigating motion controls, Boch's suggestion is similar to that of the Double Fine team -- prototype quickly from an early stage.

"Motion is more nascent than pretty much everything else you're probably considering adding to your game," he notes. "If you're trying to find new mechanics, you really shouldn't prototype with a standard controller."

Fiddling around is necessary, since there are so many strange cases to account for that you will constantly come across during your prototyping -- as with any game development.

"So often I hear the story of Miyamoto spending a huge amount of time just walking, jumping, etc. with Mario during the development of Mario 64," says Boch. "Months and months until it felt right. This type of unwavering commitment to game feel is probably necessary for a majority of motion games."

And when asking yourself whether motion control is suitable for your game, consider this: "Did it occur to your team to incorporate this feature because you were excited about it? Does your team utilize the feature without being coerced or prompted? Is the game more fun as a result? If the answer to any of those is no, it’s probably not a valuable addition to your game."

Eddie Lee, Funktronic Labs

Funktronic Labs has been playing around with the Leap Motion technology for a while now, and the studio's Eddie Lee is preparing to give a talk at the App Developers Conference next week on developer apps in the third dimension.

"I feel that the most important part of integrating motion controls into your game is to aim for maximum intuitivity," he tells me. "Motion controls in 3D-space is definitely a new interface paradigm, so you cannot assume that people will understand motion-controlled mechanics as well as people understand other human interfaces like the keyboard or mouse."

Therefore when looking to implement motion controls, studios should attempt to draw inspiration from real-world physical behaviors such as swiping, turning, throwing, and other common movements that can help users to immediately connect with an experience.

"The ultimate goal is to make the controls so natural and intuitive that the user requires no tutorials or instructions to be able to play your game," he adds. Of course, there are plenty of ways that a developer can make a motion control experience unpleasant too.

"One thing a developer can do wrong is to not provide enough real-time feedback for their actions," he states. "Deliberate actions and gestures should be immediately recognized  (with visuals and/or audio) so that the user should immediately understand that their actions were registered. The worst thing you can do is to make the user wonder if their actions were properly executed; providing not enough feedback will disconnect the player from the experience and cause the player to simply check out, so make sure the feedback is crystal clear!"

When trying to decide whether the motion controls in your game "feel right", the best thing you can do is get people from outside your studio to try the mechanics out, and observe how they react, says Lee.

"Most people have not experienced motion controls before, so you cannot assume they understand motion-controlled mechanics very well," he notes. "Thus, your goal should be trying to make the controls feel as natural as possible and you would tweak your values accordingly. You should get into a loop of testing, observing, and tweaking; and you repeat this until you feel that users are no longer frustrated but having fun!"

And trying to shoehorn motion controls into an existing experience is always a big no-no, Lee says, as players will immediately notice.

"A shoehorned-in controls scheme usually ends up feeling very gimmicky and is a total disrespect for the art as well as the gamer," he adds. "And just like you cannot simply port a D-pad controlled game onto mobile by adding on-screen virtual D-Pad controls, you can't assume that plopping in motion controls into a game will be a good idea."

By taking time to evaluate if your game will actually benefit from motion controls, you'll quickly be able to determine whether or not you're better off going with a more traditional control scheme.

"For example, a heavily menu-based game might not be a good idea as motion control does not fare well in navigating menus," he says. However, physically interactive games such as Cut The Rope have transitioned very well to motion controls.