Interview: Float Hybrid Plugs Into The Matrix With Kinect

Visual effects veteran and innovator John Gaeta says he can't show you everything he and his team at the recently unveiled Float Hybrid is currently working on. But the stuff he can show you includes some of the most interesting uses of the Kinect's motion-sensing technology yet, and represents the tip of what Gaeta says will be possible with the advanced motion-sensing technology of the future. Gaeta, who is best known for his graphical effects work on movies like The Matrix trilogy, says he's been interested in branching out into interactive media for quite some time. Float Hybrid was not established as a game developer -- Gaeta says the company "did not begin with a mandate to do anything in particular," but was created as a place to explore how to "create more depth to an interactive experience overall." One of Float Hybrid's main focuses, as shown in multiple demos on its YouTube channel, is letting players easily navigate and interact freely in 3D worlds using the Kinect. For example, the Float Hybrid team has developed a system that allows players to guide an avatar freely using just the upper body -- lean forward to move forward, back to move backward, or twist at the shoulders to turn. Float's demos also show players using outstretched arms to aim and fire projectiles, both from first- and third-person perspectives, and even dueling over a network. "Everything that you're looking at can be done seated," Gaeta says of the team's control scheme. "One of the abilities of our company is we can scale and tune to a very refined amount of space and movement and still have fluid interaction. We can do everything sitting with minimal motion." Gaeta says such an understated, direct control scheme is important to attract adult players who aren't going to want to stand up and jump around for long periods after a day at work. Being able to sit also allows for longer Kinect games that can be more understated, at points. "In a film, you want to develop energy rhythm, you want to create a very careful map of energy flow," he says. "You don't want to be at 11 all the time, you need to create rhythm and cycles. ... The films that are the ones that tend to exhaust you and overwhelm you -- and I've worked on some -- are the ones that are at 11 all the time or 1 for too long. The same goes for when you design a good game, you need to map flow carefully." Outside of new control schemes, Gaeta's most ambitious ideas have to do with the melding of games and cinema. This doesn't mean infinitely branching, interactive storylines, Gaeta says, but rather the ability to let players inhabit the world of a movie and interact with it. He recalls spending long days using wide area laser radar to digitize whole mountainsides for scenes in What Dreams May Come. Back then, he "never thought people were going to be given real-time laser radar. Measurement can be the glue of all things. Measurement is the way you couple reality to virtuality." Gaeta's vision of the future includes films that are captured completely volumetrically, so viewers could watch from the viewpoint of the filmmaker or jump in and explore the movie world from any angle as the universe continues to evolve around them. "If we acquire reality in different ways, one could basically have interactive experiences, not just sculpted performances that are frozen in time, and you can explore the world, and you can play in the exposition of the film." Gaeta sees the viewers' role in this new kind of interactive movie as "carving one thread" in a living, dynamic universe -- in essence "incredibly sophisticated machinima" where your captured performance is inserted automatically into the recorded performances of the filmmakers. The idea has been imprinted in his head ever since The Matrix movies, and he thinks we're less than a decade from being able to realize it. Float Hybrid's introductory video, which uses two Kinects to insert a Gaeta avatar into a 3D scene, represents what he calls a "little crude" way of demonstrating this vision. But really realizing the dream is going to require some better technology. Gaeta considers the Kinect the rudimentary "version one" of a motion capture technology revolution that, much like Edison's early moving picture experiments, will eventually cause a sea change in entertainment. "Version 2 is certainly going to be a game changer once the cameras become HD and the depth sensor becomes longer range and high fidelity," he said. That additional data detail will allow developers to use features like simple face recognition, and even "drivers and filters based on behavior and on identity. There's a lot that can happen and will happen, that's why it's interesting -- from a social point of view, but also from the point-of-view of how you can actually create a much more immersive relationship with what's on screen." Gaeta also sees miniaturization and decreasing costs leading to a world where more sophisticated Kinect-like technology is integrated into practically everything. "It's going to be merged into TV sets, laptops, cell phones," he said. "It's going to be made modular and smaller. Maybe it'll be wireless, you can glue a bunch around your room, and you have full 360 [degree] sensing. You just have to imagine what the potential is of having full surround awareness of all motion and shapes in a space. It's the building blocks of very serious stuff." For motion control to really achieve its full potential, though, Gaeta warns that Microsoft needs to open up the technology to all sorts of developers, so experiments with motion sensing can flourish like touch-screen and tilt-sensing apps have on the iPhone. "My main concern, opening up the marketplace for people to join... making it very simple and very easy for third parties to provide countless experiences that wouldn't be able to make it through a large process that is required to make AAA games or large games," he said. "It'll be as wide as people's imaginations will carry it, but before people will be able to do that, it requires an environment that allows people to come in. It can't be hacker ghettos for that market."