Mobile Q&A: GestureTek On Mobile Gaming's New Angle

Super Happy Fun Fun's 3D Tilt-a-World is more than just a great mobile game. It's the first implementation of GestureTek's EyeMobile Engine technology - a mobile extension of the company's Eyetoy and Xbox Live Vision technologies currently at work on the Playstation 2 and Xbox 360, respectively.

We spoke with GestureTek CTO and Co-Founder Francis MacDougall about the company's Eyemobile Engine, and it's potential impact on mobile gaming at large.

GamesOnDeck: First off, can you give me some general background on GestureTek and the EyeMobile Engine, for those unfamiliar?

Francis MacDougall: GestureTek is the pioneer and world leader in real time interactive computer vision applications, starting with the Amiga computer in 1986 as Very Vivid, Inc. We have issued patent and technology licenses to Sony for the PS2 EyeToy, and to Microsoft for the XBOX360 camera, as well as to vision enabled toys and other products.

The EyeMobile Engine includes three trackers which we call Shake, Rock, and Roll. These three trackers are available as a Brew Extension on the Qualcomm network, allowing developers to easily add gesture control to their games.

GOD: Can you explain a little more specifically how the technology uses a camera to sense movement? What if the camera is pointed at a varied (or not varied) backdrop?

FM: For Shake, we look at the amount of motion from one camera frame to the next, temper this by the scene complexity to create a uniform response, and provide a value to be used in the application. For Roll, we compare one camera frame to the next, and determine which direction the camera moved in between frames.

This amount of motion is converted into a “joystick style” input value for the game programmer to use. The tracking will not work in total darkness, since there is nothing to track at all. However, we find that even a plain white wall has a sufficient shading gradient for tracking. Clearly, scenes with large variation play into this form of tracking.

GOD: How do EyeMobile-enabled mobile games effect a phone's battery life, due to its constant use of the camera?

FM: We have not noticed a substantial impact. The light of the display has a far greater impact than the camera.

GOD: Tilting your phone to roll a ball is an obvious and intuitive use of the technology - what other genres or game concepts would benefit most?

FM: We are effectively providing a joystick input, so any game that can be driven using a joystick would be suitable. Snowboarding is a natural, also any racing game like motorcycling, etc. Target games like hunting and darts are quite effective, with GoSub60 coming out with a title in this genre shortly.

We’ve also looked at augmenting other game styles including golf, where highly accurate putting stroke measurement can be had, first person shooters, where the user needs to rapidly look around an environment (literally point and shoot).

GOD: Talk to me about the BREW development tools. What do developers currently have at their disposal? Can they set movement sensitivity, or code in recognition of specific gestures, for example?

FM: There are three levels of tracking available. They are described as Shake, Rock, and Roll. For Shake, we provide the amount of “shake” as a single value that the programmer can use as an input for shuffling MP3 playlists or whatever. For Roll, we provide the joystick style input that SHFF is using in Tilt-A-World. Rock is a gesture recognition system built on top of the Roll engine.

So, we detect if the phone has been rocked in one of four directions (up, down, left, right). This is useful for stepping through a grid or paging through a list, or for simple yes/no answers. For more detailed gestures, the programmer would have to take the output of the Roll tracker and build their own recognizer.

GOD: Would you say that gestural control makes more sense for portable gaming than console?

FM: I think it has strong applications in both areas, of course. For the console, the Wii controller from Nintendo has shown that there is a hunger for innovative interfaces that bring the user into more of a 1-1 relationship with the game experience. The ultimate will occur when a full body 3D avatar capture system is made available to the home user.

For mobile, the buttons are small and hard to access for complex gaming. Moving to a gestural model opens up all kinds of possibilities. We have additional trackers for mobile that are coming, including a very accurate 3D finger position tracker that will allow for sword play and user interface control. So the list will just keep growing in both camps.

GOD: Is EyeMobile Engine BREW-only for the time being?

FM: We have Symbian and Linux builds as well, and we have just signed a deal with one of the largest wireless carriers in Asia. Under this deal, handset vendors will be embedding the EyeMobile Engine into their handsets and providing a Java interface to allow downloadable Java applications to access the trackers. Phones based on this model will start shipping in spring 2007.

GOD: In a broad sense, what do you think this all means? What do you believe the greater impact on mobile gaming will be?

FM: My expectation is that all camera phones will incorporate this functionality within 3 years. The sensitive control offered by this tracking technique will enable novel user interface controls in addition to the gaming applications. The larger game development houses (EA, etc.) will wait until the broad deployment opportunity is there, so we will see smaller novel games developed in the near term.

3D Tilt-a-World

GOD: 3D Tilt-a-World is for the time being the only mobile title to support the technology - can you shed some light on the future? Are there concrete plans for any other EyeMobile-enabled titles in the near future or into 2007?

FM: GoSub60 will be launching their Camera Darts title in the next month on the Qualcomm network. This title lets you move the phone around to aim your dart throw. It also includes a mode where you take a picture of friends, pin them up on a wall and throw tomatoes and other objects at them. For 2007, we are seeing a ramp up of titles, particularly for the launch on the phones in Asia.

[This interview was produced in association with Modojo.com.]