The Advent Of The 3D Camera: Softkinetic CEO Tombroff Sees Console Future, PC Possibilities

3D camera systems have not caught on in gaming just yet -- but they seem to have massive potential. Unlike traditional 2D camera systems, which have proved limited and gimmicky in many ways -- think EyeToy and Xbox Live Vision Camera -- these systems deliver depth as well as image, allowing the player to manipulate the game world more effectively. Softkinetic is a Belgian company that does not market cameras itself -- it does, however, sell SDKs which help developers utilize the cameras supplied by the companies which sell them. While mainstream gaming applications for these technologies have yet to be deployed on consoles or PCs, it seems likely that in the age of Wii Balance Board and Rock Band that this is more than possible. Softkinetic CEO Michel Tombroff's most exciting statement may be this: his company expects these cameras to launch on consoles. "We foresee it. We cannot predict it with high accuracy, but we believe it's something that's going to happen, yes." Tombroff was eager to explain the technology's place in the world of gaming, be it serious games or casual games -- and anything in between. And after our discussion, he demonstrated a number of game-like tech demos for the camera system -- you can see videos at the company's official site. These demos, while fairly rudimentary from an execution standpoint -- it's up to actual game developers to deliver engaging gameplay with seamless control -- are a very believable proof of concept for this technology, and easily illustrate what separates 3D imaging from the EyeToy of yesterday. For more background into this technology, you can check out our interview with Tomer Barel, marketing VP of 3DV Systems, a company that is working on an actual 3D camera device to be launched in the consumer market. Can you talk a little bit about your technology? Michel Tombroff: Yes, absolutely. First of all, we provide the software technology that enables game developers to use a new type of hardware devices -- that is, 3D cameras. So those 3D cameras basically take a 3D view of the scene -- the depth dimension of the scene. And so what we do is we extract from that image all the necessary information that the game developer then can use to build a game by detecting gestures, body parts position, velocity of moves, etcetera. And so what we do is export that as an SDK to a game developer that can then do any kind of gameplay with that. So now, like you said, one of the big advantages of those 3D cameras is that it enables, first of all full-body detection. So from foot to your head, without any markers. No controllers. Which doesn't mean that using an extra controller is not fun. We are working also on projects where you can combine controllers, you know, an extra accessory, but the idea of this technology is to do fully immersive, transparent, intuitive gameplay. We only do the SDK. But what's very important is that we support all the cameras. So our SDK has been adapted to all the cameras that are available to you on the market. And we work very closely with the camera manufacturers to make sure that we do that. And right now the cameras are only available for PC platform. There's nothing on console for it, is there? MT: As of today that's right. And it's evolving rapidly. What features do you have in your SDK that make it worth licensing? Because some of these camera systems, they come with software for developers. MT: Absolutely. So first of all, the cameras, they come with a basic low-level device driver that enables you to get the image and do some stuff with it. So what we bring on top of that is, first of all, we spent a lot of time, about four or five years now of research, to reconstruct from that image an understanding of the body part's position. Any part of your body: hands, elbows, shoulder, head, et cetera. And we provide that to, again, the developer so you can reconstruct the avatar and re-immerse the person back into the game. So as far as we know, nobody else does that today. You compare here the driver part with the more sophisticated gesture-recognition aspect we provide, yes. So you've done research into developing software that has advanced recognition techniques compared to what's available from the stock products. MT: Yes, exactly. And since most 3D cameras are relatively new to the market, we've been working on 3D technology in general for about a few years. Nobody really has ever worked on this with much depth. But now that the cameras are coming to the market, we see a demand growing for those kinds of tools, for being able to extract useful information from those images. You've spoken about getting the camera on the market and having it available, but there's a lot of interest -- especially with the success of the Wii -- in alternative input methodologies, and with the casual PC market also, it seems like an intersection right now that could work for people. MT: - You're absolutely right. [In] casual games, gesture-based gaming is really taking off, and so this will bring this technology forward. There is also the fact that, with this new technology, people can explore much more immersive and intriguing gameplay, because, comparing that just to the Wii-mote, which is just one point of contact and directional movement. Here you can really go through the full 3D usage of your body. So it's next generation, yes. You can tell the distance at a fairly high level of complexity -- it can read the curvature of your face, or read your fingertips. MT: Exactly. And so we support three levels of such -- we call that "interactions". We have the full-body interactions, where the person is sitting five feet away from the screen in place, with the full body. For which we don't also need to detect the nose, a finger, the facial expressions. We have very close body interactions. So that's when the camera is very close to you, and we can detect an individual finger movement, et cetera. And then you have the upper body, where you have, like this... [Tombroff mimics playing a game from the torso up] and you do moves with your head, and we detect. So depending on the type of game that the game developer will want to develop, he or she will use different modules of our product, to detect different parts. When you do full-body casual games, you don't need 3D to detect fingers. If it's a dancing game or a sports game... you optimize the usage of the SDK to the gameplay you want to make. And do you have non-game related applications that you see for this technology as well? MT: Absolutely. Actually, maybe to categorize them in different sectors. The serious games industry is very, very important to us. Industrial simulation, military applications, and education in particular. And then there is one very important market for us, which is advergames, where the usage of this technology with marketing, in a marketing concept, allows you to [create] interactive minigames for people to interact with a marketing environment. And you see a lot of demand for that at the moment, yes. The uptake of peripherals -- obviously there's a lot of people that have webcams and stuff, but -- the uptake of something like a gaming peripheral like this might be a little bit difficult to envision unless you have a really strong partner. It could be tough. MT: I totally agree with you, and the people who make this technology a success are the tier one companies that will bring new titles, exciting titles to the market, new accessories, new games. And we see that happening in the near future, yes. The success, we think, will be in making sure that the first few titles that will come out will bring an experience that is so [much] better than existing experiences, that it will make it a success. And so we are exploring that with our clients now, to make sure that they explore the 3D technology to the maximum. It's not just adapting existing games to 3D -- which is easy to do, and is quite fun. I actually will show you some examples -- but it's really to take it to the next level. We need to make short outlines for the game developers -- make the best use of the SDK, and take that to the market. And so we are not developing games ourselves, but we are involved into their efforts to develop those games. So we are really working very closely with them. Because first of all, we are the only ones who know how to use this technology at the moment, so we bring that expertise to them. And also making sure that they use it the proper way, to explore the possibilities. How does the SDK function with other mature game development technologies? How do you get it to interface with the technology people already have? MT: That's a really good point. So we've made sure that the SDK we provide is either compatible, or easily adaptable, to 3D game engines and other middleware that people would use during the development of their applications. And so the example that we'll show you later on based on the Virtools game engine, for instance, and we are supporting others as they come in. This effort is actually really trivial to adapt to different platforms. We support C, C++, C# APIs for developers. It's a standard environment. You can develop on the PC, and then once we're available on the console, we can migrate to the console, so it's a relatively straightforward environment. It takes a couple of days for someone who's coming from those environments to adapt to the new API. Earlier we discussed the idea of the camera coming out on consoles, which you seem to expect to happen in the indefinite future. However, with the PC there's many different ways to go, and it might be harder to tie it to a product, from a game side, especially with so many casual games being either online or download. But there's a lot of ways things could go. MT: Yes, and... actually, that's a good point. That's why we really put a lot of effort to support all 3D devices. So that if a consumer buys one from Manufacturer A and has a game that works with that -- if a game has been developed with the right platform under it, it should work with... like any game works with any controller, we expect the same to happen with these games. So that the consumer does not need to select one camera for one game, necessarily. It has to be a standard platform -- which it is. There's a standardization of the 3D camera? MT: No. And actually that's where we come to the picture. We standardize the signal. And so when you are sitting on top of our SDK, you can plug any camera and you get exactly the same thing, yes.