Why Valve's new SteamVR Knuckles are important for VR

User interfaces in virtual reality start with your hands. We know that now with the Oculus Touch and the HTC Vive, but even when VR was simply a screen strapped to your head many felt that hands were the future. We began to develop The Gallery on the first Oculus devkit with the Razer Hydra (a Sixense technology) to deliver surrogate hand tracking and body presence back in 2013. And when Valve invited Cloudhead and the first wave of developers to see what would eventually be known as the Vive, Valve showed a commitment to that same vision. Now Valve has invited us and a new wave of developers to begin working with their latest prototype—the SteamVR Knuckles, a wearable VR controller that tracks not just your hand, but each finger too.

Back in 2014, we didn’t realize exactly how accurate SteamVR tracking was—the whole notion of roomscale VR was almost incomprehensible. Up until that point, there was still some motion latency in VR, so you never felt completely attached to the actions in your hands. But once the SteamVR Lighthouses started tracking objects in a 3D space, it was a deep and immediate connection of, “Holy crap, that’s actually my hand in VR.”

Our goal then was to deliver an experience where the player doesn’t have to think about the controller, and has only natural, gestural interactions. We wanted to demonstrate why this kind of input—your hands themselves—was meaningful. When we received our first Vive devkit (wired at the time) we were taping them to our hands in order to feel more immersed, and we even spoke to Valve about crude ways they could strap the controller on.

Those early Vive prototypes already showed an incredible level of fidelity, capable of measuring the tiniest fraction of a movement. It’s like the Moore’s Law of motion control; each incremental improvement in tracking brings with it new possibilities. So as soon as you have that kind of fidelity with your hands in VR, you need your fingers to be more purely represented. And that’s what the Oculus Touch started to do by bringing capacitive, gestural input to the controller.

Where Touch differs from what the Knuckles offer, however, is that you’re still pushing a binary button in the end; Touch feels more grounded in traditional gamepad design. Specifically, you always feel like you’re holding something with buttons—and that works perfectly for gun games and sword games. But the Knuckles take that further by removing the abstractions of first-gen VR inputs. Even though it looks like a more complicated device, it’s actually a much simpler one.

With the Knuckles, you’re not holding a controller; it simply straps to your hand and rests in your palm. If you relax your hand into a natural flex, the controller stays put and keeps tracking your fingers. If you reach out to grab an object in VR, your hand wraps around the base of the controller, giving a tangible feeling of grabbing something. That physicality is something you don’t get from data gloves, or vision based inputs without any device, and that feeling can then be fine-tuned with haptic feedback. Plus, you’re not passing around a sweaty data glove between your friends.

When reaching out to an object with the Knuckles, I’m not thinking about the controller I’m holding in my hand, because I’m not holding one. I’m not thinking about how to use my fingers either, because they’re not assigned to a button press. I’m not even thinking about my hand, and that’s where the magic comes in—I’m just thinking about grabbing the object, as I would in real life. That entire grasping motion is represented in VR, whether I pinch with two fingers, scoop with my hand, or close my fist around it. The Knuckles track your fingers by the distance they are from the base of the controller (your palm), and represent that movement in VR. It’s second nature.

As developers, when we receive prototype hardware like the SteamVR Knuckles, it makes us want to push the capabilities. In the past, that’s meant radically rethinking our stack of interactions and locomotion systems—virtually redesigning the game. We’ve written about what these kinds of changes have meant for The Gallery in the past, but the long story short is that new controllers like the Knuckles aren’t just affecting the complexity of interactions. We now have new possibilities for game mechanics and design that haven’t and couldn’t have been done before. It’s to be seen how these controllers will impact Heart of the Emberstone in September, but they’ll be a core focus in designing Episode 3.

Think of an interface-heavy app like Tilt Brush. Dials can be intuitive, but using your fingers is organic. There’s a possibility for gestural movements to call functions and navigate dense data; there could be an entire language built out of using your hands to manipulate paint brushes and pencils and sizes and colors. Once you take the mental load of an interface off the player—once they stop thinking about the controller—you can leverage that partition into experiential design and organic controls.

The kind of technology that the SteamVR Knuckles offer is not just impactful to the future of input in gaming, but also the future of output. I can go to a social VR space and point to something, or offer a peace sign, or tell somebody to hang loose without having to think about it. The controller doesn’t guess your gesture, or snap to a new position, it represents your fingers based on the distance it calculates. The more natural and intuitive the interface, the less we think about hardware. And the less we fixate on hardware, the more present we can be in VR.

Prior to the Knuckles, hardware developers were looking for something that would be more broadly accepted by the general public. Something that resembled a Wiimote, like the Vive wands, or something that when put together resembled a gamepad, like the Touch controllers, meant that VR input was familiar. Strapping an alien device to your hand in first-gen VR would have been too much too fast. But I think it took the evolution of those two controllers to get VR to the point where the public could be comfortable with the idea of a controller strapped to your hand.

There are so many moments in life in which using your hands is a vital part of the experience. There are implications for education and communication—with audiences who don’t generally understand videogame controllers—because the SteamVR Knuckles open the door to that broader audience. These are pick-up-and-play controllers where you don’t have to think about the input, you just reach out and interact in virtual space.

User interface in virtual reality starts with your hands. And once users are empowered in that way, and don’t have to be told how to use the technology, the next generation of virtual reality is here.