MIGS 2010: Harmonix's Solution For Kinect UI Design

At this week's Montreal International Games Summit, Harmonix's Ryan Challinor discussed the difficult prototyping process for Dance Central's UI, which is fully-functioned and users preferred to any of Microsoft's own first-party Kinect titles. When it comes to developing a Kinect UI, said Challinor, "things that seemed intuitive were impossible or really difficult to pull off," which disheartened the team working on the game. However, he said, Harmonix was able to create a UI where "you can do everything you can do in any other game." The resulting UI tested significantly better than any of Microsoft's own games -- in a test of 432 users conducted by the company, two-thirds of users preferred it to any of the first-party launch titles, and over two thirds found Dance Central's UI "very easy" to navigate; none found it "very hard." Challinor is a programmer who works exclusively on Harmonix's UI designs -- something he said was "usually... straightforward". However, "when Dance Central came along it gave us an opportunity to work on a whole kind of new UI." The team's goal was to create a UI that allowed the player to fully "navigate the UI without picking up a controller or pushing any buttons," and which nobody would find so frustrating that they would wish to abandon it for a traditional UI. Unfortunately, said Challinor, since this is a new space, "there was nothing on the market we could look at for usable examples, or copy." A "strike team" was formed, with Challinor in charge of prototyping and then presenting these prototypes at bi-weekly UI review meetings. "Everyone all had input, and we had to sort of sell these ideas to get the rest of the team to buy into it." Each meeting ended with action items for Challinor, but outside of the meetings, bureaucracy didn't weigh on him -- he was free to experiment. He defined the needs for the UI based on Harmonix's previous games. It required forward and back animation, buttons, checkboxes, and the ability to display a long list of songs (particularly necessary as to incorporate DLC.) Failure To Kinect When ideas "failed, we had to get rid of it easily. A lot of the code wasn't going to last more than two weeks." The first plan was for a "virtual desktop", which would have several buttons on it which players could select with their hands. While it would display as flat on screen, the team realized that the system had to interpret the surface as curved, "like a section of a sphere," for it to behave like people expected -- due to their arm movements. The team quickly abandoned it, however. The team tried buttons with a "push arm to click" mechanism, but this failed for multiple reasons: "One [reason it failed] was that the action of extending your arm caused you to move your cursor position... another was that if you put your arm in front of your body, you would occlude your body and it would make Kinect very jittery." "We were also having problems with the extend-to-click system" because people mostly navigated with their arms extended to begin with, said Challinor. Also, "in practice, pressing on air just feels terrible", and none of these "big button" solutions could accommodate a song list. The team tried "sticky buttons" -- buttons which would stay selected once you moved onto them, until you hit another button. This had lag and jitter problems, and the team was never "able to hit the correct balance" to account for this. Prototype Kinect systems and software were particularly susceptible to jitter, so it was a big problem -- the team was not sure how much the device would be refined by the time it shipped, so couldn't rely on the hope that it would be taken care (which, says Challinor, is now largely the case.) Challinor worked on an iPhone-like page flicking system -- but it resulted in a lot of false positives due to wind-up and position-returns. "It works on the iPhone because you lift your finger between each one. Essentially [with Kinect] it's like your finger is always on the screen." For the lists, he also tried a virtual scroll-wheel, where the user would "grab onto the wheel and pull it down... but unfortunately the depth information you get from Kinect can't tell if your hand is open is closed... you couldn't disengage from it." He was so confident in this particular idea that he "spent too much time making it a fully-fledged implementation, but the first time we tried it, it didn't work." The experience solidified his belief that all prototype code should be functional hacks -- isolated and easy to pull out of the code later, but fundamentally they're hacks that are just good enough only to get things up and running, so they can be easily abandoned if the idea doesn't work. Taking A Step Back Two months into the process, said Challinor, he was "failing over and over again." He needed to take a break. "Trying to come up with new ideas is really demanding. And we had no idea where the finish line was. And Kinect was at the time a new technology so we weren't sure." He went off to do a week-long project, e.g. leaderboards, to refresh his perspective. When he came back, he had an insight -- the team had to abandon anything that used depth and make the UI 2D. Sliding buttons offered the user visual feedback, but it looked ugly, it was hard to disengage from a button once selected, and the lack of a visible cursor made it tough for players to handle. Challinor decided that the Kinect "cursor is like a pen on a piece of paper, that you can't lift or perform a click action." This lead to the creation of "notch buttons" which the player would hook a cursor into and move -- but they had learning curve issues. "Magnet buttons" with a sticky center were unintuitive and no workaround to give feedback did anything but confuse players. These efforts were further complicated by the newness of Kinect, said Challinor. "Using your body as a pointing device is undefined; there are no conventions. With your body some people thought it should be from the center of your body, some thought eye through fingertip, some thought wherever your forearm is pointing." Though the team went back and forth on a cursor, eventually it was dropped totally -- at the behest of the art team. The decision was made to return to the slide buttons after an artist did an effective prototype video in Maya -- which clarified how they could work to Challinor. Tuning the swipe mechanism was very difficult, Challinor said, because people swiped differently. The Kinect would also get confused mid-gesture, when the hand passed in front of the body, creating broken swipes. Also, he said, "tuning this was a really difficult task. It was too strict for people's swipes and too loose for other people's." Keep Iterating His eventual solution was to teach the players the right way to swipe with a short tutorial on the game's title screen, rather than try to learn to detect many different types of swipes. And though visual feedback is great, auditory feedback worked even better, in some cases -- shifting the pitch up as navigation speed increased, for example. His ultimate conclusion? In new fields, polish is more important to prototyping. At the meetings, no matter how good his ideas were, they'd get "shot down immediately" if the strike team couldn't understand it in example. "Don't just talk about it, prototype it. With something like this that's totally unknown, you can't talk it out in concept." However, the approach that was ultimately selected was almost lost forever; it was tried and rejected before being taken up again thanks to an artist who believed in it. Another lesson? "Following your intuition turns out to be a really bad guide" in uncharted territory -- as shown with the iPhone-like page flicking example. "Take a half baked idea and make it exist, and then keep iterating on it," advised Challinor. This new policy is working really well outside of gesture UI, he says, and it's now his general policy for new features.