Blitz Games Studios studio design director John Nash takes you through the process his team devised for coming up with Kinect game interfaces during the development of titles such as Yoostar 2 and Fantastic Pets 2.
[Blitz Games Studios studio design director John Nash takes you through the process his team devised for coming up with Kinect game interfaces during the development of titles such as Yoostar 2 and Fantastic Pets 2.]
Though we could spend many moons debating what a designer actually does and perhaps even longer debating how we teach designers, one thing is certain: the designer's job is getting harder by the day.
Of course the constantly evolving face of the game development landscape is an issue, as is the polarization of the gaming marketplace, but these problems boil down to a few basic issues. The one I will be focusing on here is the recent proliferation of gaming hardware interfaces and the subsequent challenges that the different pieces of hardware pose for designers.
For 30 years the triumvirate of the mouse, button, and joystick has ruled the roost as the preferred gaming interfaces between us and the video game.
Recent proliferation in the accessibility to gaming platforms, general global gaming demand and first party brinkmanship has meant that new ways to interact with digital entertainment have been developed. The goal of these new interfaces is simple: to reduce the entry barrier for everyone whilst still including traditional gamers.
This contemporary profusion of new interfaces has had a profound effect on gaming. The new interfaces have created a new breed of mechanically simple, often aggressively-priced games that have in turn attracted new groups of players to the world of gaming. This technological chicken-and egg-cycle means that we are now able to interface, control, and design games in a number of novel ways.
The impact on designers from this interface explosion has been huge. Not only do they have to learn to adapt to these new interfaces, but they must also adopt a more pragmatic approach to the entire design process. The market is changing, the gamers are changing, and so must our mode of development.
The good news is that although the challenge is great, there are a few easy-to-adopt approaches to make the creation process simpler and more rewarding.
To illustrate my point, consider the following diagram:
(click for full size)
With the help of my good friend Wikipedia I have selected a few key events in the chronology of gaming interfaces and dropped them on a timeline. Even though the choices I have made are relatively arbitrary, two things are glaringly apparent.
Firstly some interfaces were developed surprisingly early on and secondly, in 2011 there are an awful lot more of them to deal with. Other more esoteric or one-off interfaces have come and gone, and contemporary interfaces like Kinect blend several elements to create a new interface.
In a bid to avoid a "geek-speak" fueled war of semantics, I thought I'd define a few interface types to make sure we're all on the same page (at least for the duration of this article!) Broadly speaking, it's useful to think of the new interfaces (in the sense that they have become part of the core gaming landscape, not necessarily brand new technologies) in three categories:
Multi-touch - this covers all of the touchscreen interfaces seen on the current raft of phones and tablets.
Motion control - I'm defining these as hardware devices that are held or attached to the player which translate body movement into game control signals. A great example of this is the Nintendo Wii.
Natural User Interface (NUI) - remote-sensing devices that allow image, gesture and speech recognition in various configurations, to be used without the player touching any hardware interface. The obvious candidate here is the Microsoft Kinect sensor.
The main trigger for this article and its rather sensationalist title is the Microsoft Kinect sensor. With its innovative combination of hardware sensors and software processing, Kinect is the first truly mass-market NUI for the global video game market. Blitz Games Studios are one of the world's leading Kinect developers, having been developing games and technology demonstrations on this platform for nearly two years.
Throughout that time we have developed new ways of thinking in terms of game design, a healthy respect for the pitfalls and a love for the experiential space of possibility that the Kinect offers.
The challenges of developing a game based on a NUI stem from one simple concept -- the player approaches the game with an uncommon interface, their body, to interact with a game where the designer has to simultaneously channel the player's expectations and actions whilst accommodating 80 percent of their collective self-expressive interactions.
Put simply, people approach a NUI game with a wide range of expectations and abilities and the designer has to deliver a compelling game experience without the certainty of intent and leveling of a conventional joypad.
Confidence is a great word where NUIs are concerned and it has two very distinct meanings that occupy the efforts of the designer.
The first is the designer's confidence in predicting what 80 percent of players will do at any given point in the game. By using controls that are intuitive, working with the expectations of the average player and channeling the player towards specific interactions with good instructions and subtle multi-sensory manipulation, the risk of player frustration is significantly reduced.
The second meaning is the confidence in the engine (and tangentially the designer) that the interactions enacted by the player are being unequivocally understood.
The success of a NUI lies in delivering confidence in both these instances, whilst also delivering the intended game experience without interruption. If a game expects something unusual out of the rules or fiction of the game, or indeed fails to detect a valid input, the illusion fails instantaneously.
Kinect is the first truly mass market NUI for video games. It provides six major data streams to the designer, which provides a massive space of possibility for creating new game experiences.
If you also consider that these streams are programmable, then potential upgrades via clever team-side custom code and official software updates will ensure an exciting future.
In simple terms, the Kinect delivers a number of streams of data, and the combination of these streams defines the type of experience that the player will have. The range of possibilities is huge, but so is the amount of potential misinterpretation at every stage. There are six primary streams:
Camera video - video stream
Camera stills - face recognition
Depth buffer - skeletal data
Depth buffer - depth data stream from the depth sensor
Microphone array - audio analysis
Microphone array - voice recognition
Let's jump right in with a deceptively simple example: throwing a ball for a dog using Microsoft's Kinect. We experienced these issues during the development of Fantastic Pets 2. There are three main challenges here: the player understanding that they have to throw the ball, the detection of the throwing style of the player, and the release point of the ball.
Assuming that the game is named appropriately, in this case something catchy like 'Fetch', that the dog is eagerly waiting in front of you, and that an avatar has demonstrated the throw on-screen to the player, we can safely assume the player is primed for some top quality throwing action.
The biggest paradigm shift for design is centered on point two -- detecting the throw. Essentially the designer has to construct a set of rules that will detect most peoples' throwing style with a high degree of confidence. By defining a rule set based on combinations of joint positions, angles, vectors and velocities, it is possible to detect the throwing of a ball irrespective of the actual throwing technique.
Finally, the release point of the ball has to occur without the use of a traditional button press. In these situations, the use of predictive confidence (discussed below) is the way forward.
The communication of the game, and the subsequent interpretation of the player of that communication, both rely heavily on the effectiveness of iconography. Traditionally we are used to selecting graphical representations for a game and using them until release.
The effectiveness of the iconographical mechanisms can and should be tested exhaustively with a usability program. It is no exaggeration to say that the use of one graphic over another can save untold numbers of problems. Truly, a picture can be worth a thousand words!
For a demo featuring a driving mechanic, we tried a number of techniques to instruct the player how to steer a car. Everything from onscreen animated arrows to speech was attempted. Trying to get the player to understand how they should extend their arms and steer in mid-air seemed to be a tricky problem.
One of the artists had an idea and simply placed a graphic of a steering wheel onscreen. In tests, this worked perfectly; clearly, using the right image to trigger real-world actions is the most compelling way to interact with a NUI.
During the development of Yoostar 2 the usability testing spawned a lot of very useful information. On the first screen we had a slider button to start the whole experience. The design went through several iterations until we settled on the single 'hover' button solution.
Evolution of the START button
The development of this button was driven by user feedback from the usability testing. To illustrate the kind of feedback we were working from, here are a couple of excerpts from the usability reports:
“How to interact with the 'handles and rails' UI component is not intuitive. Visually, it resembles a VCR-style play button with corresponding descriptive text. As a result, most users tried to push or grab the button as there is no indication that it should slide.”
“Many users hovered over the handle, then began the swipe action to select their choice.
However if they didn't move in a horizontal line, it was common for the action to be cancelled, as the cursor had moved too far in the vertical direction. Typically, users gesture in a diagonal motion, not horizontal.”
“Some users had difficulty staying within a handle. The 'gravity well' effect worked well in snapping them to the handle, but it was sometimes not strong enough to keep them contained within it.”
One of the jewels in the crown of NUI systems is the ability to use voice control. Whether it's phonetics-based or waveform analysis, there is no doubt that speech recognition is the area that requires the most investment in terms of design and technology as compared with the other sensors in Kinect.
The key to success here is twofold: firstly make sure that all active words or phrases are very different to one another, and then ensure that the system is only trying to identify a limited number of words or phrases. Too many, or non-dissimilar, words and phrases always results in much lower recognition confidence that will inevitably lead to recognition errors. In terms of the player's experience, remembering the phrases, saying them at the right time and accent issues will all complicate matters further.
Gestures are a cornerstone of NUI interaction. Again, restricting the number of gestures that can be recognized at any one time is the best approach. From user testing it is also apparent that players can only remember a certain number of gesture shapes.
These shapes need to be regularly repeated by the player, or at least there must be a non-invasive memory jogger system such as environment shapes or tracing shapes. Additionally, gestures often work best in games where there is a good context for them, for example a wizard game where you are casting gesture "spells".
Front-end navigation is also a very obvious and viable use of gestural control. As more titles are released, some control systems are clearly more "usable" by a greater part of the gaming population than others. Development guidance started with handles and rails but seems to be polarising between the pointers and 'hover' buttons system, and the arm sweep menus and swipe system. Interestingly, approximately 20 percent of players seem to have difficulty with the best gesture systems indicating that there is still much to learn in this area.
So we know about the variety of sensor streams and the fact that the players can do pretty much whatever they want in front of the sensors. What exactly then does NUI design need over and above regular joypad game design?
To be honest, I'm struggling to find a term. In essence it's a true second order design process much like emergent design. There are relatively few basic components but the range of possibilities between player interpretation and system confidence means that the designer must employ some sophisticated techniques to reduce error and confusion.
By combining clear instructions with solid iconography and continual positive feedback, the designer should, and must, be able to channel the player towards an expected interaction that produces a satisfying experience. As the days of pressing X to jump draw slowly to a close, so the days of perceptive design beckon us into a new era.
For situations that require the game to react without user confirmation, it is necessary for the game to predict that the player has completed the action. With the ball-throwing example, there is currently no real way to determine the point of the ball's release.
In this case, if the game has detected a throw, it is reasonable that the player will complete the throw; therefore the game should release the ball at the expected point without player confirmation of the release.
This system is also useful when driving pre-set animations, as opposed to using avateering. If the game has given good instructions as to what it expects the player to do and is waiting for a small set of actions, it is possible for the game to partially confirm player intent before triggering a pre-canned animation.
The key to making sure that this system is transparent to the player lies in listening to the feedback from the players during development; a heavy-handed system or one that suffers marked latency will shatter the illusion very easily.
Internally at Blitz Games Studios we use a number of systems to make sure our design and production processes are player focused. These systems have distinct functions as follows:
Focus testing - viability consultation of new game concepts and IP with gamers.
Usability testing - testing functional game builds on the target market with rigorous usability data recording and analysis.
Analytics / metrics - code-based solution embedded in the games that supply play-style data to facilitate player experience improvement.
Whatever these rather evocative phrases conjure up in your head, they mean only one thing: listen to your players. If you're wondering exactly when you should be listening to your players, the simple answer is always. Focus test a feature set with players before you implement those features, then prototype them and let the players give feedback on those features in usability testing.
If you're really smart, you'll have an analytics or metrics engine running in the background feeding your dev team with constructive feedback 24/7. Seek out feedback, listen carefully to it and plug the outcomes into the next iteration.
Collaborate: like all quantum shifts in the technology and craft of making games, collaborative knowledge sharing is paramount if you wish to keep up with the pack. Everything from the lessons learned from using iconography to player interpretation during testing should be recorded and passed on. With multiple new data streams and the loss of the certainty of a button press, designers would be well advised to compare notes if they wish to adapt and create experiences that endure difficult market conditions.
Guide the player: designers no longer dictate every aspect of the game to the player. NUI experiences afford the player a vastly greater level of self-expression. Whilst this self-expression does massively increase the chances of misinterpretation, the careful designer still maintains the right, and indeed the duty, to channel the player to a desired action.
Less is more: in most aspects of design for the NUI, ensuring that the game is only looking for a small number of inputs in terms of gestures, movements and/or sounds is the key to success. Low numbers equal high confidence.
Informed iteration: through the use of focus testing, rapid prototyping and frequent usability testing, many design assumptions and omissions can be mitigated. The best people to tell you what is wrong with your game are the players themselves. One easy and perhaps obvious lesson we have learned is to actually listen to and trust the feedback. The age of designers 'knowing better' than gamers is at an end.
Innovative tech and design: though it seems obvious, there are always clever ways to circumnavigate bottlenecks and technical cul-de-sacs. As an example, on The Biggest Loser: Ultimate Workout we required the ability to detect floor-based exercises. The Kinect SDK does not natively support floor-based skeletal detection, so we wrote our own system to fill the void. The result was a genre-leading exercise detection system.
A contentious title for sure, but why does it resonate so well now? NUI systems like the Microsoft Kinect have opened up a cyclone of creativity that will continue to sweep through the landscape of game development. The demand for these new gaming experiences will drive developers to seek new genres and in doing so they will have to rewrite some of the rules of game design.
NUI games are a positive because they remove more barriers preventing players from engaging in our experiences. The flipside is that as developers we find ourselves facing ever greater challenges of coping with the random inputs and expectations of a very large and varied game-playing population.
It is clear that as these interfaces become ever more complex and intuitive, so our game designs and processes must become more flexible and able to adapt. Iterative design based on a backbone of usability testing is the only sure way forward.
The age of testing games just before release is gone; in fact the idea of only exposing our creations to the players at the end of the development process now seems positively ridiculous. Social games and NUI games have proven the validity of regular player feedback and in this way the player has indeed become the producer.
As a boy I dreamed of owning a jetpack but sadly, like many other dreams, that dream was dashed on the rocks of slow mechanical technological advance. So I'm now focused on a new dream; now I want my own holodeck. For me, NUIs like Kinect are the next logical step on a line between the potentiometer and the holodeck itself. Being an impatient designer, I want to do as much as I can to bring that day closer.
Read more about:
FeaturesYou May Also Like
.png?width=300&auto=webp&quality=80&disable=upscale)
