Q&A: What devs should know about making games for Magic Leap

After years of speculation, Magic Leap's mysterious mixed-reality headset is now broadly available to developers.

Launched in six cities this week with an asking price of $2,295, the "Creator" edition of Magic Leap One may appeal to devs around the industry who are looking to blaze trails in mixed-reality game design.

It may also appear to share several weaknesses with the current crop of MR headsets: like Microsoft's HoloLens, for example, the One is expensive, still in active development, and only capable of displaying digital objects in a limited range within the wearer's field of view.

But unlike the HoloLens, or at least the HoloLens model I tried several years ago, the One's field of view seems large enough that clever devs will probably have no trouble making Magic Leap experiences feel immersive. When I visited Magic Leap HQ last month and demoed a One under controlled conditions, I was impressed by how well the headset meshes the real world with a virtual one, and how "real" its virtual objects can feel when devs take full advantage of the One's capabilities.

Of course, the demos I was put through are the result of years of work from folks who are intimately acquainted with Magic Leap's hardware. Anyone who buys a headset today and starts developing for Magic Leap next week has a long road ahead, one that's lacking in paths or guides. 

Context is key

With that in mind, I asked some of the leads at Magic Leap to share what they've learned about how to best design games for the One.

(By now you can also learn a lot more about developing for the One via Magic Leap's dev portal, which is filled with guides and code samples.)

As you might expect, they were all bullish about the headset's potential and eager to see more devs getting into Magic Leap. But I asked anyway because many of them are also veteran game devs, with a lot of experience working at triple-A companies like Ubisoft, Blizzard, and Electronic Arts. Their passion for the industry seems genuine, and on more than one occasion I was told that years of designing games for other paltforms gives you many skills you can apply towards developing for Magic Leap -- but you'll also need to start thinking within the limits of the hardware and the context of the real world.

"As a game developer working on this brand-new platform, it's sort of split in half,"  says Magic Leap Studios senior creative director Jeremy Vanhoozer, who previously worked at PopCap on Plants Vs. Zombies. "You have this half that's really recognizable, working in engines that you know, using a lot of the same language that you know, and figuring out how to create a lot of the same tools and experiences that you know.

"The other half is completely new. It's a brand-new platform, and something we've all had the chance to experience is working on software and hardware as they're being built, which I think is something developers are all too familiar with...but I also think there's opportunity," he continues. "So for me it's about taking a lot of the game development learnings that I had and then getting to use them in completely different ways...I feel comfortable in one sense, and completely exposed in another."

A promo shot of Magic Leap One, worn by the person in the sick denim jacket 

Multiple times during my visit I heard some version of this refrain: developing for Magic Leap means developing for the real world, giving up some control of a player's space in return for immersion in it. 

"As much if not more than the technical challenges of MR are the design shifts. Like, just everything is fairly different, from a design side," says Magic Leap Interaction Lab director Brian Schwab, who previously worked at Blizzard on games like Hearthstone. "A lot of things that make a hell of a lot of sense for a screen don't make much sense when you have all the room in the world and a very contextual environment and the ability to put pixels exactly where you need them, right when you need them, and then they'll go away. It makes a lot of rules about things we used to do on a screen not applicable.

"That being said, there's a lot of stuff that still does drag along," he adds. "You can do a lot of the things that you can do in a video game, but you can do them in a much more bio-mechanically sound way, because we have the ability to know where you're looking. We have the ability to have your hands in view if you're in the right place."

In my experience, the Magic Leap One is pretty good at tracking your hands and arms if you're waving them around in front of your eyes. This can give you the sensation of "touching" digital objects, even if there's nothing there to actually touch, because they can react to your movements fast enough to seem believable. 

This means devs can definitely do things like luminous, sci-fi menus which glow and hover in space, or affix themselves to nearby surfaces. However, Magic Leap staffers warn that flashy UIs can feel ephemeral and confusing if you don't render them in a contextual and believable way.

'Bias towards the body, and comfortable interactions'

"It might be helpful for developers to imagine this not as a 3D display or a wearable display but just as a wearable. To bias towards the body, and comfortable interactions," said Savannah Niles, a senior member of Magic Leap's UX team. "Another best practice would be to just keep [the player's view] clear. It's a see-through clear display for a reason; draw as little to the display as possible, allow for people to experience the real world, and leverage that. And then another thing we say in here is 'try to design experiences that take advantage of the best in bits and atoms.' So use physical real-world metaphors where it's appropriate, have digital things have object integrity where it's meaningful, and then use digital models and digital interactions where it's appropriate."

Multiple Magic Leap staffers cautioned against relying too much on purely digital objects when designing mixed-reality experiences. Since the ability to blend digital objects into the real world is the platform's selling point, they were quick to suggest devs include the physical world in their experiences whenever it makes sense to do so.

If you're designing a button someone has to press to launch a game mode, for example, you're encouraged to render that button on a countertop or other surface so that the player's hand pushes through it and hits something physical. It's not a real button, of course, but the folks at Magic Leap say even that little bit of physical resistance provided by a real surface is important to keeping players immersed in your game.

An example (pulled from a sample project built with Unity) of how Magic Leap renders virtual objects

"If you've ever tried to hit a button in like mid-air, with some exceptions, it's pretty unsatisfying," says Niles. "But the passive haptic feedback of the real world really brings that experience to life."

Magic Leap devs also have to get used to the headset's range of possible inputs, from gestures to location to facial expressions, none of which easily replicate the condensed, abstract input of a gamepad. The One does come with its own controller, a wireless remote (called the Control) with 6DoF tracking that sports a trigger, buttons, and a touchpad.  But it's hard to see how this might match the potential input complexity afforded to devs designing for gamepads and keyboards, and Magic Leap staffers are quick to suggest devs focus on breadth, not complexity, when designing mixed-reality experience.

"The controller's a good one, because I think for anyone who comes from game development, especially on the gameplay side, I remember spending so much time mastering the pad. Because developers who are used to traditional video games have this one input," said Aleissia Laidacker, interaction director at Magic Leap and a former Ubisoft vet. "So much of it is focused on, you know the user is sitting and looking at a screen, and the only bit of interaction is how can I make this game controller become super challenging. How can I make it like 'oh my gosh I beat this level with the skill in my hands.' 

"But now, we have so many different inputs. We can use our eyes, our hands, our voice. I love that because it's much more natural input, and I think one of the shifts for game development is, it's going to move away from the mystery of inputs and how we interact with things, and much more towards the accessible side," she continued. 

"One of the things that we've been seeing a lot, especially with our early access partners, is that now instead of just focusing on challenging inputs, they're really more focused on the experience happening around the user. One of the things we're trying to push for a lot is trying to think about how you create experiences that get someone to move around the space around them, to get them to maybe reconnect with their living room or the house around them, and let it become their playground. We hear in video games that open-world games are the playgrounds, and now your playground can also be your living room or the house around you. So how can developers on this platform start utilizing the space around them, and thinking about how to play in it?"

Designing games around the bodies that occupy that space is also a key concern for MR devs. In my experience the Magic Leap One isn't great at dynamically responding to other people in the space with you, but it seems to do a solid job of tracking your own body and responding to it. Some of the folks at Magic Leap think this gives devs opportunities to design for non-verbal interactions, the sort of things people do without thinking.

"I spent 25 years in the game industry as a gameplay and AI guy. So the thing that's fascinating for me is yes, learning about environmental context and getting AIs and systems that are very aware and seem intelligent because they're using the world in the right way," said Schwab, referring to his past work on games at Blizzard. "But quite honestly, for me as a human, it's been fascinating to see my body as a piece of input. Like when we really start to tease out the notions of non-verbal communication [in game design].

"If an AI character asks me a question and I lean forward, it's probably because I'm interested," he suggested. "But if I lean with one ear pointed forward, it's probably because I can't hear. And if I turn away, I'm probably checked out. All of the things that we now have access to, that we've never had access to before, because if a character asked me a question I'd get to answer by either pressing the A button or the B button. We never got any of the goodness of all of this rich input that humans do without having to be taught, without even really being able to control it. We can tap into [players' nonverbal signals], and that's really exciting for me as a gameplay person."

The most common tip I heard about developing for Magic Leap (and presumably, most mixed-reality platforms) is to compensate for the hardware's limitations (limited field-of-view, limited fidelity of digital objects, limited compute) by taking advantage of all of its capabilities. If you design a game of 3D chess except the chess pieces are fluorescent virtual dinosaurs, make sure they sound like it. Make sure that sound seems natural, and that it moves in relation to the player. Make sure the dinos are opaque, that they aren't likely to be occluded by anything, and that they behave in a way that feels alive (if not realistic).  

"To me, as we've been working on these experiences, there's 4-5 key things that come together to give you some assurance that something exists, even on the digital plane," said Vanhoozer. "Is it occluding? Is it opaque enough? Okay yep, i'm buying in. Is it acting relatively in the way that I would assume something like it to act? Can I interact with it? So I think sometimes it's less about what does it look like and where is it, it's more about the ecosystem of how you put those pieces together. There's some suspension of disbelief, of course, but I think if we set up a few basic rules in the beginning, and those rules hold fast, I think that's a key thing to getting people to buy in. That they can interact with it."

"There's a couple motifs we use throughout the OS and throughout our experiences that do help with these metaphors of like real world vs digital stuff," added Niles, the UX expert. "One of them is portals, like you see portals in everyone's experiences because portals are such a convenient way to put 3D content in a variety of different environments, and ground them to a plane like a wall or a table. Another one is materialization and dematerialization...but at the end of the day you do need a healthy dose of surreality to make it work, because this ultimately is a surreal medium."

"I would also call out spatial audio," Vanhoozer said, picking the conversation back up. "As we've developed these experiences, having things really react, and hearing audio around your room, whether it be something you're moving or you can hear a character rooting around behind your couch, I think audio is a huge component for getting people to buy in."

Looking ahead, the folks at Magic Leap seem genuinely excited about the prospect of game devs picking up their product and running with it. Laidacker told a story rediscovering what stealth games could be after playing a mixed-reality game that required her to hide from a creature in the room, crouching behind couches and chairs.

Another staffer, senior engineer Alan Kimball, said one of the first things he tried to build in MR was a 3D Missile Command clone that can be played on a tabletop -- and it proved incredibly hard to directly port from 2D into the real world, as he learned what does (and doesn't) work in mixed-reality game design.

"We talk a lot about experiences, but from a technical point of view, I think there's a lot of interesting tricks that will come out, that we haven't figured out yet, about how to do things like make shadows look good. We notice if you put a slight purple tint to it and feather the edges with a brighter color, that just looks right," Kimball said. "I think there will be thousands of little discoveries like that. It's like the first gen of any console, everyone is focused on learning and understanding stuff...and there's a lot of obvious answers, where like you do what you do in VR, and then there's gonna be more subtle ones where we take advantage of optics and psychology stuff to really ground it. So the generic drop circle of black is not going to be how we do shadows, but maybe the community will be able to figure out how shadows feel good in mixed reality, and we'll start seeing it everywhere."

Something else I saw while demoing the One is how easy it is to involve other people in your Magic Leap experience without their consent.

While the headset doesn't always do a great job of tracking other peoples' bodies, especially if they're moving close by, I was able to use the Magic Leap's 3D paint program to draw things around people and affix digital objects directly to them. Only I could see it, of course, but with a screenshot utility I could easily share what I'd done with the world.

Thus, Magic Leap (and mixed-reality tech in general) seems like fertile ground for whole new branches of harassment; when I asked folks at the company how they plan to safeguard the privacy of Magic Leap players and developers, Schwab said working in the game industry for so long has taught many of them to take toxic behavior seriously, and to prepare for it.

"Because we come from games and we come from the concept of griefing....games showed us what truly weaponized tomfoolery can do to somebody's experience," Schwab said. "And we're very aware of that. All of us have worked in triple-A. We actually serve quite frequently...for the people here who didn't come from games, they'll be making something and we'll be like hey, you might want to not do that, just because of this potential bad use case.

"So features end up getting put into the browser or whatever because we can see 'oh, if I wanted to misuse this, I could,'" he continued. "So we have this ability to call out potentially odious behavior and plan for it, so we spend a lot of time thinking about that. Because very much, [CEO] Rony [Abovitz] believes in the whole 'Mom Rule' thing. Like an 'I wouldn't put this on my mom' situation. So we try to abide as much by that as we can. Largely because it's a very agreeable thing to think about, and also because we all believe in it as well. We came here from games, I think, because we wanted to do something new, and we didn't want to drag that particular part of gaming with us. And given the fact that now there's no boundaries to the screen, it become a much more nasty problem if you don't think about it ahead of time."

For more insight into the development of Magic Leap and the company's plans for developers who use its platform, check out our wide-ranging interview with company founder and chief Rony Abovitz.