Navigating the Unknown: Developing a Mixed Reality Game from Ground Up vs. a Console Game

There are some very specific questions to address with MR games. The player is playing a role in the narrative. They are playing themselves in their home as they are in real life. It was important for us to keep it this way to reinforce the immersion. Our narrative angle was to create an event (explained in the story) why 2 realities collide and why the player has a whole new world in their home. The player is not playing our main character Atalanta, the small 3-inch character they control. This is a big narrative difference from a first-person flat screen video game. We had to introduce a reason why the player is controlling Atalanta and why they should help her, essentially what is the player’s role in the story!

To reinforce the presence of Atalanta, our main character in Demeter, we created a lot of lines for her to address to the player, but also a one-way communication with them (as we can’t react to the real-life sentences the player could genuinely say). Atalanta talks a lot, but this helps to create the strong narrative we were after – a narrative that only works if the player has a connection to our main character, driving to a grand finale.

When we decided to create a game in MR, we had a ton of ideas. There were so many possibilities, we had to make a lot of choices (and cuts) to create a game in one year.

First we decided to create a platformer that would adapt to all furniture and walls of the room. Depending on the scale, each level would stretch in a way that the size of the character would not change. That means that we wanted to procedurally add or subtract parts of level design to create unique levels for the player in accordance with the size of available surfaces (tables, couch, walls…).

Another idea was to create biomes that would expand with the player adventure and fill the entire room. This has not been not possible due to production reasons. We worked on it, but the quantity of elements in the room in regard to the quality of design we wanted was too much for the hardware and the production resources.

Consequently we had to change to fixed levels that could spawn in different parts of the room depending on which horizontal or vertical surfaces, or empty volume in the room, were declared and available.

We had to adapt to any kind of room and configuration, and create hundreds of false rooms to test it. There were so many challenges, because as fixed as a room seems, it has so many reasons to change from one game to another, but also during the game itself. It could change with the normal use of the room in real day-to-day life, or the detection may be altered by the user, or anything else. In the end, we decided to keep in memory the position of every element during a single run, but respawn dynamically everything at each game start based on the lastest room declaration.

MR on VR headsets is still an evolving technology, so like any new technology, the SDK is developing alongside your development. This means you are working on a changing technology and need to adapt in parallel.

Novelab had the opportunity to access Meta’s Q3 dev kit, which proved to be a turning point for the developers. It has a better environmental detection (even if it was at this point in time still in development), but, the color and highly defined passthrough reflected the room environment so well that our game seemed real in the room. It was hard to tell how the game would look and feel with the black and white Q2 passthrough, but in Q3, the game was becoming real.

We realized early on that we had to scale back each level. Even if an idea was interesting and really appealing for a single user, after one run through they would not notice these small game enhancements. The only way for it to be a new experience would be to change the room with a different configuration and replay what they had already played. This case was so rare that it would have not been worth the effort in such a short time of production.

As the levels spawn in the room, we wanted them to be as real as possible and be part of the room. We decided to spawn them (depending on the size and orientation) in a specific place. Some are hung on walls, others are on tables, couches or on the floor. And there are some that spawn in the empty space!

We sometimes did not have the information of a surface specifically needed to spawn a part of the game (such as a table). In this case, we decided to have an alternative support for almost all levels. If there was no table, we picked an empty declared space in the room and we added a foot to a level to create the illusion that the level is on a long pillar that is set on the floor.

One of the biggest issues we had to consider was the undeclared furniture. It is a MR game where we ask the players to move around in the room to play. But, we are using only the surfaces of walls and furniture (not the global room mesh) and some elements of the room may not be declared by the player. Moreover, some elements could be occulted by a virtual element of the game. Consequently, we had to create game levels that would occult as little as possible to allow the player to be aware of their vicinity wearing a headset.

Sometimes the walls or furniture are declared, but some real-life elements that could constrain the player’s access (such as a bookshelf or a desk light) are not declared. We want the player to move around, explore and interact with the game, so we added an accessibility feature that allows the player to grab any levels from afar and move and rotate it to a free space. For cognitive reasons, the game has to remember where the level was put, so that if the player returns there, it will still be at the same place as in the real world.

We also wanted to use a room’s empty space to reinforce the immersion, a kind of environmental element, something like the rain. We quickly realized that although it would be amazing in a perfectly declared room, it could quickly turn into a mess if not all the room was declared. In the end, we decided to focus on creating subtle elements in the middle of the room in the empty space that evolves chapter after chapter. This way, we fill the room’s space without any problems with undeclared elements.

We also added some interactions to reinforce the player’s presence. Atalanta looks at the player’s face or controller when not far away from her or during specific cutscenes. In a game usually when there is a cutscene: the scene is launched. In Demeter, almost all the cutscenes have been designed to be in-game. We had to find a narrative way to launch a cutscene and explain why the player is not controlling Atalanta anymore and why after the cutscene is over they can again. There were so many new things like this to think about.

Everything was a challenge, event-level transitions. Moving the character around the room was tricky, going from one place to another (potentially behind the player), jumping to a level that may have been moved previously with the accessibility mode, or the room has changed and it is now inside or behind some furniture. We opted for an easy solution, a kind of ball of light teleportation (skill we are using as a game mechanic in other parts of the game). We tried many different solutions, this simple one was the most efficient and connected with the game’s lore.

The look and feel of the game were choices we made almost at the beginning. Some elements changed (a lot), but the art direction stayed the same. We wanted something that would be appealing and from another dimension, but also that would be different from real-life objects to reinforce narration and player security.

Spatial sound effects and music were also important. If you are moving next to a waterfall or missile, a spatial sound, depending on the position and distance, must create a ‘real’ life effect.

We had a lot of questions when it came to the game’s UI also. Where do we put the menu without breaking the immersion? In front of the player like most VR games? But what if there is a wall or furniture in front of the player? Do we put it in the middle of the room? There were many solutions but we opted for a mix of both.