Analysis of a Scene: Building Our First VR Prototype

by Eiran Shalev, CTO, NewPathVR

When Lisa Padilla first approached me with her vision for NewPathVR, I was immediately inspired by what we could achieve. Imagine the possibilities to not only unlock the hidden power of the brain but to empower our users to look inside themselves and find a way to improve. To feel better. Gain the strength to refine, and then to share that with others. While VR may be many things, to the folks at NewPath, it is the best tool to reprogram the brain for success. We huddled around and researched hundreds of papers where studies demonstrated how positive reinforcement, perception, and sensory filters can influence our behavior, as well as our memories.

Research and Design Phase
We knew we needed a prototype to prove VR as the right medium for spiritual growth, but what platform could serve us best?. Was this going to be a seated experience? How much interaction should it have? And what to build? For instance, did you know that if you perceive yourself as taller in VR, it actually makes you feel more confident during and after you remove your head-mounted display (HMD)? As it turns out, this is very true. Our discussions turned towards identity, and how to connect our users with their VR self. In gaming, this is called your “player self”. You have three “selves” actually. The first self, the real you, is what you do outside of games or VR experiences - ie your life: work, job, family, etc. The second self is the you that plays the game using peripherals, and experiences the content through the point of view of one or several characters in the game. You become a “player self” and share characteristics with the “game self”, but you are not the character. The third self, the “game self”, is the content’s avatar that represents you, and has a role to play in the content’s story or scenarios that your avatar experiences. By witnessing the story, and in some cases, by making choices for your avatar “game self”, your “player self” gets to experience those same emotions, and thus, share those same experiences. The cool thing about VR is that the player and the game self boundaries become blurred, such that you feel as if you are literally inside the content, and you feel much closer emotionally to the experience then you would be if you were observing the content through a monitor or TV screen. Keep in mind that taking an experience designed for a flat screen does not merit porting it to VR. All content in VR should be unique and specifically designed to transform and empower the user.

When building a VR experience, game mechanics that are based on challenge-reward systems create much more value for the user if they incorporate your senses. Adding a 360 visual experience may not be enough to trigger personal, real-world change. Adding 360 audio to that experience brings us closer, but is still not enough. By adding the ability to use your body, such as walking around and touching virtual objects, to influence the content, our team realized that we could create a world where consequences could have just as much impact on the “player self” as incentives. What’s more, if we incorporate at least 3 senses, this combination activates the memory centers or the brain. With the right experience, a player may create an association between something he/she experiences in VR, and a similar experience in his/her real life.

So, we knew we had to make a room-scale experience, and we knew we wanted it to leverage game mechanics that could change a player’s mood. The obvious choice for platform was the HTC Vive. But what about the content? I volunteered that for a prototype, we should keep things simple and demonstrate that we could achieve a basic goal. We wanted to transform a user’s mood from a negative or an indifferent position into a positive one. However, going through a sequential experience in VR, will usually not improve your life on exposure alone. In the real-world (or what we perceive to be our reality), we can usually learn any skill and master it, by practicing it over and over again. In our VR prototype, we needed to do the same. We decided that if we could create content that would teach our players some moral or zen-like lesson. A takeaway. But then also provided an opportunity to apply it, then we could create real personal growth.

Our “Self-Compassion Buddy” vision was born. In our prototype, we essentially introduced our users to their avatar self by literally creating a virtual mirror. The system tracked each of the user’s arms and mimicked his/her movement through the avatar that was reflected in the mirror. Our research showed that in order to strengthen the bond between the player-self and the avatar game-self, our users would need to interact with their mirrored reflection for approximately 70 seconds. This seems like a long time for a prototype, but in a future, commercial version of our product, the interaction could involve a game mechanic with a reward incentive. Next, we focused on the story ingredient. We did not require an elaborate story to demonstrate our vision; only a simple scene based on some narrative context. I believe that in order to create positive change or at least to invoke positive feelings, you need to have contrast, and that means placing the user into a negative situation - for a very short time of course. Then, follow it up with a positive environment filled with good energy. By placing the user into a slightly distressed state, and then moving him/her into a comfort zone, you can generate a sensation of emotional gratification. Games also apply a similar approach when they create a difficult challenge, in which a player must learn a new skill to overcome it. Once the skill is learned, the obstacle is easily navigated, and the player moves on to claim his/her reward. But more importantly, in your own life, when you undergo challenging times, and not only survive them, but learn to be stronger as a result of them, you then undergo positive change within yourself. Through overcoming these challenges, you may either improve your level of independence and self-sufficiency, or you may grow more carefree by successfully navigating stress and becoming familiar with it.

The Prototype Phase
We applied this to our “self-compassion” prototype. Imagine being immersed in a dirty, poorly-lit, virtual environment that exhumed negativity. You find yourself staring at your reflection in a mirror. You move, it moves. After a lengthy exposure to your reflected avatar, your avatar aka “buddy”, starts moving independently of you. It steps out of the mirror. Charges at you, invading your personal space. The result. You start feeling threatened. Your “inner bully” points his red finger at you and verbally abuses you, calling you “a loser...and a failure in life”. After a few moments of this, your brain switches to panic mode - a sort of fight-or-flight response. We kept the user in this state for about 7-10 seconds before interrupting the experience with another friendlier avatar. Any longer than that and we would have risked spoiling the whole experience and alienating our user.

The friendlier avatar, a nurturing female guide appears and rushes to your rescue. Freezing the bully in action, she explains to you that the bully is you. Recap: by belittling yourself, you lose confidence in yourself. The female guide then offers words of encouragement to rebuild your confidence. Her words manifest in a new scene in which soap bubbles shower from the sky. Some of the bubbles contain cute gifts such as adorable stuffed animals and pets. We introduced a bit of the fun factor in this scene. When the user pops one of these soap bubbles containing a gift, the female avatar aka your “guide” offers words of encouragement with a positive message, ie: “Lots of people care about you.” Each time you pop a bubble, the gift item inside goes into your collection, and new positive words materialize. In the next scene, you have an opportunity to apply these gifts and redeem yourself. We call this the “pay-it-forward phase.” You observe three couples, standing at surrounding points around you. These couples each reveal a virtual buddy figure matched with his respective inner-bully avatar. Similar to your initial case, the bully verbally terrorizes his victim. This continues in an endless cycle, until you interrupt the buddy, and hand out gifts from your previous scene's collection. Each time your gifts are received, the words that were initially associated with the gift, retrigger audio playback again. The buddy’s inner-bully vanishes and the buddy then begins playing the ukulele. When all three avatar buddies receive their gifts and play the ukulele, a well-known song begins playing, lifting the mood even more and embracing a sense of relief and closure. The three playing buddies merge into one buddy enclosed behind a giant mirror. And you are once again faced with your own buddy reflection. The simulation ends when the environment cross-fades into a sunny sky, and you find yourself standing on Cloud Nine. Literally.

Post-Analysis Phase
We quickly discovered that when a user goes through this VR experience, they feel better coming out, then going in. Our research also suggests that if we had ended simulation after the initial soap bubbles were popped, gifts received, and words of encouragement heard, then the effect would have been short-lived. By adding the “pay it forward” scene, where the user returns the favor and gives a gift to his surrounding buddies, we essentially teach our users to apply their acquired skills and pay them forward. To share. Therefore, our users resolve to help themselves, feeling a sense of contribution and meaningful value. Generosity in VR impacts our users in the real world and has longer lasting effects on their mood. It builds confidence and self-love.

Our biggest challenge in building this first prototype was testing it. I relied on using Unity as our main game engine development tool. Unity not only has established partnerships all over the world, and is compatible with 27 target platforms, but it is also free to use for development. Due to our limited resources and limited access to the HTC Vive, I ended up building our prototype using game object placeholders to represent both the HMD and the two game controllers. I then parented OpenVR’s game controller objects under these placeholders. When I repositioned the placeholder game objects in Unity’s simulator, I was able to estimate fairly accurately how the HTC Vive’s game controllers’ movement would impact the VR environment around them. On a weekly basis, under limited time and limited test access to the shared hardware, I methodically validated and tested our experience on the Vive hardware, tweaking and improving our prototype step by step. Unity is not quite WYSIWYG (What You See Is What You Get) when it comes to the built-in development simulator. That’s why end-to-end testing is so important, especially in VR. I set up a mode that would allow me to switch between my mocked game objects and the hardware game objects in the scene project. Doing so, allowed me to execute a series of tests on our shared hardware device while continuing development in our mocked Unity environment.

Our second challenge was getting our 3D buddy avatar to move correctly in our initial mirror reflection scene, used to build an identity association with your inner buddy. I solved this by building a mimic-engine that tracked the delta positions of the mocked, game-controller-object placeholders. The engine then inverted these vectors and applied the new deltas to a basic, rigged stickman model. I added constraints on the limbs of the stickman and locked the lower limbs so that only the upper body would be affected to move freely. And it worked. Additionally, since the Vive is a room-scale experience, the position and orientation of the stickman (aka our buddy reflection) needed to map to my HMD game object’s position, such that when I moved left or right, my reflection (facing me) would move in the opposite direction. And because the mirror image itself has borders all around, our 3D stickman was piped through a render-texture camera, that projected the image onto a 3D mirror game object as a texture. The mirror game object itself had no reflection, but projecting the stickman as a texture on top of it, gave the illusion of a mirror reflection.

No matter how you choose to implement your own VR project, remember that VR is highly immersive. Due to VR’s transformative nature, the underlying purpose of your content should support a key responsibility for contributing to social goodness, and hopefully, empower our users to live more fulfilling lives.

Eiran Shalev is an experienced technical, hands-on leader with 18 years of professional expertise overseeing top teams on mobile, social, and web technologies for products ranging from multi-player mobile & social games, to streaming video ads to interactive television and more. He comes from Disney Interactive, where he established the technical vision, and helped to scale and deliver Disney’s mobile technology platform to all game studios. Before that, he spent time at Koolbit, Kabam, and RockYou! He has built more than 50 games. He joins NewPathVR as CTO.