3 biggest hurdles faced in bringing online multiplayer to GearVR

My name is Richie Hoagland, and I am one half of the game studio RalphVR. This past week we updated our GearVR game Daydream Blue to include online multiplayer and positional voice chat.

We have been working on multiplayer for Daydream Blue since the end of the Oculus Mobile VR Jam back in May. The jam is where Daydream Blue got its start, and we are really excited to finally have our multiplayer out there in the wild. We have already met some really great people online, and we hope others get as much enjoyment out of connecting together in Daydream Blue as we have.

Along the way, there has been a number of hurdles we have had to overcome, as well as a couple of surprises. But there were three major challenges we discovered along the way. So without further ado, here are the three major obstacles we faced in bringing online multiplayer with avatars and positional audio to Mobile VR.

What is the challenge?

VR comes with it the promise of presence. When developers execute well, presence is the reward. However, with presence comes expectations. When I suddenly feel as if I am occupying space in an environment, not only does the logic of that place need to be consistent, but certain hardwired expectations of the human brain must be maintained for the brain to accept the reality it is being fed. Otherwise the sense of presence is easily destroyed.

There is also another form of presence we have been exploring which we like to call social presence. A feeling of existing in a space and someone else existing in that space as well. Similar to presence of self, when developers execute properly, this feeling is so powerful and new it can change someone’s understanding of VR. But social presence has its own expectations, and humans are very keen on picking up on the nuances of our fellow humans, or lack thereof.

We knew that in order to realize the multiplayer shared experiences we hoped to make possible, we needed to create a strong sense of social presence. The first step to this strong sense of social presence was in representing the player.

#1 How to represent the player

We needed to give players a physical embodiment in the space. The problem was that the GearVR provides high resolution low latency tracking of the head rotation--with striking results by effectively capturing the nuances of the heads body language--but no information about the rest of the human body. So do we have a body when it will not be mapped to the person’s body? Do we have a face when we don’t have facial tracking?

To begin, we created a simple masquerade ball hawk mask. A mask removed the need for facial expression and pupil direction while still providing players an object to identify as the other player. And because we had a representation, we could use the head rotation input, which conveys a surprising amount about someone.

Unfortunately, masks left players with an unsettled feeling. Seeing nothing behind the eye holes was unsettling and there lacked a sense of physicality to a mask floating in air.

Around this time we implemented our first version of voice chat in the game. Having a voice emanating from the mask, but with no mouth to cue in on, further added to the unsettling feeling of the mask. Instead of feeling like our friend was near us, it felt like some trippy sequence from a 70’s movie about a rock band, with our friends voice emanating from nothingness.

With the experience described above, it was apparent we needed a face. While we put more voice chat exploration on hold, we worked to solve the issue of representation.

We did this by adding a head with eyes and a mouth. The mask did a solid job of concealing the eyes just enough to make them feel alive, and to really sell it we added a nice blink that occurs at a random interval based on average human blink time. We are really happy with the result we have considering we don’t have any eye tracking. Having masks and heads also meant that masks were not necessary for those that did not want to wear one or have a fear of masked people.

However, we still had an issue with our heads floating in the air. We solved this by making the avatar feel a bit more solid by adding in a torso. We decided not to add arms and legs because we have no tracking for these body parts, and we err on the side of letting the brain fill in the fiction, as opposed to trying to do it ourselves and ending up with lifeless appendages. To obscure the fact we lacked arms and legs, we did however add in a detailed backpack. This backpack also plays into the fiction of the game as your backpack is your menu system in game.The final step was to add some rotational interpolation to the body to match the head and voila! we had a nice physical form in the space that was starting to feel like another person.

Unfortunately, as we had anticipated, as soon as our minds started to believe there was another person near us, we wanted to talk to them, so our next task was bringing in voice.

#2 Voice

Shea worked some magic with the open source audio library https://github.com/fholm/unityassets/tree/master/VoiceChat and we had a bare bones version of voice up and running pretty quickly.

Our goal was and still is a very strong social presence, and we knew that positional audio was necessary. For this reason, our very first implementation played the voice audio from an Audio Source at the avatar’s mouth position and used the Oculus Audio SDK to spatialize the sound.

In case you are on the fence about adding positional audio, it really is necessary. We have accidentally broken the spatialization at times, and the result is quite jarring. A player's voice comes from all around, or perhaps more precisely, from inside your head, and the sense of social presence is shattered. The sensation is that the “real” person is on a phone line, and the avatar is a puppet or doll representation. (With true social presence, this distinction between avatar and person fades and your friend becomes the avatar).

So we had our voices coming from out avatars and localized by the Oculus Audio SDK, and it felt like the player was speaking.

Sort of.

The cartoon smile just kept smiling and again social presence was broken. By this time we had a design philosophy of abstracting social behaviors out to simple representations, taking a cue from animation greats like Studio Ghibli and Nintendo’s wonderful work with Mii’s. So we made a set of sprites representing four mouth positions and swapped these randomly when the player was talking. This simple solution significantly sold the avatar speaking, and the brain seemed to easily accept and fuse the audio with the visual cues.

With these issues solved we were ready to support playing games in multiplayer. We had all this working around July, and we thought we were maybe a month from being done.

Nope.

Turns out that to maintain the sense of social presence, players need to share everything, otherwise the experience is not shared and the connection between people is broken. Social presence MIA.

So we started syncing the world.

#3 Pretty much everything needs to be synced

In Daydream Blue there are different mini-games and activities for players to engage in together. Players can compete at golf, skip stones, or just hang out and throw things at one another, amongst other things.

In golf, getting the gameplay synced was pretty easy, the only major hurdle was a way to keep the ball in sync when the Unity Physics system is nondeterministic. But once we were playing, matters that we never anticipated sprang up. For example, we inject UI elements into the world to give the player feedback while they are playing, such as an arrow to indicate their shot direction and strength. Because it was in the world, and players share this world, it felt off that players could not see other players input. The feeling of shared space/shared experience broke. So we added in the UI elements for other players.

This means that each player’s instance of Daydream Blue has almost everything synced. From time of day, cloud position, when leaves fall, wind direction, to the diegetic UI elements for how much money is earned from catching a fish. (I say almost because we still have not finalized our solution for when players are in their menu systems). This task of syncing everything became the biggest hurdle, especially for a two man team with Shea writing all of the networking, and a team that had never released a multiplayer game. It took us months to solve all the issues, implement everything, and test it all to get a stable build.

On December 8th we finally launched. We are really proud of Daydream Blue and hope people can experience the social presence we have worked so hard to foster. We are getting some great feedback from players, one player said they hung out with their sister who lives 1,000 miles away for over an hour playing games and fishing! Please let us know if you try it out, or tweet us at @daydreambluevr so we can meet inside the game :)

I look forward to all of the fascinating experiences people craft with multiplayer in VR. If you have not tried a shared VR experience, I encourage you to do so. When done right, it is magical.