Mixed Reality Content Creation for Virtual Reality Games

Mixed reality! It's a hot topic these days, with Microsoft using the term for their post-Hololens pursuits and partnerships with Acer, et al. But back in the day (aka a few months ago) "mixed reality" was a term I more commonly associated to a type of video recording. Mixed reality video takes live, chroma-keyed footage of a VR-player composited between the foreground and background of a VR game. Makes the player look like they're actually in the game, as the main character. 

It's nifty. And a pain in the ass. In this post, I won't focus on the how-to of setting up mixed reality as much as on the reasons it's useful and ways I've seen it done successfully.

All resources, software, and hardware I mention are found at the end of this post in the "Additional Resources" section.

Why make a mixed reality video?

In my humble opinion, it is the best possible way to show off VR games to audiences watching in two-dimensions: examples being video content produced by YouTubers/Twitch streamers, and trailers created by developers. At some point, VR video content watched by audiences in a VR environment may become popular (check out VREAL for a company working on this!) but it's not quite there yet. Even if it does become a thing, there will still be a need to cater to people watching in two dimensional space to loop them into the VR ecosystem.

1st person VR footage can be unpleasant to watch for long stretches of time due to the camera being, well, someone's head - the footage is shaky, it's crazy how much we move our heads around without realizing, and can be nauseating to watch. Typical streams/videos also have a small space for webcam footage overlayed on top of the gameplay, usually of the player's face. This is much less enjoyable and relatable when the player's upper head (and therefore eyes, the window to the soul!) is covered by an HMD. With mixed reality, the player can use their whole body to convey emotion and show how they're interacting within the virtual environment to draw viewers in and create a more relatable experience.

Using OBS (or XSplit), you can easily switch between a mixed reality and 1st person scene to change up the perspective of your video. This is especially useful in gameplay moments where the player is solving a complex puzzle or minigame (ex. the battery charging part of Rick and Morty: Virtual Rickality) where the audience would benefit from having a 1st person perspective to better witness the action. Tilt Brush is another good example: when the artist moves in to do detailed work, it may be better seen in 1st person.

For game developers thinking about using mixed reality for trailers, look to the success of the Fantastic Contraption trailer. It's entertaining, easy for viewers to understand the gameplay, and characterizes the atmospheres of the game well. 

Mixed reality trailers won't be applicable for every game, but the bottom line is that mixing up the perspective in your footage is good. 3rd person perspective helps give potential players an understanding of how they relate to the game environment as a whole, as well as provide a sense of scale. If, for whatever reason, having a real person composited into your world doesn't work for you, consider using an avatar in place of person (think Kert Gartner's Space Pirate Trainer trailer) to get a similar benefit. Long story short, mixed reality shots show off the scale and context of the world, 1st person shows off the details. Switch back and forth when necessary to give your viewers both experiences. 

Another use for mixed reality is as a marketing tool at events. For devs looking to build a mailing list, it's a great way to collect email addresses from attendees by recording a video of a person playing and having them sign up for the list to receive it. You can also use mixed reality videos/gifs at events for a social media campaign and have attendees post the video on Twitter or other platform with a particular hashtag. We use the LIV cube for this as it's a portable mixed reality solution.

Ok I'm sold, let's do this! But....how?

Ay there's the rub, actually doing it. I mentioned I won't be going into too much detail myself on the how-to. That's a whole 'nother essay and I'm already working on an illustrated how-to, so I'd rather link to resources and open myself up for questions. You can find all that at the bottom in the additional resources section.

I will say mixed reality is expensive in terms of time, space, and equipment. If you're unable to make the investment on your own, look for a mixed reality studio in your city or link up with content creators in your same area to invest in a shared space together (shameless plug: if you're in NYC and want to mess around with mixed in our studio hmu.)

There's a number of things that can go wrong, and often do, but luckily as MR has become more popular there are a number of resources to go to for help and answers. Many mixed reality YouTubers have a Discord you can join, and you can check out the Steam or Unity forums for help.

There are also some solutions being worked on that can help simplify the mixed reality process. Two big ones are the LIV cube and Mixcast VR by Blueprint Reality. We use the LIV Cube in our office as it's both a hardware and software solution and it's saved us a lot of time in getting a greenscreen room setup. If you're a dev working in Unity, check out the Mixcast SDK and see if it's something you can apply to your game! Anything that makes mixed reality recording easier on the user will make your game more appealing to showcase by content creators.

So I can make a mixed reality video...I'm set4lyfe right?

NOOOOO!

Never let yourself be caught up in the "coolness factor" and rely on it to carry you to success. Enough people do mixed reality videos now that a single vid done poorly ain't netting you instant fame or getting your game to the front page of r/vive on reddit. Since this technology is so new, yes I've seen standards be a little lower than usual when it comes to MR vids - heck, I'm guilty of it myself. There've been times when my calibration gets a little off and I think "eh, it's ok, no one will really notice because my viewers think this is so cool." In some cases this may be true, but the technology is moving fast and standards are getting a lot higher, so it's better to be on the high-quality edge from the get-go rather than having to play catch-up.

By high-quality I mean a few things. I know that the equipment is expensive, so there are many things one can do to create a quality stream even without investing in the toppest of top quality tech across the board. In my experience, high-quality mixed reality streams are those that (in no particular order):

I said earlier this list would be in no particular order, but I lied, because this is my most important piece of advice. What I love to see an a quality mixed reality video is....acting. It is my greatest sadness when I see folks being static or emotionless in their mixed reality video. Not to mention, it's boring. Even though the tech itself is cool, you'll lose interest and engagement if you rely on that alone. What makes mixed reality videos so awesome is that you are adopting the presence of a main character in an entirely virtual environment, like a reverse Who Killed Roger Rabbit. I view it as a completely theatrical experience. Have your actions and reactions much more dramatic than you usually would, especially your arms. As long as your feet aren't glued to the floor you should be moving them. Dance around in Audioshield. Duck and weave in SPT. Crawl on the ground, if you're able. Heck, put on a costume and get really into it. 

I always use this video for inspiration as I think it's the perfect example of theatrical MR. I have no idea who the player is, but through this short video my mind develops a story about her based on the way she's interacting with the virtual world. It's compelling. I want more. This past E3 another video of her swept through my social media. Something about her and her interactions with the virtual world are clearly compelling to people, resulting in viral content - learn from this.

If this isn't something you're used to or comfortable with, try taking an improv course or watching some acting videos online. Also, check out comedians who rely on physical comedy for their humor and how they use dramatic gestures to draw in an audience. 

Devs looking to film trailers or viral content should look into hiring someone to be the main character. Consider hiring a dancer, actor, or cosplayer depending on the content of your game.

Other tips and musings

-Get a friend to help you. Our streaming team is two people: myself and my partner-in-crime, Cindy Mallory. One of us holds the camera, makes sure things aren't broken, posts on social media, and chats with the community if we are streaming.

-I've seen successful streams that have the mixed reality video going on at the same time as an interview with the devs seated nearby. This takes some pressure off the player to be super entertaining, since the dev interviews become the focal point. Game devs could use something this for video devlogs of their VR development process while showing off footage of their game.

-Mixed reality is possible in Unreal Engine, we've done it using a third-party software called Spout. The annoying part is you have to integrate it into the game from the backend, so unlike Unity games you can't just go around streaming any Unreal game using this technique unless they already have it built-in, and no game does. We'll be continuing to work on solutions for ourselves, but I hope Unreal gets around to integrating mixed into their engine someday!

-We've been livestreaming on Facebook, it nets a lot of views (more than Twitch or YouTube, for us) but the video quality is low. Facebook limits their livestreams to 720p and you can only download the streams in SD. I recommend recording on OBS simultaneously as you stream so you have a high quality video to pull from for GIFs or other edited content. Using LICEcap is an easy way to make quick, low quality, small size GIFs.

-Don't wear green, unless you want to be invisible ;)

Thanks for reading, folks! I hope I provided some helpful information on how to stream mixed reality, and how to stream it well. 

If you have any questions please feel free to ping me on Twitter (@rrbenett) or drop me a comment here! 

-Becky

Additional Resources!

Sarah Northway's mixed reality article: http://www.gamasutra.com/blogs/SarahNorthway/20160111/263285/Mixedreality_VR_Twitch_streaming.php

Mixed reality in Serious Sam VR: http://www.croteam.com/mixed-reality-serious-sam-vr-last-hope/

Mixed reality in The Unspoken: https://developer.oculus.com/blog/sharing-vr-through-video-mixed-reality-in-the-unspoken

Steam discussion on mixed reality: https://steamcommunity.com/app/358720/discussions/0/405694031549662100/?ctp=2

TribalInstincts mixed reality configurator: https://www.youtube.com/watch?v=OlwqAYKR_nc

Cix Liv on mixed reality with a Vive tracker: https://liv.tv/blog/mixed-reality-htc-vive-tracker

Dario Laverde's how-to on mixed reality: https://medium.com/@dariony/about-mixed-reality-and-a-how-to-part-1-28387e792a4

Google's "replace your real face with a fake face" tech: https://research.googleblog.com/2017/02/headset-removal-for-virtual-and-mixed.html

Owlchemy mixed reality with ZED stereo cam: http://owlchemylabs.com/owlchemyvr-mixed-reality-tech/

Kert Gartner's post on making high-quality mixed reality trailers: http://www.kertgartner.com/making-mixed-reality-vr-trailers-and-videos/

LIV Cube: https://liv.tv/

Mixcast VR: https://blueprinttools.com/mixcast/ 

LICEcap: http://www.cockos.com/licecap/

Green floor mats: https://www.amazon.com/IncStores-Exercise-Tiles-Portable-Interlocking/dp/B01FRAS8II/ref=sr_1_2?ie=UTF8&qid=1497293553&sr=8-2&keywords=green+floor+mats

Green vive covers: https://www.amazon.com/Hyperkin-Vive-GelShell-Wand-Silicone-Green/dp/B01LEZO2FS