In creating our new VR project Séance: The Unquiet, we set out to evolve cinematic storytelling for virtual reality to deliver something as close to the experience of watching a movie as we could but using the medium of VR to its best advantage.

For a hundred years filmmakers have crafted every shot to tell audiences where to look: from framing and composition to lighting and depth of field, every technique of film has been used to better guide the eye to the most important elements on the screen.
 
What we’ve done with Séance is to completely reverse that: we have crafted each scene dynamically to be aware of where the audience chooses to look. It’s a fundamental evolution in cinematic storytelling that requires the expertise and technologies of videogame development to pull off. (In our case, we used the Unreal game engine.)  From a grandfather clock that sounds different when you look at it to an intricately designed crescendo of music and sound design that rises dynamically in pitch and volume with the angle of your head as you turn to discover a ghost behind you, Séance shapes its experience to the audience’s gaze. The result is a new kind of storytelling for a new kind of medium.

In this series of blog posts I'll summarize these and other techniques of cinematic storytelling we have evolved and utilized in our project. If you want to see some of them in action, we have released a free five-minute preview of Séance for the Oculus Rift and the HTC Vive.

Other Blogs in this Series

Part Two: Music & Sound Design

Part Three: Character Presence

Part Four: Anatomy of a Scene

Project Assumptions

Because we set out to make a movie in VR and not a videogame, we had a couple of specific assumptions for our project. These don't apply to most VR projects, but they made sense for us:

We definitely understand the appeal of game-y roomscale interactive experiences and are not advocating for our approach as the one true way for VR. But when we look at the larger world of popular entertainment, we see an opportunity to take the seated, passive experience of watching a movie and reinvent it for VR. We believe this approach could eventually reach a huge mainstream audience while still requiring the tools, technologies, and skills only found in the videogame industry.

Part 1: Composition

In film, each shot is carefully composed in terms of both framing the image (what is in the frame and what is not) and how the content within the frame is arranged, lit, and photographed. Generally speaking, the most significant character or element in the shot is the one that is lit, framed, and focused to accentuate its presence and draw the audience's eye. In a complicated action sequence of many shots, this is critical to ensuring the audience can follow the important action beats.

How do you handle this when the VR audience's head is the camera and they control where they look at any time? Our answer has been to carefully place focal points in our scene at a variety of locations and depths and then treat each of those as a "shot" that can be composed, lit, and used to its best advantage.

In this way, no matter where our audience chooses to look we do our best to ensure that their gaze will "snap to" one of our planned shots and that we make use of each shot for specific moments. We think of the moments when the audience shifts their attention from one focal point to another as being analogous to an edit in a film.

The result feels surprisingly cinematic. The human brain does a good job of focusing on what seems important and ignoring what isn't, and the result is that when the audience looks at a focal point even in the far distance, it can still have the intensity of a cinematic close-up because the brain filters out everything else in the visual field.

The Master Shot

We started by building our master shot. This is what we consider to be the default view the audience will experience given their initial spawn point and orientation as well as their comfort (that is, it's more comfortable to be looking straight ahead). We doubled down on that by making the master shot the most interesting view with multiple focal points in the close, medium, and far distances.

Here is our master shot and the main focal points within it:

Séance takes place in a large mansion in India in 1935. While we do not establish the location or time period overtly, we have used architecture and decor to set the scene. Our setting here is a great hall within the mansion where the bulk of our action takes place. This environment was designed by our art director, Bruce Sharp with a lot of work from Stuart Cunningham, our environmental and technical artist.

There are four main focal points here:

(1) The table and the candle.
The candle is a key feature of Séance. We sometimes use it as an interactive element that can trigger events based on the audience looking at it. It also responds to wind, flickering when a breeze blows through the scene. To ground the candle in immersive reality while staying performant in VR we built a simulated dynamic shadow of the candleholder on the tablecloth because an actual dynamic shadow would have been too demanding on minspec PCs. The shadow is a layer within the tablecloth material with animated movement derived from the values of the particle system we used for the candle flame.

This dynamic shadow was designed and implemented by Stuart and by Rick Saada, our senior programmer. Stuart also created the intricate candleholder which is a physical prop we purchased at an import store and then scanned as a 3D object using photogrammetry.

In the Séance preview, you'll meet a character who takes a seat in the chair opposite you. That character is so close you can see every movement of their face, which we animated using a facial motion capture system from Faceware. This means we are treating the table focal point as our close-up shot within the master shot. In VR, when you can't zoom the camera in on a character, you have to bring the character to the camera.

(2) The gramophone.
This prop sits in the middle distance of the master shot. We use the gramophone as a source of diegetic music, meaning music that plays from a source within the scene. This makes the music highly positional, which increases the immersion and helps the audience know where to look for it. I'll get into the sound design of the music system in a later article but for now, I'll just note that shortly before the gramophone begins playing spooky music we bring up a small spotlight on it.

(3) The patio windows.
The patio outside contains both near scenery (the patio and its banisters) and far scenery (the tree line) to establish a sense of scale. This also frames the patio as its own stage where action can happen, as seen by the audience through the generously sized windows. Shortly after the music begins a large black wolf crosses the patio from window to window, his steps falling in time with the music. This focal point gives us another stage to set action on, increasing the complexity and depth of action wihtin our master shot.

(4) The foyer.
The large foyer at the end of the great hall is far enough and dimly lit enough to be a little unsettling. We designed the walls around the entrance to resemble the proscenium of a theater, creating a dramatic space. In the Séance preview, those double doors at the top of the stairs creak slowly open, the sound reverberating through the hall, and then soon after a ghost appears. The ghost soon closes the distance between the foyer and the audience to startling effect.

The Right Shot

We introduce our main character in the opening moments of the Séance preview. One approach would have been to just have him already in the master shot, drawing the audience's attention from the very beginning. But we wanted his introduction to have more impact and to even be something of a surprise.

To accomplish this, we spawn him to the right of the master shot just out of frame. He then draws your attention in a surprising way, by drunkenly firing a gun into the ceiling and shouting at you. The audience turns to look and he delivers an opening monologue that establishes our tone and catches your curiosity.

Here is our right shot:

There is really only one focal point in this shot (not including the candle and table). But there is also an anti-focal point which is just as important:

(1) The shadowy area in the far middle distance.

This anti-focal point between the foyer and the character's entrance area is purposely lit darkly. Nothing interesting is going to happen in that area.

The darkness here serves the same purpose as a camera edit in a film, by "cutting" the audience's attention from one shot to another.

(2) The character stage in the near middle distance.

For this focal point our character is framed by two columns and the area around him is warmly lit. This shot is about establishing our character's humanity, so we use warm lighting here as opposed to the cold blue moonlight of the patio and the distant foyer, both of which are where supernatural events will occur.

Character Head Tracking

This introduction scene is also the first time we use one of our VR storytelling techniques: character head tracking. While our character here was animated with motion capture technology -- both body and face -- by our technical animator Douglas Connors, we also dynamically modify those animations by directly manipulating bone rotations for the neck and eyes.

At specific moments of his monologue, we turn on character head tracking and the animated character then looks directly at the audience. No matter whether the audience is sitting or standing, short or tall, leaning this way or that, we can ensure that our character looks directly at the audience at key moments to punctuate his monologue. We'll discuss the technical implementation of this technique later in this series.

The Left Shot

In the second scene of the Seance preview, the gramophone begins playing spooky music and a black wolf stalks across the patio. The audience can follow the wolf as he walks from window to window until he stops at the last window and sits down.

For this scene, we have relit the mansion and it is now also raining. The warm light from the previous scene is gone and only spectral moonlight falls on the scene. This cues the audience that something supernatural is in store.

We have three main focal points in this shot:

(1) The first window in the near middle distance.

The wolf enters the scene here, walking along the patio. We don't mind if the audience misses his entrance because they're looking elsewhere. The wolf walks slowly enough that there is plenty of time for the audience to discover him. When they do, they keep watching to see what happens.

(2) The last window in the far middle distance.

When the wolf reaches the last window in the far middle distance, he sits down. The audience hears a persistent knocking sound from the right, where our character was introduced earlier. The wolf sits patiently, staring at the audience, until the audience turns their head to look at the source of the knocking. When they do, we despawn the wolf and replace him with a spooky demon character accompanied by a flash of lightning. This demon character holds still until the audience turns back to the left shot at which point we trigger the demon's animation and he begins playing his scene.

(3) The gramophone, also in the far middle distance.

The gramophone continues playing music throughout this scene but the music begins more distorted and mysterious as we continue. When the demon appears, his voice comes not from the demon but from the gramophone, an unsettling moment made possible by our spatialized audio. There is an invisible audio emitter that actually flies from the demon to the gramophone and back to indicate the supernatural connection the demon forms when he speaks.

Audience Attention

This scene makes use of another VR storytelling technique: audience attention. We know where the audience is looking at all times and we can use that knowledge to trigger events in Unreal. We've done a few different implementations of this technique, using both LOS intersection of volumes in the scene (when we want to know if they're looking at something specific) and also simply computing the rotation angle of the user's head (when we are dynamically modifying elements based on the user's rotational movement). Some are more performant than others and we'll share our implementation later in this series.

Conclusion

I hope you've enjoyed this initial look at how we used the cinematic technique of composition to create Séance: The Unquiet. In part two, we'll continue with a discussion of how we used cinematic music and sound design to tell our story.