Hollywood Sound: Part Two

Surround sound, as a concept and a format, has been around for nearly as long as sound itself. Indeed, there seems to have been an early desire to immerse the viewer within the picture. After almost a hundred years, the surround formats have settled down into recognizable consumer formats and brands. Game audio comes to the scene ready to inherit all the technologies and expectations of motion picture surround. In the second part of our look at Hollywood integration into game development, we examine how theatrical and consumer surround sound is used, and how it borrows from, and looks beyond, its filmic antecedents.

Games have adopted a slew of branding from cinema in terms of surround sound: Dolby, THX, DTS. This makes complete sense as it represents a standard to which video games content can adhere, provides consumers with recognizable logos in terms of quality cinematic sound and provides compatibility and support from systems already entrenched in the home cinema market that games consoles can make immediate use of. There will probably never be the development of a grassroots surround format exclusively for games, although we may see some of the more cutting-edge developments for cinema surround getting more attention from gamers than from a film audience.

Broadly speaking, video games are about physical immersion in a universe of action and reaction. Films, however, are about emotional immersion within a narrative (1). New technologies that allow further realism and higher resolution in terms of physical immersion will be better received by video gamers than cinema-goers. On the whole, surround sound does not facilitate a huge leap in the immersion of an audience into an emotional narrative; if one looks at early sound film, such as Fritz Lang's M, the use of one mono sound channel creates a very complex and layered world of narrative associations and emotional cues which are as compelling as any modern day surround sound picture. However, in films such as Apocalypse Now, much of the subliminal, poetic narration would be impossible on a single channel.

First Wave Consumer Integration into Surround

In the last generation of consoles, a significant proportion of the game playing demographic did not own surround sound systems, even though it was still considered as an area rich for exploitation on the part of content creators. An initial wave of purchasing occurred when the new consoles came onto the market, predominantly by a middle class who had considerable disposable income. This initial wave likely purchased a home movie surround set-up into which they could integrate their consoles and DVD players. This initial wave of buyers potentially deemed whether a particular console, and console titles, gained commercial and indeed critical success in order to enable the machine to be marketed to the second, third and forth wave of consumers who, in all likelihood, did not own surround sound systems, but conventional, often mono, TV sets.

The success of the DVD format, for both games and movies, can be viewed as an encouraging and pre-emptive mass consumer move into the surround sound market, as certainly prices of DVD systems and the surround speakers and decoders themselves, are something that are now being offered at affordable prices. It can therefore be safely assumed that the initial wave of surround ready games bolstered a critically successful environment which has supported demand for more affordable versions of this technology. This first wave of games (Medal of Honor, Halo) has also been able to set the critical and aesthetic benchmarks. This is how the integration model worked for the first generation of surround sound consoles. With next gen consoles, surround is a given; it is expected. Anyone with a home entertainment system will expect their games to sound as cinematic as the movies they watch, if not more so.

Immersion and Emotion

In terms of aesthetics, looking towards the Hollywood film industry for precedents and parallels can only paint part of the picture. As has already been discussed, games represent physical immersion in a universe of action and reaction. Given these terms, there are certainly closer and more recent analogies for surround sound that bear closer artistic sinews to game-surround. These parallels are the ‘IMAX' and ‘ride film' presentations, which take immersion through surround sound as fundamental to the overall experience they are selling.

IMAX

The theatrical desire for giant-scale cinema started in the early 1950s in response to the threat of television. Cinerama, unveiled in 1952, used three projectors and three screens. In 1953 came the ‘wide screen' formats CinemaScope and VistaVision, giving birth to a whole slew of formats such as Technirama, Panavision, Superscope and many more besides. IMAX grew from a team of Canadian film-makers creating large format films for Montreal 's Expo 1967. By the time of Expo 1970, in Osaka , the IMAX formula had been perfected. Since it's inception in the early 1970s the 15/70 large format film medium has been at the forefront of a gigantic immersive entertainment experience. The sound is mixed, usually in situation, in gigantic cine-sphere auditoriums.

Tim Archer, has created sound design for, as well as mixed large-format film for over 20 years, creative director of Master's Digital, based in Victoria, Canada. (2)

“Technically speaking, IMAX theatres have 6 full range speakers. There are 3 across the front screen (LCR), 2 point-source surround speakers and 1 speaker on top of the screen for height. A typical theatrical 5.1 set-up will use smaller speakers in an array as surrounds. Bass management also differs from conventional cinema surround. The subwoofer information in a 5.1 system is a physical track containing all the low-end, while in an IMAX theatre, the subwoofer information is derived from the 6 main speakers. So all low-end is filtered from the 6 main speakers and routed to several bass cabinets in front of the theatre.”

The complete immersion of the senses is the primary goal for these large format experiences. The space defined by the screen is intended to reach into and beyond our peripheral vision, something which the normal movie screen doesn't do due to its clearly composed rectangle. Many IMAX films and cinematographers will compose their shots with this periphery in mind, allowing the viewer's gaze to explore this vast space, rather than absorbing the image in its entirety. Film sound has always enabled the experience of ‘periphery' events, this is evident even in the films as early as The Jazz Singer (1927), in which we can hear the audience's reactions to Al Jolsen's performance, off screen, therefore creating the ‘imagined space' beyond the rectangular screen. Even in pre-sound film, the imagined space outside the frame was referenced through use of on-screen mirrors, reflecting a space behind the camera's position.

The philosophy of ‘large format' sound, in particular surround, is also carried through right from location recording, to the final edit and mix.

Tim Archer:

“I find the Large Format film genre more challenging and interesting to work with than a standard ‘dialogue driven' feature film. On a feature film, the location recordist is mostly concerned with recording clean dialogue, whereas my job on a Large Format film is to capture multi-channel atmospheres. These recordings are the initial fundamental layer of the sound design. I also try to capture as many indigenous sound effects as possible, to give the track a real ‘sense of being there'. This same philosophy carries through the sound edit and final mix. I will edit and premix in a large mix room then, for final mix, I gather up my gear and set up and mix right in the IMAX theatre. It makes a huge difference watching the soundtrack against the enormous 6 story screen. Obviously, each museum, theme park and science centre has different speaker configurations and acoustics and can only be mixed at that exact location.”

Surround sound is so much an integral part of the IMAX experience, utilizing all the speakers equally, not to mention the height speaker, in order to create the feeling of depth, height, immersion and spectacle.

Tim Archer:

“The IMAX picture is so large and crystal clear that the soundtrack has to equal this. The smallest detail can make the difference between sounding real or artificial. You are also attempting to immerse the audience in a realistic environment with layers of atmosphere tracks and well placed specifics, by using all 6 speakers in conjunction with each other. The surrounds and the height speaker are just as important as the front 3. This approach is quite similar to creating “soundscapes” for location-based venues like museums and science center exhibits. You are looking to simulate reality, by strategically hiding speakers behind fake walls and props, to add a reality to static images.”

The spaces around the edge of the IMAX screen are not visually composed, as would first be assumed, with the intention of being a vague, empty ‘periphery' space. The dimensions of the IMAX screen actually allow an audience to ‘look around' the screen, breaking much of the directorial need for an audience's attention to be focused on one single image after another in a standard narrative sense. In this respect the films do bear an element of visual free interaction, but the narrative is better compared to a ‘ride' whereby there are any number of things to see at any one time, and each viewer's experience of the film will be markedly different from the other's based on what they observe at a particular time.

In a film format as big as IMAX a strange reversal takes place in that the sound takes the directorial seat, focusing people's attention on certain parts of the screen, or via an omni-present narrator's voice. In conventional film, sound is used to provide the ‘periphery' and the immersion. In IMAX the periphery is visual, the focus is lost and the sound must lead. This occurs through the spatialization of sound via the surround format; if we hear something unusual over to the left, our attention will be directed to the left hand side of the vast screen area, and so on. Speakers behind the screen allow this to be ‘spotted' very accurately. The rear surround speakers still perform the old function of creating a ‘periphery' as there is no screen space immediately behind the viewer.

Games certainly mirror this ‘directorial' use of sound in that the sounds of objects or enemies can often only be located by the use of surround sound. In the confusion of a huge onscreen battle, one of the ways of navigating through the chaos is to utilize the surround audio field in order to reveal where the next objective is. Much of the use of this is currently ambient in nature in gaming, due to smaller screens and also the fact that positionality cannot be used to its full potential while there are still gamers playing games on stereo or monaural sound systems.

The other main goal of an IMAX presentation, of course, is that of ‘spectacle', taking to extreme the sensations experienced by the first audiences of Lumière's Train Arriving at a Station in the 1890s who ran screaming from the auditorium believing the train to be coming right at them. Essentially at the core of this experience is a need to blur the lines of created and imagined space.

The IMAX film has to compete, like the mainstream theatrical cinemas, with the phenomenon of consumer DVD, which now account for a huge slice of the profits of any movie. IMAX has begun several strategies to broaden its appeal. For over ten years the company has been actively encouraging major Hollywood studios to adopt the IMAX format, with limited success. Now, though, with DRM technology, it is possible to re-master conventional films for the IMAX format. It is costly - every frame has to be enlarged from 35mm prints and manually cleaned - but this has proved effective in reviving the format. One of the biggest hits in IMAX cinemas was The Matrix Reloaded. In addition to this, IMAX has developed a smaller, cheaper large-format system for conventional cinemas.

As video games make the shift ever closer towards these larger formats, and as our homes embrace larger screens and more powerful sound systems in order to aid immersion, the audio will be able to fully switch from ‘periphery' descriptor to the ‘director' of our attention.

Ride film: Shock and Awe

The ‘ride film' is a relatively new entertainment medium, originally born out of the flight simulators used by the military, this has in turn given birth to an entirely new immersive and participatory entertainment industry.

Very often the ride film will tie in to a franchise or film license and extend a particular franchise's world vision, as in the case of Star Wars, Terminator, etc. In this respect, it also bears a great similarity to the extended narrative ideas of game franchises.

Audio is generally highly ‘magnified' and physically experienced way beyond reality in both ride film and IMAX film. Indeed, it could be said that many of these films could also learn the lessons of tension and release in producing a more convincing immersive experience. However, this is about entertainment, short, sharp and shocking. It is this where the subwoofer becomes the weapon of choice in the armory of the sound designer, to provide the necessary shock and awe for the audience, who generally are there to be ‘wowed'.

Tim Archer:

“Sound design and mixing for a ride film is more about high impact, fast moving pass-bys and letting go of reality. These ride simulators are usually set-up in a 5.1 configuration. The main difference is in the room itself. Not only are you dealing with a not-so-perfect acoustical environment, as in the IMAX or feature type theatres, you are also dealing with a large, radically moving, loud seating platform. The main approach is an ‘everything louder than everything else' type of entertainment. The challenge is choosing and blending sound effects that can create the right impact.”

This certainly explains much of the need for the loud volumes present in such rides; there is a certain amount of mechanical noise generated by the apparatus that needs to be covered up first in order to suspend the disbelief of the audience. Tension and release are the agents of narrative design here in the traditional ‘roller coaster' sense. Ride film narratives generally hit hard and continuously, almost deranging the senses with an onslaught of sound, immersive computer graphics and gut wrenching physical simulation. We are not talking about a subtle medium here. Yet in order to get the most out of this they do need to build up tension and there are moments for reflection and enjoyment of a floating sensation in the Star Wars simulation at Euro Disney. The use of surround in this medium is used more extensively, not to direct our attention as in the IMAX format, but to derange the senses, to confuse the viewer's sense of spatial awareness. Use of subwoofer for sheer physical impact is nowhere better illustrated and synchronized with both onscreen action and physical movement.

In terms of an overall aesthetic, video game and the ride film share an obsession with physicality. From rumble packs to the ‘derangement of the senses' experienced in such games as Medal of Honor. This is achieved through use of the surrounds and the sub channel. Currently the .1 channel is non directional, which adds to a loss of directionality associated with low-end. This is how a game will sound ‘cinematic' to most audiences. In video games, the sub channel will be used so much more than in a movie mix. This is generally tied to action in the game, and is also used as a physical reward for the player's interaction.

Other sections of game audio are side by side with the ride film in terms of audio aesthetics. Although these depend, of course, very much upon the genre; a third or first person action shooter will make more use of sub than an RPG. Shoot-em ups may play on either a tension and release narrative structure, one subtle build-up and other part all out kill-death frenzy, or they may opt for the full out continual derangement of a ride film, in which case prolonged immersion may become a problem for the player. Ride films are designed very differently in terms of their experience. They last little more than five minutes after what may have been 2 hours of queuing up to get on the thing. In many ways even the queuing up and waiting for the ride can be mirrored in the tension and release of game play: lots of sneaking around, thinking, planning, and then five minutes of hardcore action as in latter moments of Medal of Honor. Higher quality home sound systems will enable developers eventually to better map and mix these peaks and troughs, to deliver more impact or subtlety where it is needed.

In terms of mixing content for theatrical and consumer surround, the main difference is the uncertainty of the end system as to which the game will be heard on, although THX are making tentative moves into the consumer surround market with their categorization of quality sound systems. With IMAX you have to mix in-situ on an equivalent size stage and this guarantees it will sound at its best in performance. In terms of surround the confusion of multi-point sounds in a huge battle is already up there with the ride film. The only thing needed is the assurance that the delivery system is going to provide an accurate reproduction of the mixing environment.

The major presentational problem between game and film is that films inhabit both theatrical (cinema) and consumer (DVD) spaces, while games inhabit only consumer spaces.

Pushing the Consumer Sound Envelope

While surround sound now has a solid install base of recognizable formats, the field of surround sound is anything but static. Research and development into increasing the resolution (number of speakers) of the surround field (also known as the ‘audio pixel'), as well as the realism of the reproduction of sounds in space, is steadily moving forward. There are inherent problems in getting both theatrical exhibition spaces and consumers on board with any new and improved immersion format. Few consumers understand the need for higher resolution surround images, which is why the cinema of the spectacle will probably be the place where these new formats will be introduced. The integration of games consoles into the consumer surround ‘package', integrated with DVD movies and HDTV broadcasts, means that there will probably not be any improvements in the resolution of consumer surround sound without that shift being available in all three areas; games, movies and TV.

Game developers are often challenged to create a more ‘cinematic' sound experience for gamers. Part of the problem is understanding what this actually means in terms of production. Many game developers still eschew audio to be the most neglected component of their development, mixing on consumer systems or creating sound effects in poorly isolated rooms full of computers and hardware that emit high levels of fan and cooler noise. These things alone raise the noise-floor of the development environment and inhibit sound designers from making critical decisions about the sound in games. Again, THX is doing what it can to educate and help game developers, however it is severely cost inhibitive, and these higher production costs are something that only the giant publishers are able to afford and justify.

Having the game content mixed properly is essential in a final post production phase of development. Again, this should be done in a correctly calibrated surround environment, preferably of THX standard. Currently every developer and almost every title have differing approaches to this, while it is true to say that motion pictures also have varying and differing mix strategies, the latter tend to be artistic decisions made by choice, not the bi-products of radically different mixing environments and approaches.

With next generation consoles all being fully compliant with Dolby Digital's discreet surround 5.1 format, the emphasis on mixing and surround in titles is going to become more apparent. These movements will be partnered with integration into the latest surround formats of film and TV. Whatever moves the consumer surround market makes, video games will need to, at a minimum, keep up, but where possible, innovate.

With this in mind, we can already see the move into the realm of 10.2 surround sound, in which greater emphasis is placed on the sub channel, and the resolution of the audio pixel in terms of surround channels. Tom Holman's TMH Corporation are at the forefront of this research. (3)

Immersive Audio Labs at the University of Southern California are also at the cutting edge of this research, part of their Mission Statement reads:

“While DVD and high definition television deliver 5.1 channels of sound to consumers today, in the future these systems will seem as antiquated as the monophonic recordings of Caruso, which were created in the early 1900s. Certainly better than 2 channel stereo, 5.1 channel audio is not completely convincing-it is easily distinguished from the real thing.

Research in the IMSC Immersive Audio Laboratory is focused on algorithms for capturing and rendering sound so that it is indistinguishable from reality. Our goal is to provide an immersive experience through greater imaging and envelopment capabilities than ever before. The unique component to our work is a comprehensive synergy in the understanding of acoustics, psychoacoustics, recording technique, and adaptive audio signal processing. Our hope is that this kind of immersive experience can become practical for various playback environments including movie theatres, home theatres, headphones, and even the desktop computer.” (4)

As part of this is an extension of the THX calibration process into home listening situations, it is almost pointless mixing and designing sounds in a THX designed and calibrated studio if that content is then to be played back on an uncalibrated system. Using high performance speakers and amplifier components is only part of the equation. The radically differing acoustics of consumer and professional studios alike causes radically different listening experiences. Calibration of the speaker system and EQ to match the room the sound is being heard in, is a fundamental problem associated with the THX specifications. Audyssey Labs' MultEQ (5) represents a move towards resolving this, using an ‘auto set-up' that automatically determines how many loudspeakers are connected, whether they are in-phase, satellites or subwoofers, it then analyses speaker level, size and distance in order to give every seat in the house the ‘sweet spot' effect. We will undoubtedly see more of this type of calibration in the home entertainment arena, undoubtedly led by the middle class consumers who can afford such items as Bang and Olufsen's BeoLab 5s. This not only comes with the economic limitations of the speakers themselves, but also with similar economic limitations on affordable housing in which these sound systems can be turned up loud enough not to annoy closely situated neighbous. Subwoofer bans in condominiums are surely a fast approaching widespread reality. For the majority of consumers, this ‘lifestyle' is simply beyond their affordability.

The experience and spectacle of viewing a movie in a cinema will never go away, nor will the larger format ‘spectacle' entertainments as IMAX and ride film, however, the home entertainment arena, through DVD and digital content, is where the battle for surround formats will be most fiercely fought. Games will likely play a role at the forefront of this battle, and this will be achieved by quality of content. This means cinematic surround mixes, a ride-film-like assault of the senses, as well as direction of theaction itself through surround sound.

Next up: Part Three - a look into writing inside and outside the game, voice content of video games, and the move into using Hollywood talent to capture authentic performances and stories.

Notes:

(1) This is not to say that this will always be the case, there is a larger debate about ever having a video game that creates such emotional response that it will make you cry. It remains to be seen whether this will ever occur in exactly the same way that a filmic narrative engages the emotions.

(2)  Interview with Tim Archer by Rob Bridgett August 2005, Master's Digital http://www.mastersdigital.com/

(3)  Tom Holman developed the THX sound system while at Lucasfilm Ltd. http://www.tmhlabs.com

(4)  Immersive Audio Labs Website

(5)  Audyssey Labs' MultEQ system, http://imsc.usc.edu/news/releases/audyssey.html

______________________________________________________