Re-Imagining the Sound of Pre-School Games

Summary

This is a brief (ok, not that brief) post-mortem on how the development team at Clockwork Fox Studios approached the integration of sound in the pre-school math title – Zorbit’s Math Adventure (iOS / Android). As well as some areas of in-depth detail, I hope to cover some of the broader thinking behind our sound choices, and in particular, some of the specific ways we wanted to innovate for an entry into this genre. I hope some of these ideas will be useful for developers tackling, or considering, similar challenges, but I’ll also scratch the surface a little more deeply, as underneath, this dev-story is as much about being a Dad, and re-discovering the motivation for wanting to make games.

Some Background

I’ve been a full-time game developer for close to 13 years, and for the majority of this time, I’ve worked on what would probably be called triple-A, large budget, open world, multi-platform titles with team sizes well in excess of 100 developers.

In January, 2013 my family and I made the ~7,000 KM journey from Vancouver to beautiful St John’s, Newfoundland, Canada (January, it turns out, is the best time to move to Atlantic Canada, because the weather only gets better from there on out!) where I joined Clockwork Fox Studios to help develop, design and launch both the new studio and game franchise, Zorbit’s Math.

So, why the change? The developer, team, and project could be seen as the antithesis of a triple-A studio - a small, agile group of around 15 developers working on a pre-school math title.  For me, this represented an amazing opportunity to experience development outside of the triple-A space – on a project and team with different values, different tone, scale, tool-sets and processes, and a completely different audience on which to focus the player experience.

My attraction was also a belief in the studio’s core goal and principle, which is to create meaningful experiences in which kids are having so much fun they don't even realize they are learning.

One of the most important aspects of this gear-change for me, as a recent-ish parent, was that it meant that I could not only make games that my kids could play, but that were designed especially for their age range. (In previous studios, I had to be careful of my kids even coming into work in case they saw the concept art on the walls).

As a result of this shift in audience, one of the biggest motivational differences as I worked on the game, was being able to personify and empathize with the intended audience in such a personal way, and for me this was a dramatically different game-making experience: I cared about this audience in a very  direct way through my own kids. In a sense, for me, the game became a labor of love for my own kids, to help them learn math, listening skills, to have a positive early experience with technology and, more even more than that, to simply make them happy.

Opportunities

So, how did this boil down and translate to the actual work required? In terms of scale and team size, this was going to be a very different, and more hands-on, personal approach from that of a large triple-A production team. Having said that, many aspects of approaching sound from the high-level remained the same. As with any project, large or small, one of the first challenges was to identify opportunities for the game. What areas were there for innovation? What could be the big wins for both sound and design? Almost immediately, upon looking through early documentation and playing through competitive titles - though there are a few exceptional developers who are approaching this genre in an innovative, caring and focussed way (Kidcore, Kidaptive, Toca Boca, and Sago Sago, to name but a few) - there appeared to be incredible opportunities not only for the sound quality improvements, but more especially the overall integration of sound and design seemed like huge missed opportunities in this genre.

The one thing that stood out again and again was the integration and consideration of sound for the pre-k audience. Sound all too often felt like it was not integral to the overall experience, but an afterthought, in terms of both design and execution.

The sad truth is, sound couldn’t be more important to interactive experiences aimed at this age-range, and this requires an approach that put the child first in many design considerations. Not only that, it felt that a quality and attention to detail, for want of a better term: a craft aspect, the aspect that connected the maker to the user, was rarely present.

For me, anecdotally, this approach reminded me of my Dad hand-making me a toy garage for my 2^nd birthday – all the carpentry, careful attention to detail, the choosing of the wood, materials, cutting and crafting those materials directly, the painting, finishing and TIME that was put into making something by hand for your child. This idea, or approach to quality (a time and craft approach rather than just a $ spend approach), was for me a motivational touchstone I kept returning to as we worked on not only the sound, but also the overall end-experience of the game.

Very early on, dialogue quickly became a focus for us. 3 to 5 year-old players (generally) cannot read without a great deal of help from a parent, and this was especially significant for our design approach in that nearly all direct instructions to the player would be conveyed through character dialogue. Spoken dialogue was also an incredibly valuable learning aide, in order for children to vocalize and learn the sounds of the names associated with the shapes of the numbers and concepts they were discovering and playing with.

Though dialogue was one of the more obvious area for improvement in this genre, another clear opportunity was that this genre required a similarly careful approach to overall loudness and mix as I would consider any console title. Having finished the game, I’d go even further and say that games made specifically for children require an extra special amount of finesse and care to accommodate the young and sensitive ears of the target audience. To achieve this on our end, the player experience had to be cleanly focused, balanced and delivered through a tiny mono speaker, and the overall choices, and mix of, all the sound elements played a huge part of delivering something that was clear, friendly and enjoyable.

Dual Audience

Being a parent gives you a completely different insight into game culture, and as a developer I think this insight is perhaps even more acutely felt. I knew first hand, from experiences of being the person that stood between my kids and the content on Apple’s app-store, that developing this title wasn’t a simple case of appealing solely to your player. Put simply; our target ‘player’ actually had no financial autonomy.  We really had two target audiences - the children who would be playing (and being entertained by) the game (the player audience), but also the parents who would be sanctioning the game (the gatekeeper audience). (note: our 'parent/child' focus for this game, was a very different audience and design problem to a 'teacher/child' focussed experience)

Content Filters: All content needs to consider the player audience, but also appeal & function through the lens of the gatekeeper.

We felt we couldn’t neglect either of these audiences in content choices or the overall feel of the product. Identifying and keeping both the player and the gatekeeper in mind when thinking of either the tone, or features in the game was an essential design (and parental) gut-check. With this in mind, the game’s entire sound design needed to appeal to parents in terms of safe, positive fun characters, concepts and educational value, while at the same time, appealing to kids of any gender in terms of fun, accessibility, entertainment and an extremely pleasing, non-frustrating user experience.

Back to Dialogue

As mentioned, the quality of the dialogue content (writing, recording, acting, and implementation) in this genre as a whole, at least on mobile, felt very low. This was the area I felt we needed to hit the hardest in terms of producing something that not only worked with the design, but, thinking back to that toy garage that my Dad made for me, that I would want my kids, and friend’s kids, to listen to and enjoy.

The production approach I adopted in terms of quality, was to think of this not as an ‘app’ at all, but as a math-themed kid’s TV show that you might want to watch on the Treehouse or Nickelodeon channels. Having made this theoretical distinction, this felt like more of a competitor to square up to and research in order to drive quality. Kid’s TV also happens to be one of the places that I’ve found so much innovative production design, writing and performance happening right now.

Consistent dialogue levels, clarity and performance were especially important for our design in terms of communicating the educational and instructional aspects of the experience so audibility and intelligibility at all times were critical.

Also, from a performance standpoint, these elements had to be balanced without compromising any cuteness, silliness or entertaining humor that the characters have, so these elements were always the two parts of our balancing gut-check.

The cast of characters

Production value (and by this I mean 'quality' in the sense of early integration, time, and iteration more than simply dollar spend), something that would be an expectation in a triple-A production, was sadly not an expectation in the crowded mobile space.

Casting and recording also became a core focus quite early in the project. For help with this I turned to another recent parent and good friend, Rob King of Green Street Studios. I’ve worked in the past on a fair few console productions with Rob, and we already have a huge amount of mutual trust, so I knew we could hit the quality we wanted to bring to the game. Casting went exceptionally well, and we received and listened to over a thousand auditions (recorded specifically for us – not just demo reels) from several LA-based union agencies to cover the six speaking parts (characters). It turned out this approach was awesome value and was crucial in terms of getting exactly the right voice types and performances for the roles.

The casting process was also something that the whole team could easily contribute to (and get excited about) by dipping into the auditions folders, selecting their picks, and placing them into a shared 'finals' folder. The quality was generally so high, that the finalist’s folder ended up being very large, with at least 20 to 30 really good choices for each of the six characters. Eventually the leads group locked themselves in my studio and we narrowed the choices down to the final six that we have in the game.

The final part of the dialogue equation was hooking all the dialogue events up to play back in the right places in the game, which was done over the course of a month using all placeholder content recorded here at the development studio. This was another critical step in terms of helping production, in that all that remained to do once we had received the final edited voice-over from LA, was a simple file replacement drop. Each character had somewhere between 100 to 300 lines of dialogue, which totalled around 1,200 lines for the entire game - not large for a console production, but certainly weighty for a mobile title. (Around 40MB for all English dialogue encoded/optimized in Vorbis at 80% quality)

Sneakily Promoting Good Listening Skills

This brings us to the presentation of the dialogue within the game, and another of the innovations we wanted to push with interaction design in this genre, is that as well as Math skills, we are also very consciously, encouraging good listening skills.

As our prototyping with placeholder dialogue got under way, when to listen and when to interact turned out to be a pivotal element of the overal flow of a kid's interaction with the game. In many games that we played for research, the player was almost always allowed immediate interactivity and could complete tasks before the spoken dialogue had even finished giving the instructions. This meant that the dialogue had to then be interrupted or skipped when the player reached the end of a task too quickly. Another issue being that dialogue cues were often re-triggered multiple times by pressing an object repeatedly. Whenever you interrupt dialogue in either of these ways, it leads to a cut-off, or overlapping, interrupted, 'unfinished', experience for the player, which makes instructional dialogue seem as though it were at the bottom of the informational hierarchy (making verbal and associative learning very difficult), whereas we felt it needed to be right at the top.

When to listen and when to interact.

Our simplified approach, the abolishing of interruptible (important) dialogue was an area we all felt strongly about from a presentational and learning viewpoint. From a production standpoint, it meant any extraneous dialogue content that was neither informational nor entertaining was removed (edited out), so it put pressure where it should have been - on getting the content right. What we ended up implementing instead of a fragmented listening experience for the player, were clearly defined moments for listening (non-interactive), and clear moments for interaction that are strongly re-enforced through visual elements such as interactive objects that only glow, or only show up on screen, when interaction is allowed. This approach also showed us that any dialogue that was really not important, like random ‘well done!’ style hints part way through activities, intruded on and slowed down gameplay and these were quickly removed. This approach was a great way to make dialogue register on everyone’s radar and the whole team could then assess the ‘value’ of any line of dialogue by its impact on gameplay, interaction and flow. This structural rigidity also really helped create a clean interplay between listening and playing.

Another important aspect of the overall listening experience was for us not to lean entirely on a wall-to-wall music model. In the very first tutorial level, which teaches the basics of tapping and dragging objects, it felt like music (score) nearly always got in the way of what was being explained and experienced. It also didn’t fit the scenario of the main character who is sleeping and needs to be woken up by the player by making lots of noise, so the simple removal of the ‘score’ left a lot of space to be more focused on the sound and dialogue that was important to that level. An ever-present ‘music mute’ button was also introduced and presented on levels with music; we felt this was an important option to help ‘de-clutter’ the soundscape if extra focus and concentration is required at any time by the children playing the game, or by parents within earshot.

The interactive music itself was designed and composed following the simple idea that it had to be able to ‘get out of the way’ of important dialogue, rather than relying of ducking it out in the mix. A simple interactive switching system (not dissimilar to that found in a lot of console games) was designed and implemented whereby whenever there was important dialogue, the music would switch to a set of calmer (but musically related) cues, and whenever activity/input was again expected, the music switched to the more active cues – this further helped underline and signal the distinction between listening and activity.

(Interactive Music Switching Flow, based on foregrounding important dialogue, a method of mixing through interactive music design)

A Safer Approach to Headphones: Ignoring Headphone Use Is Not An Option!

Headphone use was initially off the table as far as I was concerned for a children’s mobile game. However, the more I thought about this, the more I realized that parents do sometime have no other choice than to allow their children to use headphones.

(Headphone safety mode flow)

Now, I should say I’m probably one of the most overly concerned parents when it comes to hearing and listening levels of games my kids play, and I obsessively check the volume of the content that they are listening to. On our flight over from Vancouver to St John’s, one of the first things I found myself doing was giving my oldest son my headphones. So, I thought if I’m breaking my own rules here about headphone use when it suits me in certain social situations, then other parents are very likely to do the same when they need to. So rather than punish and judge parents in this situation by not addressing or supporting headphone use at all, it became one of our defining feature innovations.

(When headphones are detected overall loudness is reduced by -5dB)

My concerns were all based around loudness, and the thought that, if left unattended, my kids would pump up the volume to intolerable levels and damage their hearing through long term over-exposure – without me even knowing about it. I realized that, though I have no control over the loudness levels that other developers release their games at, here at least, I had an opportunity to stop that from happening by introducing a very simple ‘headphone safety mode’ in which the overall gain of the sound is reduced by -5dB whenever headphones are detected (accompanied by a quick visual icon).

Of course, with headphones on, our game will sound relatively quiet compared with other apps. My hope here is that we can at least start a discussion about this difference, and begin to advocate and raise awareness among parental consumer groups as well as kick-starting more awareness among kids game and app developers who want to accommodate and respect the more sensitive ears of the younger gamers. I noticed that even the volume of the speaker on the iPad and iPhone can often be too loud and distorted, so after a lot of experimentation, these are the numbers I chose to mix to…

Speaker: (At max device volume)

-23 LUFS Integrated

True Peak -2.3dB (after a half hour of gameplay)

-11.5LU Loudness Range

Headphones: (At max device volume)

-28 LUFS Integrated

True Peak -4.1dB (after a half hour of gameplay)

-15LU Loudness Range

Ordinarily, we’d be looking at a -16LU output for a mobile game speaker (with no attenuation for headphones), but this simply didn’t address a unique dual user base of concerned parents, and children that are less capable of assessing and setting a comfortable listening level on headphones, or through the device speaker.

As a side note: Detection of headphone notofications on iOS and Android devices is relatively trivial in code, as are notifications for Airplay mirroring in iOS, which make the creation of several different mix volumes (not to mention EQ & Compression if using middleware) for different outputs absolutely worth the small investment in time. For future children's titles I am looking at -16LU for device speaker, -23LU for Airplay and -28LU for headphones.

The Mix

For the final mix of the game, we had no dynamic ducking via the basic Fmod Designer feature set, but being able to connect to the iPad with Fmod and tune all the events at run-time made life very easy in terms of tuning. Even though we could have invested some time in developing simple dialogue ducking, we decided not to in favour of making smarter decisions with the content. This made the approach to the mix somewhat more simple, and meant that all content, especially music, had to be created and balanced around the idea that dialogue, when present, would always be the focus. This made composing a lot easier in terms of defining the relative bandwidth of the active (no dialogue) and inactive (dialogue) sections of the interactive score. It also felt very much like mixing in the PS2 / Dreamcast era, where 99% of the mixing happened at the asset level.

Put simply, the mix was thought of as there always being a ‘lead’ instrument – often times this was the voice of the characters, and, as mentioned, when the characters were delivering crucial informational dialogue, the music would transition to less busy cues to get out of the way. This single-sound-group focus was critical to achieving a cleaner mix and also a dynamic hierarchy of listening and interaction. When the player was no longer listening to instruction, but interacting, then the sound effects associated with their interactions naturally became the ‘lead’ and main focus. By having a design that shut-off player interaction during critical dialogue, we also solved around 95% of the dynamic mix issues (multiple events occurring at the same time requiring a mixing hierarchy) that typically occur with more complex games.

Mixing Zorbit’s Math Adventure in Fmod Designer

Final Thoughts

Being able to identify personally with my audience really helped not only provide inspiration, a gut-check and motivation on the creative side for the music, effects and dialogue, but also in terms of having easy and quick access to the intended audience at home to try out ideas. However, something I had realized during development was that having kids in the age range of the audience at home didn't automatically mean I won any design point… or make me an expert, but it did give me a solid and valuable insight into the dual audience and gatekeeper role that parents play, and also helped to have the intended audience always in mind. Another note here is that ‘thinking down’ to the kid’s level really isn’t useful either - we found you still have to make a fun, satisfying, coherent, engaging experience that as a developer you have fun playing and pride in making, then let your audience experience it hands and ears-on via play tests, and make tweaks from there. Going back to that dual audience, if the parents have fun playing the game, then they'll more likely buy it for their children.

For more empirical data we leant heavily on play tests during Alpha (though we could have done more, earlier), in which team members could observe kids actually interacting with the game. These were inspirational and invaluable in showing us where user experience could break down and where simple tweaks to interaction problems, or dialogue hints, could vastly improve the experience.

Looking back on my experience of working on this game, I’m reminded of a quote from a GDC 2013 Advocacy lecture by Coray Seifert (Slingo). That “… making games is important stuff, it makes people’s lives better.” The bigger context here being about why we do what we do as developers - At first this quote didn’t fully ring true for me, but, it stuck with me, and, over the course of development, I began to rediscover this very simple motivation behind why we do what we do and why it is so important. Nowhere did this feel more important than applying some of the more hands-on, craft-oriented approaches of sound design to the Preschool genre. For me this project totally changed from simply getting a product out the door and in the app-store at a decent quality bar to me becoming an advocate for better design and the absolute best experiential quality in the children’s game and app market.

In retrospect, being in-house has had perhaps the biggest impact on the degree to which I was able to become embedded on the design and production teams and work multi-dimensionally, rather than at the (often) arms-length 'over the wall' approach to sound in mobile. Someone on the inside has to be an advocate of ‘sound thinking’ and a co-ordinator of the experience overall experience with sound in mind as much as visuals. I think overall, things are heading in a good direction with some developers focusing their efforts and expertise entirely on kids game development and, even more generally, on emphasizing quality user experiences through sound for audiences of any age. I believe (and hope) we are going to see more full-time, in-house audio production roles within small and mid sized developers, and also freelance audio designers and composers consulted and retained much earlier in production, to help iterate and deliver better experiences in these important products.

Advocacy

I believe this is a genre that deserves a lot more time, care and attention from developers, especially in terms of sound and presentation, but of course across all disiplines. We desperately need more advocacy for children’s digital listening and learning experiences. A child’s first contact with mobile technology should absolutely consist of the best experiences we can create. Sound is one such fundamental, yet underestimated aspect of the overall experience; it carries valuable information and entertainment - both directly and subtly, and enables a very different kind of connection between game experience and player. I believe that improving the balance between the direct and the subtle, the interactive connections between visual and sonic, and improving the way entertainment and education is presented and co-ordinated alongside interactive and audio-visual elements will vastly improve the valuable learning experiences on offer to those we care about the most.

To say I’m incredibly optimistic and excited about the future of this medium for this audience is a huge understatement, and over the next few years, this space is where I think some of the most exciting developments and biggest innovations can happen.

The Zorbit Sound Team:

Matthew Dominey (Audio Programmer)

Rob King (Dialogue Director)

Rob Bridgett (Audio Director)

Our Awesome Cast:

Zorbit – Patty Mattson

Zorbit’s Mom – Colleen O’Shaughnessy

Zippy – Haley Mancini

Marty – Peter Kelamis

Zorbit’s Dad (Sgt. Scrambler) – Jeff Meachem

Chirp – Christine Lakin