Over the course of my senior year in college, I've been hard at work on a visual novel dubbed Streams of Nurture. As both a passion project and academic piece, Streams of Nurture represents an interesting case study for transforming one of my most ancient interests, food production, into an interactive affair that combines the dramatic qualities of an entertainment title with a real-world topic characteristic of serious games. In other words, I wanted to leverage the concept of "learning through play" to create a game that felt both purposeful and engrossing.

With my project officially submitted to the Worcester Polytechnic Institute database, I can now share with you the report detailing the development cycle for the visual novel. Throughout the coming weeks, I shall be publishing a subsection of the paper penned for Streams of Nurture. Each part will focus on a particular design and production aspect of the game, from the way the storytelling was conceived to the iterative art process that governed the title's presentation.

This week's post in a series of design articles explains the audiovisual development of Streams of Nurture. If you're interested in reading about the pre-production and narrative phases for my visual novel, feel free to peruse the first and second articles I published on my title’s development cycle.

NOTE: I didn't single-handedly develop the game. Although I handled the narrative design, coding and writing for Streams of Nurture, my partners Liam Miller and Dave Allen were wholly responsible for the art and audio respectively. As such, the pronoun "we" will be used to acknowledge their invaluable contribution to this project.

Art production

Artistic references

During the Fall 2016 semester, the team got together to discuss audiovisual possibilities that would complement the story and setting of the game. A lot of time was spent brainstorming ideas for the graphical layout of the title (see Figure 27), which had to be both practical and stylish, and trawling the Web for reference images that could potentially be used to nail down Streams of Nurture’s aesthetics and user experience.

Figure 27. Notes detailing the project’s visual aesthetics.

Original picture by Michel Sabbagh

Once all the pictures that closely matched our presentational vision for the visual novel had been gathered, the team winnowed the pool of references and ended up with artistic works that bore an experimental and/or ethereal style. Most of those pieces of art made use of oil painting, vivid lighting, and intermingling of color and shape to create a radiant image that effectively portrayed natural surroundings and the characters in it. This eventually led to the development of the tribal motif that adorned each of the visual novel’s backdrops and personalities.

Asset list

In order to uphold the audiovisual vision that the team crafted at the beginning of development and ensure that it remained consistent, a comprehensive asset spreadsheet (see Figure 28) was created at the end of the fall semester.

Figure 28. Portion of asset spreadsheet itemizing the game’s art.

Original image by Michel Sabbagh

The asset list provided the in-game reference name, actual filename and a brief description for every piece of art created to implement the functional framework. URLs to images containing the intended features of each scene were also included, ensuring that Streams of Nurture would not suffer from incongruous-looking visuals that conflicted with the narrative.

Style transfer

As the label “visual novel” suggests, the artistic aspect of our game was critical, requiring a unique and compelling art style that would hold the attention of players and support the goal of educating them about sustainable aquaculture.

Some of the game’s characters are of Coast Salish descent, and many story situations mention the indigenous fishing practices that maintained their communities throughout generations. For these reasons, respectful incorporation of Coast Salish aesthetics into the “look and feel” of our project seemed both desirable and appropriate. This goal was achieved using a relatively new application of artificial intelligence (AI) known as style transfer.

The possibility of harnessing AI algorithms to apply the aesthetic style of one piece of art onto another was first brought to our attention by Professor Moriarty. We knew of no visual novel that had ever used this technology as the basis for its art direction, and decided to investigate its potential for our project.

Our first task was to identify a style transfer application that met the requirements of Streams of Nurture. Specifically, we needed an app with the following characteristics:

Several style transfer applications were found, but most fell short in one or more of these areas. Many offered their services for free, but severely limited the size of image output, or did not support custom styling images. Others charged a per-image fee, utilized weak algorithms that produced inconsistent or unattractive results, or required a very long time (sometimes over a day!) to stylize each image due to overcrowded servers.

Eventually we discovered Alter, an Android app available on the Google Play store, which met nearly all of our requirements. It supported custom styling images, processed entire images (unfortunately not at full HD resolution), responded quickly, imposed no limit on the number of images processed, and (most importantly) proved capable of producing visually stunning results. Alter was therefore chosen as the centerpiece of our project’s art pipeline.

Using authentic Coast Salish art references, Liam created a single image that was used as the master styling reference for every background and character image used throughout the entire game (see Figure 29). Figure 30 shows the result of this style reference applied to an original photo.

Figure 29. Master source image for style transfer, influenced by Coast Salish art.

Original image by Liam Miller.

Figure 30. Example of a source photograph (left) after filtering and style transfer (right).

Original images by Liam Miller.

Figures 31-34 detail the steps used to transform an original photograph into a style-transferred background image ready for installation in-game.

Figure 31. Step 1: Original photograph.

Figure 32. Photograph pre-treated with Photoshop.

Figure 33. Filtered photograph after stylization by Alter.

Figure 34. Stylized image with depth blur applied by Photoshop.

Original images by Liam Miller

Art process 

Once the team had the asset list of images and a method for processing them, it was time to get started on developing character models and gathering images to be used as backgrounds. Primarily, the team needed to get out into the world and collect pictures, as over 200 environmental background images were required. These scenes would feature indoor offices, fishing markets, factories, Native American reservations, and fish farming sites. Liam took advantage of his trips home to Hawaii over the breaks to gather images that could not be taken back on the East Coast. He also traveled from Worcester to New York taking pictures, and undertook a photo-gathering pilgrimage to New England’s archetypal fishing village: Gloucester, MA. After cropping selected photographs and applying style transfer with Alter, a blur was added with Adobe Photoshop to enhance the separation of background images and superimposed characters.

Collecting background images of the processes used in salmon farming and preparation for consumers proved challenging. Many such scenes would be impossible to acquire while attending courses full-time at Worcester Polytechnic Institute. To produce artwork for scenes that could not be photographed ourselves, Liam turned to the collage technique, where he took simple aspects of reference images, cropped, edited and fit them together like a puzzle to create an approximation of the required scene. Applying style transfer and post-processing to these images effectively concealed their Frankenstein-like construction, allowing them to fit seamlessly into the rest of the game (see Figure 35).

Figure 35. Example of collage-built scene, showing cropped ship over water with drawn platforms.

Composite image by Liam Miller.

Character production

Michel’s character templates supplied descriptions of how he envisioned their physical appearance and social tendencies. This made it relatively easy for Liam to produce character sketches for discussion and mutual approval.

One aspect Liam had to keep in mind while drawing these characters out was that some of them had true-to-life ethnic backgrounds and stories. For those who represented Coast Salish descendants, Liam strived to be both accurate and sensitive to avoid the possibility of offense. For those with impacting backstories, Liam had to capture the look and emotions of how someone in that position might appear (see Figure 36).

Figure 36. Original sketch of Daphne Crosby (left), and prepared for stylizing (right).

Original images by Liam Miller

A more tedious factor involved in creating these characters was the process of displaying them in at least three different positions: left, forward and right-facing. New to producing drawings of characters at multiple angles, Liam struggled to keep the characters looking the same at different positions. Defining a standard character size of 2160 x 800 pixels assisted him with controlling the relative size and height of the models.

Liam’s early experiments with style transfer on both background and characters produced results that made it hard to separate the characters from the backgrounds. Eventually Liam developed a method of “dialing down” the style transfer effect on the characters to preserve the detail on their faces and clothing, keeping them distinct from the background. Specifically, using percentages as strength levels of style transferring, the backgrounds usually ran from 16-23% strength and characters ranged from 11-17%, each depending on detail and depth of what was being portrayed (see Figure 37).

Figure 37. Joshua before (left) and after (right) application of style transfer.

Original images by Liam Miller

Sound design

When it came to creating the soundscape and music for Streams of Nurture, the team hoped to suggest the serene, tranquil atmosphere of fishing towns. We also wanted the audio to complement the visuals and narrative on display without overpowering them. To achieve these effects, we sourced and listened to various tracks spanning several genres such as New Age and minimalism.

It wasn’t until the latter half of the spring semester that we managed to find an audio partner in Dave Allen, who was willing to lend his skills to the project. Upon joining the team, Dave was provided with the audiovisual references we sought in 2016 as well as an asset list of sound effects he had to produce for the beginning sections of the visual novel.

Figure 38. Screenshots of audio workstations.

Original images by Dave Allen

Given the scope of the project and the amount of time left in the academic year, Dave had to adopt a modular approach with regards to composing the tracks and sound effects for Streams of Nurture. Using digital mixers and synthesizers (see Figure 38), Dave wrote multiple pieces of original music, using a variety of strategies to capture the relaxed atmosphere of coastal life.

When writing the music and designing the soundscape, Dave found it important that the audio stayed in the background, as the visual novel format is focused more on narrative development than audiovisual spectacle. This led him to create a minimal soundscape based on layers of environmental ambiance, with each cue having its fade-in/fade-out values adjusted to properly reflect the change of locales.

And that's all she wrote (for Part III, at least)! As I mentioned before, more parts will be posted on a weekly basis.

Let me know what you think of my article in the comments section, and feel free to ask me questions! I’ll do my best to get back to you as promptly as possible.

Personal blog: https://michelsabbagh.wordpress.com/

Twitter: https://twitter.com/Watfen64

LinkedIn: https://ca.linkedin.com/in/sabbaghmichel

Facebook: https://www.facebook.com/Michformer

Email: [email protected]