Deep Dive: Developing Yarn Spinner, a writer-focused tool for creating interactive dialogue

Author: by Jon Manning

Game Developer Deep Dives are an ongoing series with a goal of shedding light on specific design, art, or technical features within a video game in order to show how seemingly simple, fundamental design decisions aren’t really that simple at all.

Who: My name’s Jon Manning, and I’m a cofounder of the Australia-based independent studio Secret Lab, where I’m a developer on Night in the Woods. Recently, I just finished the port to iOS. So that’s cool! 

What: Night in the Woods is a narrative-focused game, and the story is told almost entirely through conversations between characters. The game was also built by a very small team, and Scott Benson, the lead dialogue writer, was also the lead artist, which meant that the system for representing dialogue needed to be something that didn’t need much knowledge of programming.

At the same time, the game contains lots of linear cutscenes, which needed to take into account actions taken by the player - whether they’d hung out with one character over another, whether they’d taken the time to watch TV with their dad every evening, and so on.

This created some very interesting constraints for us. We wanted to build a tool that made creating narrative games and narrative experiences easier. We wanted to build a tool that meant that writers, who were not necessarily incredibly technical, or interested in also being programmers, could write the scripts for games. We wanted to enable them to set ‘stage’ directions, character moods, and the tones and flow of the scene from their scripts.

This meant that we had an interesting problem to solve: it had to be expressive enough to be able to control almost every aspect of the game, while also being simple enough that it didn’t require a background in programming. The system also had to support easily writing very long scripts that could be re-read and re-drafted later, which meant that the syntax for this had to be extremely minimal.

The hardest problem in computer science, as we all know, is naming a project. Scott had worked with Twine, and suggested that the dialogue system for Night in the Woods be called Yarn. With the most arduous task out of the way, we got to work on finishing up.

The very first version of Yarn was a simple line-by-line interpreter. A text file would be read in, one line at a time; if it contained special characters, it was treated as a command for controlling the cutscenes, and if it didn’t, it was a line of dialogue for a character to speak.

This syntax worked extremely well. By treating a single line of unadorned text as a line of dialogue, Scott was able to sit down and write a screenplay, which effectively became executable code. Cutscenes could also be blocked out with placeholder lines, which were replaced later during a polish stage. Finally, by designing the entire thing from the start as a text file format, tracking the file in version control became extremely straightforward.

However, this simple approach created limitations. Longer-range syntactic constructs like ‘if…endif’ require state tracking, and it became very challenging to handle nested syntax. Nevertheless, there was something extremely appealing about this system, from both the perspective of writers and programmers.

It was around this time that I joined the Night in the Woods team, and I got to work creating an improved implementation of the Yarn language. Yarn was the language, so I called the system that worked with it Yarn Spinner.

The original issue was that the writer didn’t want to become a programmer. To solve this problem, I built a programming language for him. Haha.

Here’s the thing about programming languages. The reason they’re complicated and hard to make is that they need to let you solve any task. Our programming language, though, had to solve precisely one problem: “what line, command, or collection of choices should the player be seeing right now?”

It’s possible to create extremely low-friction programming languages, as long as those languages are able to place tight constraints on how they’re intended to be used. As a part of this, we created a formal grammar for the language.

When you build a programming language, you’re building two things:

Solving the first problem - building a parser - is common enough that there’s a wide variety of tools out there that solve this problem for you. The solution that we went with is a tool called ANTLR (“ANother Tool for Language Recognition”), which takes a formal definition of the language’s grammar, and generates C# code that parses text according to that grammar. This C# code forms the first stage of the tool.

The great thing about using a formally-defined grammar is that nested data structures are easily representable as recursive grammars. For example, to solve the earlier problem of nested ‘if’ statements, we can say that an ‘if’ statement contains multiple statements, and one of the things that a statement can be is an ‘if’ statement.

By using a parser generator, we also get very nice features like syntax error detection, where the system is able to detect the specific point in the input at which it realized that there was a problem in the code. This was a lot harder to do in a simpler interpreter.

The second task - compiling the syntax tree - is what produces the final asset used in the game. Given a tree data structure that represents the input, we ‘walk’ the tree, generating instructions for low-level operations like ‘fetch data’, ‘add number’, and - uniquely to Yarn Spinner - ‘run line of dialogue’.

This bytecode, along with the original text of the lines, is what’s stored in the game, and is what’s executed during gameplay.

When building a system like this, it’s very easy for a programmer to fall into the trap of assuming that everyone else is as excited about twisty, complicated structures as they are. An ongoing challenge for Yarn Spinner has been to keep the design free of ‘programmer-isms’. This is a language for writers, not a language for coders; it’s a tool for building interactive screenplays that are embedded inside games, and not for building the games themselves.

One of the most significant features of Yarn Spinner’s grammar is that any text that is not interpreted as other syntax is considered to be a line of dialogue. This can be surprisingly difficult to implement, as most parser generation tools are designed around the idea of every word and symbol having a precise meaning in a precise context, whereas our lines of dialogue add the concept of “but sometimes, anything goes”. However, by defining this language feature, the additional work that’s created for the compiler developer pays off many times over in terms of language accessibility.

Why:  We didn’t have to build this tool as an entire programming language. However, by plunging straight into the deep end, a number of long-term benefits are made possible.

Because it converts its input into a formal syntax tree, Yarn Spinner is able to generate a control flow graph and explore all possible paths through a story, and find lines and content that cannot be reached under any circumstances.

We also used it during the Japanese localization of the game, where we generated instructions for playing the game that guaranteed that the player would see every single possible line of dialogue (over multiple runs of the game), with minimal overlap. (To those who are about to post a comment: no, we haven’t solved the Halting Problem; we just don’t allow loops in the checker.)

We’re also able to perform automated migration from one version of the language to another, by parsing input from one version of the language into a syntax tree, and then re-writing that tree into the second version’s format. This keeps the tool nimble, and allows us to add features that break backward compatibility with the knowledge that we aren’t adding costly re-writing workloads to the rest of the team.

Additionally, because we extract and compile the language as pre-compiled byte code, we separate the player-facing language from the script’s control structures. This makes localization and translation much easier, since a line of dialogue can be swapped out based on the user’s locale settings.

Result: We released Yarn Spinner as an open-source project in 2015, two years before the release of Night in the Woods. Since then, it’s been used in two games that won the IGF Seumas McNally Grand Prize (Night in the Woods, and A Short Hike), and is now used in games tiny and enormous. The project is released under the open-source MIT License, which has made it straightforward for using it in games ranging from solo devs to larger indie teams.

The project has actually become so successful that it’s become the primary focus of our studio, and lots of our current work is in building Yarn Spinner, either directly on the project or as part of contracted narrative engineering work for other studios.

We think that complex storytelling should not be the exclusive domain of those with technical skills, or those with the resources to hire or build a large team. They should also be the domain of the underrepresented, the diverse, and those who have a passion, or interest, or inclination in writing an engaging interactive work, but don't necessarily have the technical skills or resources.

We consider Yarn Spinner to be an overwhelming success, and the growing size of our community of novice and experienced game developers reflects this. If you’d like to learn more, please join the project’s Discord server here, follow the project on Twitter at @YarnSpinnerTool, and visit the project on GitHub here.