Bringing CastleDB to Unity

About a month ago I got the idea to try and make Nicholas Canvases’ game database tool CastleDB meaningful integrate with Unity. If you just want to try out the plugin, see the repo here. If you want to know more, read on!

Introduction and Background

Separating game data from logic in Unity has never been something that's been particularly easy. The introduction of ScriptableObjects was welcome addition that addressed some of this issue, but despite their utility they have certain limitations that still make them a non-ideal data container. 

Namely:

None of the above makes working with ScriptableObjects a “bad” experience. I think if you’re working on a small project they are totally serviceable, but as projects grow in scope ScriptableObjects can easily become unwieldy. 

(For a pretty good overview of the benefits of ScriptableObjects, check out the link above.)

I was thinking about this recently and thought that there had to be a better way to manage data in Unity. Something that really allowed you to separate data management from Unity, but also provide a great interface to using it.

I think that a lot of people that get this itch settle on finding some way to manage external data through .csv or .xml files and then manage integrations with those data types. This satisfies the need to have your data be centralized, but even this felt unnecessarily bulky, as you still need to manually create objects and classes whose signatures somewhat match the data your loading. I instead wanted something that felt like it flowed, something that made data feel like an intrinsic part of my code, something that went beyond data loading and unloading.

Discovering CastleDB

It wasn't until a few weeks after having this thought that I found this article on Dead Cells on Gamasutra and learned about a curious little tool called CastleDB which made me think that something like what I wanted to do was possible.

CastleDB is, in short, a JSON editor that makes working with JSON feel more like working with a traditional spreadsheet. It’s also made with the explicit purpose of being a “database editor for games”. It was created by Nicholas Canasse, a relatively prolific game developer who is probably better known as the creator of the Haxe programming language (which is also what CastleDB is written in).

CastleDB is also meant for Haxe, and as such uses a specific language integration that, once I saw it, looked exactly like what I was imagining when I was thinking about “data as code”. The code sample below is how you interface with CastleDB in Haxe:

package com.castle.demo;
import haxe.Resource;
import com.castle.demo.db.MyData;
class Main 
{
    static function main() 
    {
        MyData.load(haxe.Resource.getString("test.cdb"));
        trace(MyData.myDB.get(Dragon).HitPoints);
    }
}

In the code sample above, you can see how, after loading the database with a single call, you can directly access the database's data in a strongly typed manner.

I won’t spend too much time talking about the appeal of Haxe for game development (there are already a few articles out there), nor is this article an attempt to convert you to Haxe (we're talking about Unity!), but it is worth touching on what makes the Haxe/CastleDB interop special.

CastleDB leverages Haxe’s ability to use macros. A macro can mean a lot of things but in this case it allows you to do some really interesting things with edit-time code completion. In the video below you can see how, by creating and then importing a database object, you have direct access to the data at edit time and can, in a typesafe manner, access items from your database as soon as they are added.

Pretty nice right! Make a database, add some columns and lines, go back to your IDE, and they they are, nicely auto-completed. This was exactly what I was looking for! But given that I'm programming C# in Unity, how could I unlock something similar? CastleDB is targeted at Haxe users — was there a way to make it work with other frameworks or engines?

CastleDB + Unity

With this in mind I set out to figure out how I could go about making CastleDB work with Unity. Starting out, I needed to figure out what “working” with CastleDB would mean. Just being a JSON editor is interesting, but I wanted to unlock a similar flow to what I saw in Haxe. But how would you do this in C# without macros? I needed to figure out how to interpret the CastleDB file itself and figure out a way to make it "knowable" at edit time in Unity.

The first step seemed kind of obvious, make the CastleDB file readable in Unity and parse the JSON file.

What made the most sense to me was to think about the database as a collection of sheets, where every sheet defined a specific type of object (a C# Object/Type). Columns then would be the fields of that defined Type. It then follows that if a sheet is an Object and the columns are fields, then every line in a given sheet is a specific possible instance of that Object.

So a sheet called "Creatures" with a column called "Health" and a row named "Dragon" would mean that there is a Dragon type of Creature, and all Creatures (like the Dragon), have a Health property.

This started to make sense but I also needed to figure out how to get the CastleDB file into Unity, as Unity doesn’t natively recognize the .cdb file. Luckily, Unity recently added an experimental ScriptedImporter API that would let me treat the file as an actual asset. Because it was just text, the actual “import” portion here was easy:

public override void OnImportAsset(AssetImportContext ctx)
{
    TextAsset castle = new TextAsset(File.ReadAllText(ctx.assetPath));
    ctx.AddObjectToAsset("main obj", castle);
    ctx.SetMainObject(castle);
}

I read in the JSON file as Text and create a TextAsset out of it. I used a TextAsset because all I really want out of the .cdb file is the ability to read in its text.

The importer also allows us to run code any time that object is imported, which meant that, in the importing process, I could do something to make the CastleDB (.cdb) file affect the overall state of the Unity Assembly. But what?

Stating plainly what I wanted to happen made what I needed to do clearer. At the time of import, I wanted what was defined in my database to be available to me in my editor. Saying that differently, I wanted CastleDB defined Types to be automatically generated when I imported my .cdb file into Unity. Ah, so I just wanted code generation!

Generating Types from a Database Object

Now, full disclosure, I had zero experience with C# code generation going into this issue, which I only say because if this part of C# is weird or confusing to you, you can totally learn it! You can also skip the mistake I made of trying to learn C#'s AssemblyBuilder and instead skip to the part where I find out that Unity has it's own in-built AssemblyGenerator class that is far easier to use than the native C# one. You can then also probably skip needing to use AssemblyGenerator at all and just write out some text to a file!

Instead of needing to emit C#'s intermediate language using obscure OpCode calls (which, even if you do right, isn't guaranteed to work due to Unity's managed version of C#), you can easily generate classes in Unity by writing out strings of text to a file location. If that file has a .cs extension and is inside your Assets/ folder, it will get automatically imported and built into your project’s solution!

In the AssemblyBuilder documentation, Unity has pretty good (and working) bit of example code that does this and is copy-paste-able into your project. Check it out here.

For this project, I basically use this example but without the AssemblyGeneration portion. I write all the text directly to a path in Assets instead of using a generated .dll (though that's totally possible!). I make heavy use of Unity's new-ish ability to support interpolated strings and build the classes from their component parts before writing out the file. Here’s the whole class in the repo, but here’s the gist of file writing:

//build out the class earlier in the code
//compose the final class
string fullClass = $"{fullClassText};
//write the class to a file in the Assets/ path
File.WriteAllText(cdbscriptPath, fullClass);
//refresh the AssetDatabase to alert Unity that you have new Assets
AssetDatabase.Refresh();

Easy right? This generation code is then called inside the imported script:

public override void OnImportAsset(AssetImportContext ctx)
{
    TextAsset castle = new TextAsset(File.ReadAllText(ctx.assetPath));

    ctx.AddObjectToAsset("main obj", castle);
    ctx.SetMainObject(castle);

    CastleDBParser newCastle = new CastleDBParser(castle);
    CastleDBGenerator.GenerateTypes(newCastle.Root);
}

So, parse the CastleDB file on import, generate the code that creates the type, and done right?  Well, yes and no. Unity is, I'm going to say, notoriously lacking when it comes to serialization/deserialization of JSON files. Their native JSONUtility has righted some wrongs of the past, but it still suffers from other issues, like the inability natively traverse JSON deeper than one level. This isn't a massive issue if you properly define what types you're trying to serialize or deserialize beforehand, but our issue is that, when deserialization happens, we don't know what we're deserializing. You could easily add a whole new sheet to the Database, which would mean a whole new type, that Unity would have no idea about unless oyu manually went in and created the matching Type (which is then antithetical to this whole idea!). Unless you change your deserialization code every time, there isn’t a way for Unity to natively know what JSON you’re going to pass it. I needed something that didn’t care, and could handle whatever I, or any other user, could throw at it.

Parsing Unknown JSON

Without beating around the bush, I quickly landed on using SimpleJSON for the parsing of the .cdb file. It's pretty small and has a nice API, so integrating it into the project was painless. Because it gives you the ability to index into a JSON file by a string value like this:

node[“item”]

I could essentially look at the cdb file’s field descriptions and then use what I found to parse the whole JSON tree. Hats off to Nicholas Canasse here as well — the format of the JSON file contains field and sheet metadata, which proved invaluable when building this integration.

For every column in a CastleDB sheet, the .cdb file holds a “typeStr” value that is... a number string that maps to the type a column is supposed to be. So if a field has a typeStr of "1", this always maps to a string type. A "2" is a bool, "3" is an int, etc. This is a static map as well so I can just hard code column types to actual types in C# based on this value by using a big switch statement. You can see how the native Haxe library handles this here.

SimpleJSON let me dig as deep in the file I wanted, and the CastleDB “typeStr” made the whole effort possible. Before I went on to writing the code that actually leveraged the parsing, I then wrote a small shim class that parses the .cdb file with SimpleJSON, and builds out a typesafe representation of the data that can then be used to generate the actual dynamic parts of the database. So instead of needing to call something like RootNode[“columns"], I can, after parsing, just call RootNode.Columns and get the same return data back.

After all this is in place, you can generate a sheet’s class string by doing something like this:

foreach sheet in Root.Sheets
    // make a new type with sheet.Name
    string newClass = $“public class {sheet.Name} {...”
    ...
    string fieldText = “”;
    foreach column in sheet.Columns
        // add in type names
        fieldText += $"public {column.type} {column.Name};\n";
    newClass += (fieldText + "}");

What’s nice about the code generation above is that you can just use a string as a type. That string “turns into” the type once the code is generated, which is part of the beauty of code generation.

You then put all of those disparate strings together in a beast that looks like this:

//use string interpolation and string literals to easily construct the class
//the component parts are constructed before this bit of code
string fullClassText = $@"
using UnityEngine;
using System;
using System.Collections.Generic;
using SimpleJSON;
using CastleDBImporter;
namespace {CastleDBParser.Config.GeneratedTypesNamespace}
{{ 
    public class {sheet.Name}
    {{
        {fieldText}
        {possibleValuesText} 
        {ctor} 
        {{
            {constructorText}
        }}  
        {getMethodText}
    }}
}}";

Spit that out into Assets, and now look what you can do!

Leveraging Code Generation for some Nice Things

On top of being able to access a given row, as part of the generation I’m also making it so that when you get the row you’re also getting it initialized with all of its proper database values. I do this by dynamically creating an enum in the generated class that acts as a row lookup into the rows of the sheet the Type is built from. That looks like this:

public Creatures {
//other field text omitted
  public enum RowValues { 
    Squid, 
    Bear, 
    Dragon
  } 
}

I then turn that enum into a JSON lookup (that is dynamically generated) that spits out code like this:

Creatures Get(CompiledTypes.Creatures.RowValues line) { return new Creatures(parsedDB.Root, line); }

And the object’s constructor looks like this (also dynamically generated):

public Creatures (CastleDBParser.RootNode root, RowValues line) 
{
    SimpleJSON.JSONNode node = root.GetSheetWithName("Creatures").Rows[(int)line];
        id = node["id"];
        Name = node["Name"];
        attacksPlayer = node["attacksPlayer"].AsBool;
        BaseDamage = node["BaseDamage"].AsInt;
        DamageModifier = node["DamageModifier"].AsFloat;
        foreach(var item in node["Drops"]) { DropsList.Add(new Drops(root, item));}
        DeathSound = (DeathSoundEnum)node["DeathSound"].AsInt;
        SpawnAreas = (SpawnAreasFlag)node["SpawnAreas"].AsInt;
}  

This gives you a typesafe method to create objects using your defined row values!

Conclusion

It's not exactly macros, but it does feel like it has some of the same niceness of them. I think that over time your database scheme will likely (and rightfully) stabilize so the type completion becomes less important, but what will still be relevant is the ease of data editing/altering. You'll be easily able to change references and data inside of the database without needing to use Unity. 

Hopefully with some of this work I’m able to “unlock” our data from inside Unity! I also hope that with this plugin, it can allow for non-coding collaborators on projects to feel more able to work with actual game data without the overhead of knowing how to code or knowing how to use Unity. Just open up CastleDB, change some values, and you’re done!

As I said above, if you’re interested in checking out this integration, I highly recommend you look at the project page and give it a download! I’d love to have some feedback on the project, and would love to know if people start using it in their games as well!

Also, feel free to sign up for my Cantata newsletter here!

Final notes:

An interesting extension of some of this that I haven’t really pursued yet is making CastleDB natively use Unity’s native types like Vector3’s, Quaternions, Rects, etc. CastleDB supports custom types, so you could theoretically make a custom type that maps to a Unity type.

It’s also worth saying that a subset of what I’m doing here would be possible with XLS or CSV, but those formats aren’t necessarily human readable and don’t easily manage inner row referencing. A .csv file is also aggressively bad from a source control perspective. JSON is literally “object notation”, and as such is a better form for managing data objects. CastleDB also has some niceties in how it works, like auto sheet generating for nested types, that reduces overhead and gives you new types “for free”.