Data-Driving Unreal Engine 4 for Workflow, Hotfixes, and Live-Ops

This article was originally posted on the Joy Machine blog; maybe check it out too! <3

When I was working on Starhawk, we had an critical feature in the game that allowed for “hotfixes” to be deployed for data updates (game/balancing adjustments, generally) without requiring a major patch that would have to go through Sony’s week-long (at a minimum) patch and QA process. This was my first experience with game updates that didn’t require a full-on binary update.

Shortly thereafter I worked on mobile games for a couple years and, as I’m sure many people know, any opportunity to bypass the iOS App Store’s reviewing process is a wonderful — well, necessary — thing to do. So, for our team, we handled all game data through an extensive amount of JSON data for level definitions, player profiles, settings, game and balancing data, triggering special events and deals, so on and so forth. I believe we also stored player data on a secure server/database as JSON data, but that wasn’t and still isn’t my area. Thankfully.

Goals for Game Data and Live-Ops

I like storing as much game data as I can in text-based data files (specifically JSON files). And when I refer to game data, I mean things like mech data (which is, largely, just a list of references to other data files and game asset paths), mech part templates that are used to procedurally generate in-game items, game world parameters, player profiles and settings, and so on. These same formats are also used for storing data that is created in-game; while an item may be generated from one or more data templates, it eventually has to get saved and stored somewhere. Those files aren’t really ever touched beyond that outside of saving/loading.

That said, those files also retain references to their original templates, meaning that if some template was found to be asininely overpowered, the original template can be tuned, and the effects of doing that will ripple out to whatever was created in-game.

Since these are all basic text files I can access, I can easily tweak them and just refresh them in-game. The ability to easily modify game data in-editor and out with a simple rile refresh is just lovely, but it’s also been the core of game development and design for ages (called data-driven gameplay/content). Unreal Engine 4 allows for this entire workflow, but at some point game content has to be packed and, to my knowledge, beyond that they are contained within binary packages. And one thing I like about my game data is that it’s always easy to get at, tweak, load, etc. during the course of development regardless of whether it’s a loose build or a packaged build. Something I’ve also gotten accustomed to after a few years in mobile games is not having to rely on a full binary update/patch to a game in order to make changes, especially if it all amounts to little more than a tuning pass or a non-binary-patch hotfix. I’d much rather have players just load up the game, have a ten-twenty second update period, and just have all the updated tweaks. This whole process is also the core of any post-launch “live ops” setup that I’ve worked with: update some files on an authoritative CDN, players download them when going online, and now you have a 24 hour special event with unique items.

And as I’m starting this article, I really hope that the first comment somewhere isn’t “wait, you didn’t just use <x>?”, revealing a critical and completely relevant feature set that would have been incredibly useful. But, I don’t think that’s the case in what is to come.

A Surprising Realization

When I first started working with Unreal, I was doing all my prototyping in blueprints. As I’ve noted before: that didn’t work out well at all. I was under the assumption that blueprints were intended to be a nice frontend over, essentially, scripts. And that these blueprints could easily be tweaked/updated dynamically after building a project.

After writing that, I realized I should note: this was very early on in my time with UE4 (and I had never used a prior version of Unreal Engine).

Anyway, yeah, so: no. That’s not the way blueprints work at all; in fact, it’s quite the opposite: upon cooking a project’s content, blueprints actually get built into C++ code — potentially the lowest rung on the ladder of easily-updated game data. So, it really went in a different direction than I was expecting.

There was a fairly lengthy duration of time after Blueprintpocalypse 2017 that I had to just spend establishing the very basic codebase necessary to begin work in earnest on the actual game. And then establishing the basics of the game’s critical actors/components/etc. And it was after that initial pass of development that I started searching for a raw (uncooked. get it. because… well… content gets cooked? hush.) way to maintain game data, configs, local profiles, and other such things that didn’t require cooking to be used by the game runtime.

Despite my desire to just jump to JSON for game data, I wanted to see what the conventional/recommended practices were for UE4 for data-driven design and development. My inclination was to use Data Tables given that they’re covered in documentation that is literally titled “Data Driven Gameplay Elements”. And it sounded, well, it involved Excel, so it didn’t sound great, but it was something. And then I looked into it and discovered that you could manipulate the data externally, but it would eventually have to be imported into UE4 to be used. You can export it later, if need be, but eventually it has to be reimported. And eventually cooked.

So, yeah. Wheeeeee.

Yes. At least once a day, when I internally think about JSON, I instantly hear JASON! JAAAASON! JASON! in my head. Anyway, yeah, I liked working with JSON files either due to their simplicity or just habit, so I then went about figuring out how to make that work. Luckily, I found UE4 had Json and JsonUtilities modules!

So, I checked out the documentation after a cursory glance at the source files. And the documentation for Json and JsonUtilities was... Well. It was what you see. And in case anyone reads this in a year or something and it changes, for posterity let me inform you: there isn't much to look at.

That said, the source wasn’t complicated to wrap my head around. Putting it to use in practice, however, was a different story. My first test of the modules’ functionality was to serialize out an instance of my mech actor. And I was able to get it working pretty quickly and easily! And if you’re expecting a “but…” to follow that statement… … … Yes there is a but:

The serialized output of an actor is a horrifying disaster of data that you would never want nor even need to see in a maintainable data file. It was the entire actor. Including data you didn’t even know actors had. And it did an admirable job of recursively serializing everything in said actor, but it still was a lot of data that was clearly not meant for the purpose of, say, a data config for a mech part.

After a bit of digging — and a fairly extensive amount of trial-and-error — I eventually got to the point where, so long as I stuck to structures consisting solely of fairly simple data types, I could use the UE4 serialization utilities to easily serialize and deserialize the structure’s data.

Using that data structure in the nitty-gritty of the rest of the game code, however, isn’t quite as simple. C++ with UE4 is not entirely standard C++. A structure, to be recognized by the Unreal Header Tool (and Unreal Build Tool), must be signified by the USTRUCT macro. And, unlike standard C++, a USTRUCT struct is not "basically" the same as a class. It's intended (and strictly enforced) to be a structure of basic data that doesn't need to be carefully managed by the engine's garbage collector. I get a bit fuzzy on the specifics since I haven't worked on this stuff in a while and also working on all areas of a game tends to obscure the specifics of work done a week or two ago; a month ago is just way too far back to even consider.

Long story short: to really be able to just use the data that gets serialized/deserialized via a USTRUCT, I decided to, essentially, create a near-carbon copy UCLASS (with some transformations/setup occurring during the transfer in some cases). And once that UCLASS is all filled with delightful data, it can be used all over the place as intended.

I tossed up a gist (seen below as well) of an older version of my highest-level mech part data structure that exists in my codebase (as you go down the tree, it gets rapidly more dense). That gist also uses my absolute favorite, if totally ugly, macro sets I’ve ever created: the accessor macro generators (these do not result in the methods defined being UFUNCTIONs, however -- which basically means they can't be exposed to blueprints/external modules; in this case, that's exactly what I want). This is nothing more than a shortcut for generating basic get/set accessors on the UCLASS version of a data structure (so, not entirely unlike C# properties), but it was a handy way to trim down the definition and remove as much room for error in implementation as possible. That high-level part data structure serves, essentially, as a great-great grandparent to more specific part logic, but each child is only responsible for managing its own data; any parent to that child will take care of its data and that parent's parent will take care of its data and so on.

// FMechPartDataBase Data Structure.
//	This structure is solely for serialization/deserialization purposes (it gets transferred to the UObject instance after that process is done).
USTRUCT( )
struct FMechPartDataBase_SerializationStructure
{
	GENERATED_BODY( )

public:
	FMechPartDataBase_SerializationStructure( )
	: ConfigName( NAME_None )
	, PartType( 0 )
	, Mass( 1.0f )
	, HealthMax( 1.0f )
	, AssetPath_MaterialPrimary( NAME_None )
	, AssetPath_MaterialSecondary( NAME_None )
	, AssetPath_MaterialAccent( NAME_None )
	, AssetPath_MaterialChrome( NAME_None )
	, AssetPath_MaterialMisc( NAME_None )
	, AssetPath_SoundCue_Impact( NAME_None )
	, AssetPath_SoundCue_Ricochet( NAME_None )
	{	}

public:
	/*
	 *	Config Name.
	*/
	UPROPERTY( )
	FName ConfigName;

	/*
	 *	Part Details.
	*/
	UPROPERTY( )
	uint8 PartType;

	UPROPERTY( )
	float Mass;

	UPROPERTY( )
	float HealthMax;

	/*
	 *	Asset Reference Paths.
	*/
	UPROPERTY( )
	FName AssetPath_MaterialPrimary;
	UPROPERTY( )
	FName AssetPath_MaterialSecondary;
	UPROPERTY( )
	FName AssetPath_MaterialAccent;
	UPROPERTY( )
	FName AssetPath_MaterialChrome;
	UPROPERTY( )
	FName AssetPath_MaterialMisc;

	UPROPERTY( )
	FName AssetPath_SoundCue_Impact;
	UPROPERTY( )
	FName AssetPath_SoundCue_Ricochet;
};

// UMechPartDataBase Class Definition.
//	This is the class used for actual game code (as opposed to the data structure below, which is strictly for serialization.
UCLASS( BlueprintType )
class UMechPartDataBase : public UObject
{
	GENERATED_BODY( )

public:
	UMechPartDataBase( const class FObjectInitializer& ObjectInitializer );

private:
	UPROPERTY( )
	FMechPartDataBase_SerializationStructure BaseDataStructure;

protected:
//	TSharedPtr< FMechPartDataBase_SerializationStructure > DataStructure;
	FMechPartDataBase_SerializationStructure* DataStructure;

	UPROPERTY( )
	TEnumAsByte< EMechPartType::Type > PartType;

public:
	virtual void SetSerializationData( const FMechPartDataBase_SerializationStructure* SerializationData );

	// Read-only access to the data structure (weapon data structure, specifically, unlike the prior two).
	const FMechPartDataBase_SerializationStructure& GetBaseSerializationDataRead( ) const;

	// Writable access to the data structure (weapon data structure).
	void GetBaseSerializationDataWrite( FMechPartDataBase_SerializationStructure& SerializationDataOut );
	FMechPartDataBase_SerializationStructure& GetBaseSerializationDataWrite( );

public:
	// NOTE (trent, 1/25/18): These accessor-generation macros do not result in the methods they define being treated as UFUNCTIONs (blueprint-exposable).
#define DEFINE_METHOD_SET_ACCESSOR( MemberType, Member )			\
	FORCEINLINE_DEBUGGABLE void Set##Member( MemberType Member )	\
	{																\
		DataStructure->##Member = Member;							\
	}
#define DEFINE_METHOD_GET_ACCESSOR( MemberType, Member )			\
	FORCEINLINE_DEBUGGABLE MemberType Get##Member( ) const			\
	{																\
		return DataStructure->##Member;								\
	}
#define DEFINE_METHOD_ACCESSORS( MemberType, Member )				\
	DEFINE_METHOD_SET_ACCESSOR( MemberType, Member )				\
	DEFINE_METHOD_GET_ACCESSOR( MemberType, Member )

	// Mech part type.
	inline void SetPartType( TEnumAsByte< EMechPartType::Type > PartTypeIn )
	{
		PartType = PartTypeIn;
		DataStructure->PartType = PartTypeIn;
	}
	inline TEnumAsByte< EMechPartType::Type > GetPartType( ) const
	{
		return PartType;
	}

	// Part JSON config name.
	DEFINE_METHOD_ACCESSORS( const FName&, ConfigName )

	DEFINE_METHOD_ACCESSORS( float, Mass )
	DEFINE_METHOD_ACCESSORS( float, HealthMax )

	DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_MaterialPrimary )
	DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_MaterialSecondary )
	DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_MaterialAccent )
	DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_MaterialChrome )
	DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_MaterialMisc )
	DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_SoundCue_Impact )
	DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_SoundCue_Ricochet )

#undef DEFINE_METHOD_ACCESSORS
#undef DEFINE_METHOD_GET_ACCESSOR
#undef DEFINE_METHOD_SET_ACCESSOR
};

Anyway, yeah, so that’s the gist — at a very basic level — of how I ended up handling text-based data-driven design/development. It may not be (and I actually hope it’s not) the best or most elegant solution, but it accomplishes what I want: external data files that do not require cooking to be used by the game; though, you do have to ensure that any data files that are necessary for a cooked game project are included into the resulting project builds. Unless you just want to keep everything on a server and download it at startup, which I can do as well, but I don’t want to force first-time players to have to get online to even get into the game, so a “starting set” of data is included with builds.

Also, I can’t toss up the entirety of my JSON serialization module, but I did include a very, very basic (but super helpful) wrapper for basic operations: JoySerializationManager.h and JoySerializationManager.cpp.

NOTE: I’m glossing over a whole lot of complexity that exists as part of this whole system and setup; partially because this isn’t the sexiest of topics to write about and I don’t know if anyone will ever even see this note. But also because it was really annoying and somewhat time-consuming to sort out, so I’m not 100% confident that the solution I have is the right one. I don’t like peddling lies. Well. At this moment in this context.

Additional Features and Future Work

Ease-of-Use (“Quality of Life”) Features

I generally keep my entire JSON file library open in VS Code at all times as it is, by far, my favorite editor for work that isn’t done in C/C++/C#. If I need to change game settings or tweak the balancing of a mech weapon or its legs (I tweak legs sometimes), I just alt+tab into VS code, make the change, and go back to work.

What was immediately obvious to add to my codebase after a day or two of doing that is the ability for the editor to watch (certain directories of) JSON files and, upon a change, automatically reload the file.

And I say certain directories as there are some file changes that I do not want to just get reloaded willy-nilly. If I’m in a session, make a change to a mech loadout, I don’t want to alt+tab back in and have my legs taken from me because I spelled something wrong and then crashes and then I would likely cry a lot. So: most directories. Being able to reload a mech loadout mid-session is handy, however, so I added a console command to refresh the current config file or load a different config altogether. The mech is entirely re-created in code but, due to an excess of iteration and annoyance, I don’t know if I’ve ever created a more gracefully-failing-and-recovering class in my life. The mech code is just a machine. Yes. Pun’d.

Servers and Security

As I mentioned, having the ability to send/receive JSON data to/from a server is absolutely crucial; being able to update server-side data files and then rely on the game client to query and retrieve the updates upon launch/online connection is the entire goal of the endeavor, after all.

There is the matter of security for these files which, admittedly, is not an area I’m an export or even a neophyte in, but a couple of Joy Machine “team members” (there is no money to pay anyone and so no one actively works — save my COO and CFO, but they have additional perks — but they still help plan/brainstorm/are wonderful) have started investigating a handful of possibilities. I’ll write about that when they tell me about it. Maybe. It may be an encrypted article.

The End Game

My dream of dreams, beyond live-ops and hot fixes and such, is to manage the game’s “asset library” through simple JSON file updates. And, no, I don’t mean magically transfer meshes and textures and maps and so on and so forth. That’d be silly. I mean that I can maintain an asset library of asset identifiers (“short names”) to meshes and textures and such, and that identifier is mapped to a path to the asset the short name represents (so it’s more like a registry). Asset library entries can be added and deleted and all that jazz, but they can also — if desired — update the asset that a short name refers to in the event that one asset is removed, but the short name is still applicable to something taking its place. None of this would require a change elsewhere, it’djust work.

And also worth noting: this is great in practice, but also it’s essential to not do a dumb thing and leave an asset library with a bunch of references to a single asset in existence forever. The idea is that for any actual update/patch, the library is cleaned and rebuilt and redistributed — and as part of that build process, it updates any data file that was using a soon-to-be-deprecated identifier as-needed. And, no, the rebuild is not the quickest of processes in the world, but it’s not manual, so I like that.

Again: dream of dreams, but I really look forward to seeing if I can eventually easily add completely “new content” to the game with the addition of a couple new part configs consisting solely of preexisting assets but assembled using a different ruleset (a given part is built from a subset of meshes/materials in order to form the actual “part”) for different purposes without users even realizing that a change occurred. Though, I don’t like the idea of invisible changes, so I’d include a constant “change log” prominently somewhere in the UI flow for players to scour through if they so desire.

In Closing

As I said at the beginning, I both hope and do not hope someone is immediately like “HEY YOU DIDN’T KNOW ABOUT <X>?” and ruins my night, but regardless, the systems in-place at the moment are already working wonderfully, so I can’t complain too much. And, unlike Blueprintpocalypse, the likelihood of a code change resulting in a JSON file crashing VS Code whenever it’s opened up is… Low.

But all of this is being done to get closer and closer to goals for the systems I want to end up with both for Steel Hunters and for future projects as well; I’m pretty constantly evaluating workflow, efficiency, and usability for ALL THE THINGS (also doing as much as possible to reduce human error when dealing with a large amount of data at a given time or a very, very long night), and that mentality has actually been yielding some really great stuff already. Even if it does occasionally mean yet another refactor of something. This doesn’t happen as often as I probably made it sound. Not as often. It does happen. And: crying.

Anyway, here’s a screenshot from one of a series of high-level layout “sketches” I did a week or two ago:

BROBOT looks intensely at what a potential future building to destroy may be. Maybe. If that “sketch” ended up sticking or being ruthlessly torn apart.