Moving the Needle with Tech

A little while back a friend asked me what kind of tech I would build if I had $1 million and I wanted to move the needle on game development. My first reaction was, "Nothing. $1 million isn't enough." The question has stuck with me though and I decided to write about it. My focus is on mobile games so YMMV.

Historically the focus on tech has been in developing a next generation game engine that enables better immersion, more realism, etc. I don't think there's much point in investing a lot of cash/effort in that area. A great place is some really cool hook for a specific game. The portal gun for example was an awesome little tech trick that defined the game. Those have great value, but it makes way more sense to have a game idea that needs something like that, than to create a little tech trick and go looking for a game that fits.

The area where I think the most value can be extracted is in team productivity. It's no where near as sexy or marketable as the first two. There usually aren't awesome screenshots, and the only people who can appreciate the value of it are folks who are involved in the making of games. But the business value is HUGE.

The mobile game ecosystem moves very fast compared to console or pc games. What is "hot" now is constantly shifting and competition is fierce so there's a strong need to get a title to market as quickly as possible. At the same time, the power of the hardware grows by leaps and bounds every year and just increases the tech gap between low and high end devices you need to support. You have two platforms, dozens of display aspect ratios and pixel densities, worldwide localized releases, and tons of third party SDKs to evaluate/integrate for analytics, advertising, A/B testing, live ops, etc which all add complexity to game development. And of course players (and platform sales teams) want/expect the depth and beauty of the games to greatly improve year over year ;) All of these things necessitate larger and larger development teams, but the mythical man-month is going to bite you in the ass.

Ok .. Enough with the doom and gloom. Tech to the rescue!

The maximum efficiency for a project is reached when all team members can be working with little to no interaction with each other. As the amount of required coordination between team members increases, the overall efficiency of the team diminishes. Two other pieces that massively impact productivity are time to assembly and time to defect discovery. The cost component of those last two are the same: The greater the time between when I create something and when I have to go back and work on it again, the most expensive it's going to be. In a perfect world, code and assets are delivered at the same time, and bugs are found before they ever leave the developers machine.

So with that in mind, any tech solutions we introduce should be focused on:

In no specific order, the issues that pop up as a sore thumb for me are:

Content Integration

When building a game, you need to integrate changes from all the different team members. The first step is to minimize binary assets (images, models, and audio only please) so merge tools can do their jobs. But really, trying to merge Unity prefabs with GUIDs that you don't understand is just begging to result in bad things. If the developer doing the merge doesn't immediately recognize the bug introduced by the merge it can often result in a content recreation, etc. Text files are critical for merging, but what is needed is a diff tool that understands the meaning behind the text. Integrate the branch/merge process into the editor for the game. Interpret the diff and show the UI for the fields being changed / in conflict. You can use this to preview changes/conflicts before you merge in changes! After changes are done you can turn on "git blame" within the UI to be able to see who the last person was that changed a property for a component, when it happened, and which commit it was. From what little I've heard, it seems like this is a large part of the focus for MaxPlay and I look forward to seeing what they're doing.

Interface Building

The game interface is a mess in mobile these days. Dozens of aspect ratios / screen densities and tons of languages to support. Building a UI that looks good everywhere is a great deal of work because you have to test on so many different devices so instead of spending developer time on it, you kick the responsibility to QA. Any time you update the product and send it to QA without verifying yourself you can expect a ton of bugs. The same thing happens with different languages - except your "normal" QA can't read the other languages so you end up with dedicated localization QA and wait till the very end to localize.

When I was doing mobile games in the pre-iPhone days, we built a really cool tool that let us run the game with multiple displays at the same time. You would have a primary one in the center, then a bunch of windows around the primary in all the different resolutions and orientations that you needed to support. You would run the same inputs and update routines, but you would call the render loop multiple times - one for each display. Everything would work in sync, and you could check for errors across every resolution you cared about in real time. It made interface creation / animation / etc massively easier and developers could have a high level of confidence that their UI was bug free before sending it off to QA.

The piece we didn't have, but was on the list before we switched over to iPhone, was setting it up so you would run the game in English (or whatever the developer's native language was) but behind the scenes the UI code would check for text errors in every other language. If a string is missing, or the localized text is too large for the area, the offending text is drawn in red/has a red outline, or some other visual that let's the developer know.

Client/Server

This one is near and dear to my heart. To minimize / prevent hacking and cheating in Free to Play games all of the player's authoritative state is stored on a server and every modification to the state is validated by the server before being applied. There is a ton of other functionality that can and is built off that, but at it's core, that's why you have a server in a FTP game. Unfortunately, the server code is rarely in the same language as the game, and building web-scale services is typically not in the wheelhouse of guys that are making the games. As a result, you often end up with both server and game developers and they duplicate the game logic in both systems in different languages. Best case, you end up with your game developer writing Javascript or something similar and trying to figure out why the math works most of the time, but sometimes gives different answers on the server than on the client.

This is a problem I've been working on with GameStack (http://gamestack.io) and I hope to release an Open Source version of it some time soon (please reach out if you think you could help!). What I've done is put together a generic catalog system that handles like 95% of the state changes, and then for the other 5% I use a form of distributed validation that allows the validation code to be written on the client. It ends up working like a replay for the game, which is a concept most game developers are very comfortable with. If done properly, you don't need to have a separate server engineer or write code in a foreign language to prevent hacking, which reduces team size and game complexity, and simplifies the whole dang thing quite nicely :)

Automated Testing

Last, but certainly not least, is automated testing. I should probably write a whole blog post (or several!) on this topic, but it's something I don't think game developers are using any where near as much as they could be. For now, here's a few ideas that focus on testing the content / gameplay.

For all of the above, you can also have some meta data that flags specific errors (based on line number, or some other context) as being ignored / as intended. For example, knowing that the grammar being used in some story text has been checked and approved as being intentional, or that certain images are actually used - they're just referenced procedurally.

One of the huge time sinks for both developers and QA is verifying that every quest can be accomplished. In the past, I've developed AI systems that played the game the way a player would, but at a massively accelerated rate. The AI is wired to the controllers (not the views) and is given back doors to fast forward time to get around things like construction times, etc. It's valuable to have the AI execute all possible sequence variations as it drives through all the quests in the game, which can take longer and can be run less frequently. On the surface, these kind of tests are really good for making sure all the quests are completable with no missing items, or fail if the player completes them in an unexpected order. They're also very good for triggering edges cases and exposing fragile code. In a client/server game, the concept becomes more problematic, but when using something like the client/server described above it becomes possible to stub out the server and do the entire test like a single player game.

Summary

So, where do we stand on the original $1 million question that started this whole thing? The majority of this stuff would be impractical/impossible without substantial low level integration with the development environment (Unity, etc) so unless you plan on making your own, you're probably out of luck. Maybe the folks from MaxPlay will get some/all of this right .. I sure hope so! The client/server solution is probably the only thing that can be done for under $1 million and actually move the needle, which I guess is why I started work on GameStack ;)